in this video we will deal with this website using page methods of playwright
this is quotes.toscrape.com again but it is its javascript delayed version so if you refresh
this page it's going to take a while to load all the data so we are going to learn how to
overcome this we will see two methods the first one will be wait for timeout and we
are going to wait for a number of seconds that we decide and the second one will be
wait for selector and we are going to wait for an element to be appear on the page let's copy
this code spider again I'm going to create a new one scrapy genspider this will be waitspider let's
say and let's say x.com I'm going to delete it
now let's delete those and copy the quote
spider copy it and paste it in here okay now we are going to make some small adjustments
in this code let's first import this item from items import quote item okay let's change the
url in here js-delayed and let's run this see what you'll get I'm going to open the terminal
and in here let's write scrapy crawl wait spider we haven't changed anything yet just
the URL is different let's save the data to wait.json okay let's run this and we are not
going to get anything from this website why is that because it's not going to load the items
it started doing some requests opening the playwright page but as you see if you look
at in here we have scraped zero items why is that because the page is not loaded the items yet so
we are going to use page methods in here to do this I will import from scrapy playwrigh.page
I will import page methods okay this was page coroutines before now it is page methods and
in here in this meta after this playwright true I'll will write another thing this will be again
playwright page methods page methods okay and this will be a list we are going to put just
one element in it but it will be a list this will be a page method and open parentheses we
are going to write the method in here we are going to start with wait for timeout okay I
think the website is waiting for 6 or 7 seconds so I'm going to write 10 in here wait for timeout
we have used this before in a different form in the previous video if you remember okay and
this will be for 10 seconds so I'll write 10,000 because those are in milliseconds okay now we are
going to run the same thing one more time and see if we are going to get the data let's run
this one more time the same comments if this is going to work this will take a little time so I
will see you at the end of this run let's check the results here we have scraped 10 items why
is that we just use this for the first page if I open this wait.json we got all the items from the
first page but we need to use it for every page we got so I'm going to copy this page
methods from here and paste it after this one
okay so now it's going to use it for all the pages but what is wrong in
here one of them is actually
unnecessary okay now we can run this one more time and this time we are going to get all the
items because we will wait for 10 seconds to every page to load all the data let's run this
again and see you at the end of this run let's check the results again as you see
we have scraped 100 items this time so this is working now let's also check the json file
as you see we have all the data in here this time I'm going to change this wait
for time out to wait for selector let's copy this and change this to
selector and also comment the first one out okay this time the selector will be div.quote
because we are looking for those items on the page div.quote as we use in here
let's also copy this for the other pages and paste it here okay now if you run this
hopefully we are going to get the same result but this time we are not going to wait for
10 seconds we will wait for a while until this d. code item is on the page after that
it will start scraping the data okay let's run this one more time let's call it selector.json
run this and hopefully we will have the we will get the same results see you at the
end of this run let's check the results here is the item scraped count it is 100 and if I
look at the selector.json file we will see that we got the same output with the other
one okay this is how to wait for an element on the page or how to wait for a number of
seconds that we decide on a page for the websites that are loading to data lately
we have used this page method we have imported it and put it in this meta information and with
the help of this we are able to use this parse method synchronously even if we are using some
methods of playwright okay this is the end of this video and this is the end of this section