Twitter Data Scraping Issue: Unable to Retrieve Likes Beyond the 21st User
-
-
@dayyangee It's impossible to know what you did to process this data, so it's difficult to accurately address your problem.
-
While true:
- Scrape data with the Start Loop.
- Scroll down or click to load new data.
- Repeat step 1, ensuring that you store the scraped data in a variable to avoid scraping it again after new data is loaded. Because the Start Loop will parse both the old and the new data that loaded after you scrolled.
-
-
@dayyangee said in Twitter Data Scraping Issue: Unable to Retrieve Likes Beyond the 21st User:
but the data is not getting scraped as much as I want.
If the browser does not display the data you want, I believe it is not a BAS issue but rather that the target website will not provide you something.
I mean, if this logic is the same as in the real Chrome.What answer do you expect in this forum? How can you avoid it?
-
-
@dayyangee said in Twitter Data Scraping Issue: Unable to Retrieve Likes Beyond the 21st User:
As you can see in this picture, I have added loop and also scrolled, but still, I am getting less data.
I cannot see any logic in your picture that shows me that you scrolled before that loop to get the new data loaded.
If you scrolled inside a loop while scraping data and assumed that you would obtain that data within the loop, this logic is incorrect because the HTML DOM has changed, and you need to restart the loop to retrieve the newly loaded data.
You can not parse all data within one loop. You need to have at minimum 2 loops.
While true(loop first) <==
- Scrape data with the Start Loop(loop second) from the last point or from the beginning if you are getting it for the first time.
- Scroll down or click to load new data.<==
Alternatively, you can follow another approach:
- Scroll down to the end of the page and wait for all the data to load.
- Scrape the data with the Start Loop
As you can see in this picture, I have added loop and also scrolled, but still, I am getting less data.