nsights on a potential fix or workaround would be highly appreciated. Thank you!
@dayyangee said in Twitter Data Scraping Issue: Unable to Retrieve Likes Beyond the 21st User:
As you can see in this picture, I have added loop and also scrolled, but still, I am getting less data.
I cannot see any logic in your picture that shows me that you scrolled before that loop to get the new data loaded.
If you scrolled inside a loop while scraping data and assumed that you would obtain that data within the loop, this logic is incorrect because the HTML DOM has changed, and you need to restart the loop to retrieve the newly loaded data.
You can not parse all data within one loop. You need to have at minimum 2 loops.
While true(loop first) <==
Scrape data with the Start Loop(loop second) from the last point or from the beginning if you are getting it for the first time. Scroll down or click to load new data.<==Alternatively, you can follow another approach:
Scroll down to the end of the page and wait for all the data to load. Scrape the data with the Start Loop