Error after 100.000 runs



  • I am having an issue with running a script: Input are 500.000 URLs to be scraped.

    After ~100k successes the script freezes.

    I am running this in run mode. Till then the output (writing json) works fine. The urls are loaded as external source but with the freezing, also the input file is not updated.

    Cant provide an error log as the script crashes :(

    Screenshot_6.png



  • so it crashed or just got freezes?
    has the browser process been restarted periodically or not?

    attach screenshot of resource use (system information) from process-hacker.



  • check all resources how much set in field - successful


Log in to reply
 

  • 1
  • 2
  • 3
  • 4
  • 2
  • 2
  • 3
  • 6