Dealing with huge text files in multithreads

Support
  • im using multithreaded bot of around 200 threads, no issue at all, but the longer the script runs the bigger the text files become. eventually bas becomes unresponsive because 200 threads loads a huge text file of about 30k+ lines.

    Each thread writes to this file as it needs to be updated by every threads and also read in every file. Right now I read the text file to a list, and checks if the list contains the same string.

    How can I do this to offload the computation required? High amount of threads is required, each thread will write to it and each thread needs an updated resource/list of this.

    I was looking for a "Global list" so I dont have to read/write every thread but no such options.

  • Use database instead, It is meant for
    storing and querying large quantities of data.

  • @0xSentinel you can use resources. access to file lines sharing.

  • 0 Votes
    1 Posts
    369 Views
  • Read CSV Files

    Support
    0 Votes
    3 Posts
    745 Views
  • CSS>Element-id>Match>text

    Support
    0 Votes
    2 Posts
    877 Views
  • 0 Votes
    3 Posts
    1169 Views
  • How to loop .txt file ?

    Support
    0 Votes
    2 Posts
    1495 Views