Dealing with huge text files in multithreads
0xSentinel last edited by
im using multithreaded bot of around 200 threads, no issue at all, but the longer the script runs the bigger the text files become. eventually bas becomes unresponsive because 200 threads loads a huge text file of about 30k+ lines.
Each thread writes to this file as it needs to be updated by every threads and also read in every file. Right now I read the text file to a list, and checks if the list contains the same string.
How can I do this to offload the computation required? High amount of threads is required, each thread will write to it and each thread needs an updated resource/list of this.
I was looking for a "Global list" so I dont have to read/write every thread but no such options.
GaG last edited by
Use database instead, It is meant for
storing and querying large quantities of data.
tet-vivi last edited by tet-vivi
@0xSentinel you can use resources. access to file lines sharing.