Dealing with huge text files in multithreads

[0xSentinel]

im using multithreaded bot of around 200 threads, no issue at all, but the longer the script runs the bigger the text files become. eventually bas becomes unresponsive because 200 threads loads a huge text file of about 30k+ lines.

Each thread writes to this file as it needs to be updated by every threads and also read in every file. Right now I read the text file to a list, and checks if the list contains the same string.

How can I do this to offload the computation required? High amount of threads is required, each thread will write to it and each thread needs an updated resource/list of this.

I was looking for a "Global list" so I dont have to read/write every thread but no such options.

[GaG]

Use database instead, It is meant for
storing and querying large quantities of data.

[tet-vivi]

@0xSentinel you can use resources. access to file lines sharing.