solution for massive database data check other than select db by criteria (high I/O)?
-
What I do is:
- get some data
- select by criteria db action, it gives u list
- if length of that list > 0 it means record is in db already - continue. If its not there, then it adds.
its good way to do it but i think with many threads my I/O is on max and it gets kinda slow, but maybe im wrong and its something else slowing me.
i basically have such thing as i described above running 24/7 and cant speed it up for some reason nomatter how many threads I add.CPU, memory, disk, is all ok on low usage, only I/O high
-
@yourfriendkarol said in solution for massive database data check other than select db by criteria (high I/O)?:
What I do is:
- get some data
- select by criteria db action, it gives u list
- if length of that list > 0 it means record is in db already - continue. If its not there, then it adds.
its good way to do it but i think with many threads my I/O is on max and it gets kinda slow, but maybe im wrong and its something else slowing me.
i basically have such thing as i described above running 24/7 and cant speed it up for some reason nomatter how many threads I add.CPU, memory, disk, is all ok on low usage, only I/O high
This all depends, is the data within the table long/a lot? 1million + with a high character rate?
Have you tried other Database actions and/or filters to minimize the search criteria?
Would an external MongoDB you set-up & configure yourself might change the I/O bottleneck?
External DB: http://wiki.bablosoft.com/doku.php?id=how_to_setup_mongodb_server
