solution for massive database data check other than select db by criteria (high I/O)?

[yourfriendkarol]

What I do is:

1. get some data
2. select by criteria db action, it gives u list
3. if length of that list > 0 it means record is in db already - continue. If its not there, then it adds.

its good way to do it but i think with many threads my I/O is on max and it gets kinda slow, but maybe im wrong and its something else slowing me.
i basically have such thing as i described above running 24/7 and cant speed it up for some reason nomatter how many threads I add.

CPU, memory, disk, is all ok on low usage, only I/O high

[Allister]

@yourfriendkarol said in solution for massive database data check other than select db by criteria (high I/O)?:

What I do is:

1. get some data
2. select by criteria db action, it gives u list
3. if length of that list > 0 it means record is in db already - continue. If its not there, then it adds.

its good way to do it but i think with many threads my I/O is on max and it gets kinda slow, but maybe im wrong and its something else slowing me.
i basically have such thing as i described above running 24/7 and cant speed it up for some reason nomatter how many threads I add.

CPU, memory, disk, is all ok on low usage, only I/O high

This all depends, is the data within the table long/a lot? 1million + with a high character rate?
Have you tried other Database actions and/or filters to minimize the search criteria?
Would an external MongoDB you set-up & configure yourself might change the I/O bottleneck?
External DB: http://wiki.bablosoft.com/doku.php?id=how_to_setup_mongodb_server