solution for massive database data check other than select db by criteria (high I/O)?



  • What I do is:

    1. get some data
    2. select by criteria db action, it gives u list
    3. if length of that list > 0 it means record is in db already - continue. If its not there, then it adds.

    its good way to do it but i think with many threads my I/O is on max and it gets kinda slow, but maybe im wrong and its something else slowing me.
    i basically have such thing as i described above running 24/7 and cant speed it up for some reason nomatter how many threads I add.

    CPU, memory, disk, is all ok on low usage, only I/O high



  • @yourfriendkarol said in solution for massive database data check other than select db by criteria (high I/O)?:

    What I do is:

    1. get some data
    2. select by criteria db action, it gives u list
    3. if length of that list > 0 it means record is in db already - continue. If its not there, then it adds.

    its good way to do it but i think with many threads my I/O is on max and it gets kinda slow, but maybe im wrong and its something else slowing me.
    i basically have such thing as i described above running 24/7 and cant speed it up for some reason nomatter how many threads I add.

    CPU, memory, disk, is all ok on low usage, only I/O high

    This all depends, is the data within the table long/a lot? 1million + with a high character rate?
    Have you tried other Database actions and/or filters to minimize the search criteria?
    Would an external MongoDB you set-up & configure yourself might change the I/O bottleneck?
    External DB: http://wiki.bablosoft.com/doku.php?id=how_to_setup_mongodb_server


Log in to reply