Topic about node.js and mongodb - best practices



  • Hi everyone,
    First let me say thank you to the BAS creator and the great community behind it. I find it very humble to give it for free. Already purchased Fingerprint Switcher and on my next paycheck I will purchase BASPremium (I don't I think I will use it, but it's very important to suppport the project!).

    So after weeks reading the forum (the RU version as well), the wiki, watching the videos... I feel already confident in the first part - scraping data, so I started with some more "not so beginner" scripts, but still trying to figure out the second part - Best way to store and manipulate the data?! I tried with txt files, csv, BAS database integration (local and remote) - they all work, but with limits and if I'm correct it's important to learn how to use BAS with mongodb (looks like NoSQL suits better BAS - just personal opinion based on my planned projects).

    Most of the youtube tutorials (thanks for them!) are quite basic and I didn't found there or here in the forum complete tutorial of real case full BAS project working with mongodb from which I can learn best practices from advance BAS users.

    Could someone give us tutorial or working BAS project with node.js mongodb?
    Short one - maybe database for proxies will be best example as we all use them. How to save them in mongodb with node.js with some CRUD operations - best way to add single record, multi records ( insert if not exists) delete...just to see when to call node.js, what script to include, should we close the connection ( db.close(); ) evertytime or just at the end of the project , how to insert LIST in mongodb (loop maybe) etc...
    Can be based on 1492968747062-proxyparser.xml ( thanks @Canine for sharing the script ), but instead of saving them in txt file, to save them in mongodb.

    So let me start first with 3 node.js scripts that are working, but far from good practice or for real complete BAS project.
    My idea is with your help to fix, improve and add more scripts in this post for working with node.js mongodb in BAS. Every tip you have will be highly appreciated, doesn't matter how small or big it is.

    Insert one record (not checking if exists :(

    Let's say you already set Variable in BAS with name: ONE_PROXY and value: 111.11.11.11:1111

    var MongoClient = require('mongodb').MongoClient;
    var url = "mongodb://localhost:27017/";
    
    await(new Promise((resolve, reject) => {
    
        MongoClient.connect(url, function (err, db) {
            if (err) throw err;
            var dbo = db.db("mongodb"); // database name
    
            var myobj = {
                proxy: [[ONE_PROXY]] // BAS variable send inside embedded node.js code
            };
            dbo.collection("proxies").insertOne(myobj, function (err, res) {
                if (err) throw err;
                db.close();
            });
        });
    
        resolve()
    }));
    

    Insert many records: (Still trying to find info how to add BAS list with the proxies and loop them in node.js)

    var MongoClient = require('mongodb').MongoClient;
    var url = "mongodb://localhost:27017/";
    
    
    await(new Promise((resolve, reject) => {
        /*Place your code here and call resolve to proceed*/
        MongoClient.connect(url, function (err, db) {
            if (err) throw err;
            var dbo = db.db("mongodb");
            var myobj = [
                { proxy: '192.1.1.1:111' },
                { proxy: '192.1.1.2:2222' },
                { proxy: '192.1.1.3:333' },
                { proxy: '192.1.1.4:4444' }
            ];
            dbo.collection("proxies").insertMany(myobj, function (err, res) {
                if (err) throw err;
                console.log("Number of documents inserted: " + res.insertedCount); // the log is not working. I dont know why.
                db.close();
            });
        });
        resolve()
    }));
    

    Delete many records if match:

    var MongoClient = require('mongodb').MongoClient;
    var url = "mongodb://localhost:27017/";
    
    await(new Promise((resolve, reject) => {
    
    
        MongoClient.connect(url, function (err, db) {
            if (err) throw err;
            var dbo = db.db("mongodb"); //database name
            var myquery = { proxy: 192.11.11.11 }; // delete everything that match proxy: 192.11.11.11
            dbo.collection("proxies").deleteMany(myquery, function (err, obj) {
                if (err) throw err;
                console.log(obj.result.n + " document(s) deleted"); // I don't know why the log is not working
                db.close();
            });
        });
    
    
        resolve()
    }));
    


  • BAS already works with mongodb ....though mongodb used is older version.


Log in to reply