Bablosoft

@ptt-bds said in Article scanning tool for newspapers:

I am using BAS to develop a tool for scanning new articles on over 200 newspaper pages. I fetch articles from one of three sources:

Articles listed in the sitemap.
Articles available in the RSS feed.
Articles displayed on the homepage.
However, yesterday I encountered a case where an article did not appear on the homepage or in the sitemap. Instead, it was located in a subcategory.

I would like to inquire about a more comprehensive approach to scanning articles so that none are missed.

Your task shouldn't be very difficult to implement, but not with BAS. Instead, consider using other tools. If you have some experience with Python, take a look at https://scrapy.org.
Scrapy, for example, has its own crawler spider that can gather information from websites, which might be exactly what you need.

@morpheus93 As I want to keep full anonymity (I am fooling ad networks), I cannot allow myself to block images from those "anti-fraud" services, but I can at leasts block the ones of the website.

The thread you linked was very helpful. There is basically 2 methods:

First one is to use a cheap proxy to cache most elements (so you don't download it again) with a cheap proxy and then copy the profile to use it with your actual good proxy. The idea is good, but I cannot allow myself to use a cheap proxy if there is "anti-fraud" services as they could quickly flag in some way the profile by caching something you don't want. So this solution is not solid.

The second one was basically the same thing, but instead of going through the trouble of doing profiles, you have some kind of software that reroute the traffic depending on the source website. So important traffic will be routed to a good proxy while useless traffic will be routed to a cheap proxy. But the problem will still persist, as those useless traffic could be "anti-fraud" that will flag your operation and make it fail. So this solution is not solid either.

In conclusion, blocking images from the main website seems the most secure option to save bandwith, but it will not reduce the bandwith enough to make it profitable as the main costs are the ads and third-party images. For example, my website's image only consume 170Kb. So I am not saving much.

I hope there could be some secure way to save significant bandwith.

If not, I guess I'll have to look into services that offers unlimited data, but usually they are spammed and very expensive.

Hey, thanks for sharing this information! I absolutely love Spotify and its vast music library. It's incredible how many hidden gems you can discover there. Recently, while exploring the platform, I came across a fantastic music promotion service called Songlifty. It's designed to help young and aspiring artists gain more exposure on Spotify. I think it's a fantastic opportunity for artists to showcase their talent and connect with a wider audience. If you're interested, you can check out their website at https://songlifty.com/spotify-promotion/. As a newcomer to this forum, I'm really excited to dive deeper into the world of music and connect with other passionate music lovers.

@ahmad BAS runs independently on each VPS.
Learn how to control the entire VPS at the same time, instead of controlling all the BAS on all the VPS.

@Q_Q Hello. It's a great program, thanks for sharing. Any chance to update? Downloading with Watermark works but downloading without Watermark does not.

9d860a7f-8dba-49bc-a0cd-2335cbf76642-image.png

64073e0f-6ecc-494a-b524-b4eeb6a60fee-image.png
Can't use chrome-extension after applying android fingerprint :( previous versions do it very well

@DuckDuck Thank you very much. Besides blocking libraries, what else can be done to optimize BAS and reduce RAM and CPU usage? Because the tool I'm working with is expected to simultaneously scrape 20-30 news pages.

@nosnossos92 хром 114 обнова ещё не всем прилетела, у меня на моём ПК ещё 113
В такие переходные периоды надо получать 113 и 114 и применять движок. Когда 114 выйдет массово, то забыть про 113
Я себе такое уже сделал.

Well, you need to get your userAgent and browser version to match. CreepJS shows that you have v109 userAgent running in Chrome 114, so you need to set userAgent to version 114.
Though I am not familiar with other errors, it states that you have something bad going on with canvas and client rects

this is bullshit fr, now there s a new service ake. i tried it a month ago, and think i will stay with it, was another one, it is also ok, but these are just for parsing, u can find promo from their forums. i have one but idk XXCEHX u can try

@aesmer The question is simple, so the answer should be simple as well.
Just use the Logger module and uncheck the "Show msg "waiting element"" option
1.png
2.png

@xuandungpy The problem here is getting the Login Code from the application.
Do it manually:
Open the telegram application corresponding to the account you need to log in on the browser.
Then just do the steps to log in on the web browser, enter the phone number, get the Login Code on the application and enter the password 2fa.

Automatic
Run Process in BAS can be used in combination with other languages such as Python, Js... to get the Login Code from the application.

This could be a way to deal with it.
I am not dealing with this issue.
Please try.

@hellomembersha said in Database restart where left off:

@Fox If I want to filter database for any lines NOT matching a specific date in one of the columns how would I do that? It would be a reverse filter basically.

In the standard database module, the "not equal" condition exists only for the "int" data type, which means you can write a date in milliseconds to a column with this data type and then use a filter with the "not equal" condition

@BAS-Viet-Nam Thanks for responding. I thought it can be handled by using a Javascript code