BrowserAutomationStudio 24.0.2 has been released

[support]

In the new version, a transition was made to a new browser engine. This means that the architecture of the project has changed significantly, now browser workflow works like this:

1. At the thread start, a new browser instance is launched in the background.
2. This instance is absolutely the same as a regular Chrome browser and does not have the same limitations as before.
3. The browser is controlled by BAS using a mature protocol supported by Google.
4. BAS interface itself and browser actions have not changed.

What are the advantages?

This kind of automation is very difficult to detect. This is possible only by indirect tests, such as: repetition of the same actions, implausible pauses between actions, etc. Browser detection by missing functionality is extremely difficult to achieve, and the fingerprint changing system makes it difficult to collect information about the hardware.

Old BAS projects are easy to convert. Despite the fact that a lot has changed inside the software, some functionality has been completely rewritten from scratch, it is easy to convert the old project to work with new engine. In some cases, it is enough just to open it in a new version. Still there are some small exceptions and they are described below.

New improvements. The new engine system also gives possibility to add new features. Some of them have already been added into new version, some will be added later.

Extensions support.

The new version has the ability to use browser extensions. In order to do that:

1. Download the extension using this utility. When downloading, you need to select zip format.
2. Unzip to any folder.
3. Specify this folder in the "Browser Settings" action.

Unfortunately, access to the extension interface is not supported yet. It is planned to implement this in one of the next versions.

Updating browser version in the future.

A very important factor of the browser authenticity is the presence of all methods, classes and javascript constants. In other words, if the version of the browser is old, it can be detected. To avoid this, it is planned to update the browser version very often. In the best case scenario, this will be done every month along with a Chrome update.

To ensure that constant updates will take place, tests were written for actions which works with the browser (this was done by @GhostZ), detailed documentation for each patch was added, tools for patching were improved, tests for fingerprints changing will be written in the near future.

The transition to a new browser engine was also useful in this matter. It is expected to have significantly fewer automation bugs when updating the browser version.

Piхеlscan test.

Many of our users asked about this test, now BAS is passing it successfully, in order to do this:

1. You need to change the fingerprint using the FingerprintSwitcher service.
2. Allow to getting canvas and webgl data in the "Browser Settings" action OR use PerfectCanvas.

As for today, there is a bug in this test. A regular browser is defined as an automation framework, screen. This happens because quite recently the navigator.webdriver property in standard Chrome browser has been changed its value from undefined to false and the test didn't update yet. In this case, BAS is also "detected" as an automation framework, just like standard Chrome browser right now.

If you are using a proxy for this test, you need to have a resident ip.

With a new browser, the "native" canvas data is not unique, you can check it on the following page https://browserleaks.com/canvas

CPU optimization.

Additional CPU load compared to standard Chrome can occur when constantly transferring images from the browser to BAS. Reducing it turned out to be not an easy task at all. In order to do that, the data transfer protocol was rewritten. The original version wasted time compressing data and transferring it over a websocket, this approach gave unacceptable results at a high frame rate.

The new protocol helped, but the solution was still not perfect. Therefore, adaptive frame rate reduction has been added. If the user activates the browser, the fps is increased. When the browser is minimized, fps is decrased again. This method helps to significantly reduce the load in multithreading mode, and also doesn't influence script execution, since the optimization does not affect any other part of the browser functionality except images transfer.

Optimization of hard drive usage.

The profiles size have been significantly reduced in size compared to a regular browser.

1. Disabled crashpad.
2. Disabled browser metric. Used to get information about the usability of the browser.
3. Added flags to the chrome_command_line.txt file prohibiting some types of cache. This can be disabled by editing the file.

The size of the profiles had a positive effect not only on the amount of used disk space, but also on the start time of each thread.

Widevine

In the new version, widevine must be explicitly enabled in the settings, or in the "Browser Settings" action. Unfortunately, BAS is not authorized by Google, so sites like spotify won't play the content.

"Javascript" action

Added a new version of the "Javascript" action, it can execute code in the context of the browser just like the old one, but does it much better:

1. Added possibility to work with BAS variables.
2. Waiting for asynchronous actions.
3. Getting exception information.

Most of the work was done by @Oyasumi-Punpun

The old action will continue to work, but it is now deprecated.

Setting referrer.

Setting the referrer is now done through the "Load" or "Add Tab" action. This should be taken into account when porting a project.

Emoji.

The new version adds support for emoticons when entering text. This input method is still not reliable, because there are no emojis on the keyboard, and therefore the keyup, keydown and keypress events are not emulated.

The presence of emoticons in the project resources will not allow compiling the project with protection.

The characters input which consisting of 4 bytes is not yet implemented, this includes country flags for example.

New fingerprint factors.

New fingerprint factors have been added in the new version. The database has not yet been fully updated, so it is recommended to use fingerprint not older than 2 weeks.

Module updates.

New modules "String" and "SQL" have been added. The first expands the capabilities of BAS for working with strings. There are actions for formatting, editing, searching inside strings, and also for parsing text. The second one allows you to work with popular SQL databases: mysql, mariadb, postgres, sqlite, mssql. It is possible to perform primitive actions without knowing the syntax, as well as to perform arbitrary queries to the database.

Thanks @GhostZ for these modules.

Limitations of the new version.

We didn't manage to convert all browser stuff to the new version yet. Here is the list:

1. Drag and Drop is not supported. This means that you can drag the volume slider in the video player, but you cannot drag the file to the browser yet.
2. Recording http requests does not work yet https://i.imgur.com/P1JxkxC.png.
3. Displaying select items does not work yet https://i.imgur.com/ZkDmTBy.png. But the value of the element itself can be selected with arrows or text input.
4. HTTP authentication does not work yet.
5. Displaying different types of cursors does not work yet https://i.imgur.com/fVAHclk.png
6. Developer tools panel opens in the system browser in a separate window.
7. It is not yet possible to open developer tools for a specific element.

These are not the most important things, so it is not desirable to postpone the release of the version until they are completed. However, they will all be fixed in future releases.

Achieving maximum anonymity.

If the site still manages to detect a new BAS version and you are sure that the problem is not in the proxy, not in the phone number, or the actions sequence that you perform, but in the browser itself, then here is the list of actions that need to be taken:

1. Make sure that the problem persists in Chrome in incognito mode or with a new profile.
2. Disable the AjaxReCaptcha2 module.
3. Disable canvas, webgl protection or use PerfectCanvas.
4. Try the steps in manual mode (no automation).

If all of these options fails, use the algorithm described in this thread https://community.bablosoft.com/topic/12199/.

How to convert an old project.

In order to do this, you need to open the old project in a new version and make the following changes:

1. All "Set header" actions with the Referer parameter must be replaced with the Load or Add tab actions.
2. Re-create actions: Reset, Set combobox by value, Set combobox by index, Set combobox to random, Previous Page, Add Tab, Is Element Exists

Remember that the format of cookies and profiles has been changed. If your project relies on old format of profiles, you need to update them.

!IMPORTANT Users of a 32-bit Windows version won't be able to update automatically. After downloading, a crash will occur. The new version needs to be downloaded manually. There will be no problems for the 64-bit version.

Patch 24.0.3

We have fixed the most important problems with new engine.

Fixed errors when running some sites.

Stream to twitch could be interrupted with error 3000 on some proxies and only when automatic quality was selected.

The epic games store login page did not load completely.

These issues have been fixed. But for authorization in EGS, you need to enable canvas / use fingerprints and disable AjaxRecaptcha2.

These fixes are not related to these sites only and may work in other places.

Fixed ssl problems.

The "This site can't provide a secure connection" error should no longer appear.

We were able to reproduce the problem, but only in one specific way.

If the problem persists after the update, you need to provide detailed steps to reproduce it.

The default frame rate is set to 30.

This significantly reduces the CPU usage.

In fact, now the new version in multi-threaded mode consumes less CPU time than the previous one and even less than standard Chrome.

Thanks to one of our users for the idea.

This code is very hardware dependent. If the fix does not work on your monitor, please write to @Fox with a problem description.

Browser antidetect has been improved.

Imреrva protection, which detected BAS when changing proxy, now does not do this.

Change proxy on the fly.

Fixed problems with changing proxy without restarting the browser. After update the "Proxy" action has been increased for 3 seconds.

Disabled preinstalled extensions.

If there are certain values in the registry, extensions installed on the user's PC can be added to new profiles created by BAS.

After update this feature was disabled.

https://www.chromium.org/administrators/pre-installed-extensions

Improved smooth thread start mode.

After update the thread is considered to be running only when both the BAS helper process and the real browser are running, and not just if the helper process is running as this was before.

These calculations are required for the system to smoothly start threads.

Previously, the number of simultaneously launched browsers was 1, now it is 3. This value can be changed in the settings.

Header modifications for all requests.

Standard methods for changing headers such as User-Agent and Accept-Language do not change them for all requests.

This makes it easy to detect browsers that rely solely on these methods.

A new reliable methods has been added which allows to replace absolutely all requests.

It acts in a similar way to how tunneling works for proxies, only in this case header values are changed.

Below is a screenshot showing that even Chrome's internal requests are being replaced:

Please, do not to judge whether the headers are being replaced or not by looking inside the developer tools, this information is not always accurate.

Patch 24.0.5

Optimization.

Now you can set the maximum FPS. This can be done on the settings page or through the "Browser Settings" action. Changing the frame rate can be useful to reduce the CPU load on sites with animation.

By default, the value is set to 30.

Previously, when creating a new profile, the browser was doing several resource intensive tasks. After fix CPU usage on thread start was reduced.

Bug fixing.

• BAS works correctly when disconnecting RDP session.
• Fixed focus steal when running script in background mode.
• Fixed input diacritic symbols and Alt Graph modifier keys.
• Fixed obtaining selector on some sites.
• Fixed connection error for certain proxies.

Patch 24.0.7

This patch has added many performance fixes. Almost all types of PC resources usage has been optimized.

CPU

A lot of CPU intensive features have been disabled: various metrics, some crashpad functions, virtual display checks, hwid retrieval, parental controls, and more.

Functionality which is vital for browser operational was not removed. These patches reduce the load when creating a profile by an average of 10%.

Checksum checking for Chrome extensions and monitoring of these files has been disabled.

Memory

The size of the memory allocated for image transmission is now dynamic. This allowed to reduce the amount of memory consumed, especially for browser instances with low resolution.

Fixed a bug where the size of available physical memory on a machine could be determined inaccurately.

HDD

A mechanism for caching profiles has been added. In order to create profile, browser needs to initialize each file located in profile folder. This improvement allows to copy common files rather than creating them every time. Note that profiles are created implicitly each time new thread is started.

The list of BAS modules is cached in order to avoid reading them at the start of each thread.

Smooth browser start and stop

The mechanism for smooth start of browsers has been improved. The browser can only be started if the processor has not been very busy for the last few seconds.

Several browsers can no longer be launched in same moment. A short time interval must pass before the next launch. This avoids large spikes in CPU load.

Stopping browsers now also happens smoothly, all browsers don't stop at once. This behavior could cause the system to freeze for several seconds beforehand.

When the browser is stopped, the location of the profile is taken into account. If it is located in a temporary folder, the process stops immediately, this is faster, but it can lead to the loss of profile data. If the profile is not temporary, browser closing occurs correctly.

You can change the soft start parameters in the settings.

System freeze

In a previous version after opening the task manager system can hang for 10 seconds. This can happen only if a lot of threads are started. Issue has been fixed. Thanks to the user who reported this bug.

When there is a shortage of RAM, various browser components begin to free it up or perform other actions. This can also lead to system freeze when the amount of free memory decreases. After the update, the browser will think that there is enough memory, even if it is not.

Patch 24.0.8

Browser authenticity.

Improved hiding variables in the browser algorithm.

Fixed some problems with BAS detection in record mode.

AjaxRecaptcha2 module can be detected. The unsafe code has been moved to the ReCaptcha module, which is disabled by default.

If old projects work with recaptcha, you need to enable it here https://i.imgur.com/xuuiQPQ.png

Editor improvements.

Fixed working with any actions that restart the editor. For example, "Reset" or "Create or switch to local profile". These improvements only apply to the record mode.

During execution of these action, execution point does not reset, and the script can continue working correctly:

https://i.imgur.com/syahwcI.gif

Moving the execution point no longer resets the call stack, loop state, etc.

https://i.imgur.com/znDCf12.gif

Bug fixes.

Fixed work of tabs on some sites.

In previous versions, when you activate drag and drop, the browser freezes. This has now been fixed. Full drag and drop support will be added with the next browser version update.

UDP.

BAS can tunnel UDP protocol. But not all socks5 proxies support this. QUIC is a UDP-based protocol developed and used primarily by Google. This can lead to some problems in case if it is used. For example, websites with Google ads may experience freezes, pages may not load completely.

To solve this problem, QUIC is now disabled by default in BAS. It is recommended to make sure your proxies support UDP and enable QUIC if so. This can be done in the settings or in the "Browser Settings" action.

[HELP_ME]

Brilliant work!! Thank you @GhostZ and @Oyasumi-Punpun and @support.
Big thanks to @Fox as well
"New fingerprint factors have been added in the new version. The database has not yet been fully updated, so it is recommended to use fingerprint not older than 2 weeks."
So I should disuse my old fingerprints (more than 2 weeks old) and create new ones?

[pubbli]

What a great team! Thanks!

I recently started using the Drag and Drop feature, is there any workaround for this while waiting for the full support?

[GhostZ]

@pubbli, only the action "Start Drag File" does not work

[linagoode1]

BAS 24.0.2 is a Great version! Thanks!
Moudle SQL and Moudle Excel can't work normally.
LOG:
[944464877] [19:35:20] Thread #1 : Running script without embedded languages

[GhostZ]

@linagoode1, most likely, you clicked the button that launches BAS without Node.js, these modules use Node.js to work

[support]

@HELP_ME

So I should disuse my old fingerprints (more than 2 weeks old) and create new ones?

It is better to use fresh fingerprints for this version.

[support]

@pubbli

I recently started using the Drag and Drop feature, is there any workaround for this while waiting for the full support?

Need to wait for update or use old version.

[Skill]

The new version cant type @ :(

[Fox]

@Skill said in BrowserAutomationStudio 24.0.2 has been released:

The new version cant type @ :(

install the English keyboard layout in the operating system:
https://community.bablosoft.com/topic/15558/cannot-write-when-entering-email-address-with-bas/22

[support]

Patch 24.0.3 has been release, more info in topic.

[X.O]

@support where is topic?

[spacerange]

@X-O said in BrowserAutomationStudio 24.0.2 has been released:

@support where is topic?

https://community.bablosoft.com/topic/15546/browserautomationstudio-24-0-2-has-been-released

[Dasium]

@Fox @support

"Remember that the format of cookies and profiles has been changed. If your project relies on old format of profiles, you need to update them."

So how to update the profiles to makes it work?

[morpheus93]

Just installed the lastest update to version 24.0.8 and the Recaptcha2 cube in my project stopped working (opens the captcha box and instanly closes it and other strange behaviour). Have you made some changes to the Recaptcha recognition function? Also I have Recaptcha2 2 times in the context menue in the browser window of BAS (see attached screenshot).

[GhostZ]

@morpheus93, @support said in BrowserAutomationStudio 24.0.2 has been released:

Patch 24.0.8

AjaxRecaptcha2 module can be detected. The unsafe code has been moved to the ReCaptcha module, which is disabled by default.

If old projects work with recaptcha, you need to enable it here https://i.imgur.com/xuuiQPQ.png

[Fox]

@morpheus93 said in BrowserAutomationStudio 24.0.2 has been released:

Just installed the lastest update to version 24.0.8 and the Recaptcha2 cube in my project stopped working (opens the captcha box and instanly closes it and other strange behaviour). Have you made some changes to the Recaptcha recognition function? Also I have Recaptcha2 2 times in the context menue in the browser window of BAS (see attached screenshot).

https://community.bablosoft.com/topic/15546/browserautomationstudio-24-0-2-has-been-released/2

[spacerange]

@morpheus93 said in BrowserAutomationStudio 24.0.2 has been released:

Just installed the lastest update to version 24.0.8 and the Recaptcha2 cube in my project stopped working (opens the captcha box and instanly closes it and other strange behaviour). Have you made some changes to the Recaptcha recognition function? Also I have Recaptcha2 2 times in the context menue in the browser window of BAS (see attached screenshot).

Most likely, the second menu item is show due to actions from the old module in your script, since this item disappears when creating a new project.

[morpheus93]

Thank you guys for your appreciated help and the great work, especially for the latest fast updates!

[zzinan]

Thanks for the latest update. But in the last version, it gets stuck where it should go in my script. Could this be a bug?