@NewUsername1k said in Contact BAS Staff:
Hey Guys! How can I contact BAS staff? No support is required just business.
answered in PM
BAS bug in HTTP - Client, max post size
-
Hello,
I am using get element content (html) to capture the html on a variable and then I use HTTP Client / POST to send it to my php API for analysis, but I have noticed that it is working fine up to 8mb of variable content and then it is truncating the rest part of the content and other request variables, resulting in a broken/partial request.
The html in question is continuously growing as I scroll the page (IG followers / element _aano), so it always reaches sizes much bigger than the 8mb limit, I have seen truncated / broken posts up to 50mb.
Please advise me on if and how this can be fixed, or how it could be done in a different approach.
Thanks!
Waiting for your news!Anna ♥
-
@adrakou said in BAS bug in HTTP - Client, max post size:
Hello,
I am using get element content (html) to capture the html on a variable and then I use HTTP Client / POST to send it to my php API for analysis, but I have noticed that it is working fine up to 8mb of variable content and then it is truncating the rest part of the content and other request variables, resulting in a broken/partial request.
The html in question is continuously growing as I scroll the page (IG followers / element _aano), so it always reaches sizes much bigger than the 8mb limit, I have seen truncated / broken posts up to 50mb.
Please advise me on if and how this can be fixed, or how it could be done in a different approach.
Thanks!
Waiting for your news!Anna ♥
Are you having a problem receiving or sending data?
-
I see the variable in BAS seems fine (it contains all content, regardless of size), but on next action when it is being posted by HTTP Client, the Sniffer tool shows the request to be truncated at 8MB size, so the problem is posting the complete request when one variable is bigger than 8mb
-
@adrakou said in BAS bug in HTTP - Client, max post size:
I see the variable in BAS seems fine (it contains all content, regardless of size), but on next action when it is being posted by HTTP Client, the Sniffer tool shows the request to be truncated at 8MB size, so the problem is posting the complete request when one variable is bigger than 8mb
You haven't answered my question
-
The problem occurs on HTTP Client / POST action and concerns the sending of data, nevertheless, it is internal on BAS and the data never submits on my end / php api.
As I see on variable inspector, the content is properly collected within the var on previous step (get element content), but on http client / post action, the sniffer shows the big variable truncated at 8mb size, so the post is partial / broken.
-
-
On the hosting was uploaded picture size of 22mb, which means that there are no restrictions on the size of sent requests.
-
here is a sample of the request string, truncated at 1 M bytes:
https://easyupload.io/1dvn6gand here is a sample of a smaller request that is 100% valid, see in the end of the string, there are more POST variables, in the 1 Million bytes example, those and part of the captured_html, were truncated:
https://easyupload.io/tqhsakmaybe you can replicate the problem, if build a POST request with manually setting 4-5 variables, and using a string bigger than 1 Million bytes as the last variable in the editor (the http client of BAS seems to post the vars in reverse order than that set in editor, so the big var need to be set last in editor and you will see the post in sniffer, sent data to be truncated at 1 million bytes, skipping part of the last var and all the others)
-
Сan you write how exactly this problem can be repeated? What specific request and to what URL should I send? It is advisable to attach a test script.
So far I have not been able to reproduce the problem. -
please see this flow:
as you can see, this is a loop on the followers of an influencer on IG, the async function is changing the SCRIPT_RESULT to success when scroll has worked, else it will become false and the loop ends
so, every time we scroll, we get the element content _aano that contains the followers and we post it to a php script that does the parsing and saving on a db...
if the influencer is big and has lot of followers, the loop might take hours and keep scrolling and loading content... so eventually, ._aano element html becomes bigger than 1,000,000 bytes and the php script does receive broken / partial the expected variables, that are shown below:
as you will notice, the post builder has first variables action=audience, type=follower, influencer=[[CURR_INFLUENCER]] and in the end, there is the captured_html that takes the value of [[CURRENT_FOLLOWERS]], that comes from the action "get element content"
from experimenting, I have seen that BAS uses the post vars in reverse order than defined, so it first posts the last var and then the rest and I noticed that the post string on the HTTP Sniffer, shows to truncate at 1,000,000 bytes, so the var captured_html does pass partially to my php and the rest vars are missing, resulting an empty response, which I now handle to stop parsing more, but it makes the whole process inefficient and partial
are there other ways to do this or how this BAS limit can be raised ?
thanks!
-
@ryanwls Do I understand correctly that you parse instagram followers and then send their logins to your private server? Are you sure there are no restrictions on your server?
As an option, when the variable with subscribers reaches a large size, divide it into several parts and send it to the server in parts. -
Thanks for the answer!
Yes, the purpose is to find the handles that interfere with a specific influencer (audience analysis)
I send the whole HTML DOM to my PHP for further analysis, breaking the var in smaller ones would break the DOM structure and make the content invalid for parsing
How could I break the var in smaller ones without such problem ?
Is the limit of 1,000,000 bytes in POST action a known limit for BAS or it happens only to me ?
