I'm currently playing around with a local Squid caching proxy. In general it works well and reduces data consumption a lot but seems to have some problems the way how BAS sets proxy connections. Have to ask about that in another thread.
I got a same question, so watching.
when u using Squid, what about the "stay under the radar of the sites anti-bot mechanisms" ? does fingerprint still working right?
@morpheus93 How have you set Squid local proxy? I mean you use other proxy, so you settled root Squid and trhough it then you are using Luminati? Can you, please, explain, I do not need traffic economy, I need faster load and I don't need a lot of cache information, I aslo wonder how you restricted img and how does it affect fingerprint
has anyone found an individual solution that he want to share here? would be interesting to discuss the pros and cons of the differente approaches.
I'm wondering if sharing cache/cached files between different accounts (profiles) will lower anonymity (e.g. the target website can track accounts together as they share some cached pictures or css/js files)?
What are your recommendations/experiences guys?
I believe there is possible way to proceed:
Additionally, you may need to make some deals with local storage depending on the nature of your production script. Make sure to factor this into your process as needed.
In my experience, to save on traffic, I implemented the following solution:
I maintained my own list of websites for the cheap proxy. If a request was made to one of these websites, it was redirected to the cheap proxy. Otherwise, it was redirected to an expensive one. For example, if the request was for static files (CDN), I redirected it to the cheaper proxy.
I used Squid software for this purpose and wrote a script to maintain the required configuration files. This approach helped me optimize traffic and reduce costs for the websites I managed.
In addition, to further reduce traffic, I also banned some websites like google-analytics.com, hexagon-analytics.com, and many others instead of redirecting them to the cheaper proxy.
By using a cheaper proxy for requests to static files and maintaining a list of websites that could use the cheaper proxy, I was able to significantly reduce the cost of mobile traffic for that project.
However, I'm not sure that Squid is the best solution for this task. In the future, I plan to try using HAProxy because I think redirecting TCP connections may offer better performance. This could potentially lead to even further improve the performance.
@sergerdn Wow, thank you very much for sharing your experiences and many helpful hints.
I'm currently working on a quite similar approach as you described in your posting above (collecting cache data with cheap proxy and then switchig to the more expensive one).
Hope this works as expected and will not open some new anti-bot detection capabilities.
Your approach using a centralized re-direction server sounds very sophisticated. I will also look into this and how it can be implemented.
Thank you!
@morpheus93 said in Best method(s) to minimize/compact traffic consumption of project:
Your approach using a centralized re-direction server sounds very sophisticated.
Thank you for your feedback on my previous approach. I agree that using a main-in-the-middle proxy server can be a more efficient solution. By centralizing the redirecting process in a proxy server, you can avoid making changes to each individual BAS script and instead make updates to the proxy server configuration. This can save time and effort in managing and maintaining the redirection process.
In addition, using a proxy server can provide additional security features such as filtering. For example, you may want to block some network requests to certain domains, which can help improve both traffic spending and security.
But of course, you can implement both a main-in-the-middle proxy and prepare your profiles with cheap proxies to fill up the local cache. This can help improve performance and further optimize your traffic flow.
Overall, a main-in-the-middle proxy server combined with a cheap proxy cache can be a powerful solution for managing and optimizing traffic.
@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn Thank you for sharing. Can I get a copy of your config for reference? Thank you very much
I'm not sure if it will be of much help to you. Please note that I only copied and pasted a small part of the project, and you'll need some more information to make it work.
I apologize for any incorrect comments I may have provided as I do not remember specifically which line they referred to.
Here is snippet:
squid.conf:
# This ACL is used to exempt requests to any backup proxies from the other ACL rules.
# I do not remember why I did it.
acl no_backup_proxy_acl dstdomain .google.com
# These ACLs are used to match requests to specific domains that are associated with the backup proxy.
# For example, the following ACL matches requests that are destined for 'api64.ipify.org'.
acl proxy_backup_domain_acl dstdomain api64.ipify.org
acl proxy_backup_domain_acl dstdomain .mradx.net
acl proxy_backup_domain_acl dstdomain .yastatic.net
acl proxy_backup_domain_acl dstdomain .yandex.net
# This line sets up a port for Squid to listen on.
http_port 10.66.66.5:14199 name=port_14199 tcpkeepalive=60,30,3
# This ACL is used to match requests that are destined for the port set up above.
# For example, the following ACL matches requests that are destined for port 14199.
acl port_14199_acl myportname port_14199
# This line allows Squid to handle requests that match the above ACL.
never_direct allow port_14199_acl
# This line sets up a cache peer with the name 'proxy14199'.
cache_peer proxy_ip_of_proxy_provider_1 parent 9599 0 connect-fail-limit=100 connect-timeout=10 no-tproxy no-query proxy-only no-digest no-netdb-exchange name=proxy14199 login=my_login_of_proxy_provider_1
# This line sets up another cache peer with the name 'proxy14199_backup'.
cache_peer proxy_ip_of_proxy_provider_2 parent 22225 0 connect-fail-limit=100 connect-timeout=10 no-tproxy no-query proxy-only no-digest no-netdb-exchange name=proxy14199_backup login=my_login_of_proxy_provider_2
# This rule allows traffic that matches the specified ACLs to access the cache peer named 'proxy14199'.
cache_peer_access proxy14199 allow port_14199_acl no_backup_proxy_acl
# This rule denies traffic that matches the specified ACLs from accessing the cache peer named 'proxy14199'.
cache_peer_access proxy14199 deny port_14199_acl proxy_backup_domain_acl
cache_peer_access proxy14199 allow port_14199_acl
cache_peer_access proxy14199 deny all
# This rule denies traffic that does not match the specified ACLs from accessing the cache peer named 'proxy14199_backup'.
cache_peer_access proxy14199_backup deny !port_14199_acl
cache_peer_access proxy14199_backup deny !proxy_backup_domain_acl
# This rule allows traffic that matches the specified ACLs to access the cache peer named 'proxy14199_backup'.
cache_peer_access proxy14199_backup allow port_14199_acl proxy_backup_domain_acl
# This rule denies all other traffic from accessing the cache peer named 'proxy14199_backup'.
# The 'deny all' directive at the end of this block sets the default behavior for requests that do not match the above rules.
cache_peer_access proxy14199_backup deny all
@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn Hello, I now try to work with your recommand way(squid), but meet one matter about forward https requests to specific proxy. I write one conf but can only forward http request. Pleasure if received from your reply.
Squid is overkill for many tasks because its ACL is not easy for everyone. The first time I had a task, I spent about a week making it work.
And please make sure to note that I don't know Squid very well and I am not a system administrator of Linux. I am just an ordinary user.
@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn Okay! thanks a lot!
I made another approach with HAProxy. HAProxy looks like a more stable solution and much simple.
Basic idea:
+-------------+
| Browser |
+-------------+
|
v
+-------------+
| Haproxy |
+-------------+
|
+-----------------------> Proxy Provider 1 (if not google.com)
|
+-----------------------> Proxy Provider 2 (if google.com)
######### Proxy Inbound Configuration #########
frontend http-in
bind *:4000
mode http
option http-use-htx
option http_proxy
# An ACL is defined for backup domains, which are the domains that will be routed to the second proxy provider.
acl backup_domains hdr_end(host) -i lumtest.com || -i .google.com
use_backend provider_1 if !backup_domains
use_backend provider_2 if backup_domains
######### Proxy Providers Configuration #########
backend provider_1
mode http
http-request del-header Proxy-Authorization
http-request set-header Proxy-Authorization "Basic BASE64_VALUE_1"
server proxy_provider_1 zproxy.lum-superproxy.io:22225 check
backend provider_2
mode http
http-request del-header Proxy-Authorization
http-request set-header Proxy-Authorization "Basic BASE64_VALUE_2"
server proxy_provider_2 zproxy.lum-superproxy.io:22225 check
@cturan said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn It's a very clever method, and the config file works great, thanks for the idea.
You need to know that in production usage, configuring HAProxy can be a more complex task because of authorization and many other factors
