@morpheus93 said in Best method(s) to minimize/compact traffic consumption of project:
Your approach using a centralized re-direction server sounds very sophisticated.
Thank you for your feedback on my previous approach. I agree that using a main-in-the-middle proxy server can be a more efficient solution. By centralizing the redirecting process in a proxy server, you can avoid making changes to each individual BAS script and instead make updates to the proxy server configuration. This can save time and effort in managing and maintaining the redirection process.
In addition, using a proxy server can provide additional security features such as filtering. For example, you may want to block some network requests to certain domains, which can help improve both traffic spending and security.
But of course, you can implement both a main-in-the-middle proxy and prepare your profiles with cheap proxies to fill up the local cache. This can help improve performance and further optimize your traffic flow.
Overall, a main-in-the-middle proxy server combined with a cheap proxy cache can be a powerful solution for managing and optimizing traffic.
@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn Thank you for sharing. Can I get a copy of your config for reference? Thank you very much
I'm not sure if it will be of much help to you. Please note that I only copied and pasted a small part of the project, and you'll need some more information to make it work.
I apologize for any incorrect comments I may have provided as I do not remember specifically which line they referred to.
Here is snippet:
squid.conf:
# This ACL is used to exempt requests to any backup proxies from the other ACL rules.
# I do not remember why I did it.
acl no_backup_proxy_acl dstdomain .google.com
# These ACLs are used to match requests to specific domains that are associated with the backup proxy.
# For example, the following ACL matches requests that are destined for 'api64.ipify.org'.
acl proxy_backup_domain_acl dstdomain api64.ipify.org
acl proxy_backup_domain_acl dstdomain .mradx.net
acl proxy_backup_domain_acl dstdomain .yastatic.net
acl proxy_backup_domain_acl dstdomain .yandex.net
# This line sets up a port for Squid to listen on.
http_port 10.66.66.5:14199 name=port_14199 tcpkeepalive=60,30,3
# This ACL is used to match requests that are destined for the port set up above.
# For example, the following ACL matches requests that are destined for port 14199.
acl port_14199_acl myportname port_14199
# This line allows Squid to handle requests that match the above ACL.
never_direct allow port_14199_acl
# This line sets up a cache peer with the name 'proxy14199'.
cache_peer proxy_ip_of_proxy_provider_1 parent 9599 0 connect-fail-limit=100 connect-timeout=10 no-tproxy no-query proxy-only no-digest no-netdb-exchange name=proxy14199 login=my_login_of_proxy_provider_1
# This line sets up another cache peer with the name 'proxy14199_backup'.
cache_peer proxy_ip_of_proxy_provider_2 parent 22225 0 connect-fail-limit=100 connect-timeout=10 no-tproxy no-query proxy-only no-digest no-netdb-exchange name=proxy14199_backup login=my_login_of_proxy_provider_2
# This rule allows traffic that matches the specified ACLs to access the cache peer named 'proxy14199'.
cache_peer_access proxy14199 allow port_14199_acl no_backup_proxy_acl
# This rule denies traffic that matches the specified ACLs from accessing the cache peer named 'proxy14199'.
cache_peer_access proxy14199 deny port_14199_acl proxy_backup_domain_acl
cache_peer_access proxy14199 allow port_14199_acl
cache_peer_access proxy14199 deny all
# This rule denies traffic that does not match the specified ACLs from accessing the cache peer named 'proxy14199_backup'.
cache_peer_access proxy14199_backup deny !port_14199_acl
cache_peer_access proxy14199_backup deny !proxy_backup_domain_acl
# This rule allows traffic that matches the specified ACLs to access the cache peer named 'proxy14199_backup'.
cache_peer_access proxy14199_backup allow port_14199_acl proxy_backup_domain_acl
# This rule denies all other traffic from accessing the cache peer named 'proxy14199_backup'.
# The 'deny all' directive at the end of this block sets the default behavior for requests that do not match the above rules.
cache_peer_access proxy14199_backup deny all
@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn Hello, I now try to work with your recommand way(squid), but meet one matter about forward https requests to specific proxy. I write one conf but can only forward http request. Pleasure if received from your reply.
Squid is overkill for many tasks because its ACL is not easy for everyone. The first time I had a task, I spent about a week making it work.
And please make sure to note that I don't know Squid very well and I am not a system administrator of Linux. I am just an ordinary user.
@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn Okay! thanks a lot!
I made another approach with HAProxy. HAProxy looks like a more stable solution and much simple.
Basic idea:
+-------------+
| Browser |
+-------------+
|
v
+-------------+
| Haproxy |
+-------------+
|
+-----------------------> Proxy Provider 1 (if not google.com)
|
+-----------------------> Proxy Provider 2 (if google.com)
######### Proxy Inbound Configuration #########
frontend http-in
bind *:4000
mode http
option http-use-htx
option http_proxy
# An ACL is defined for backup domains, which are the domains that will be routed to the second proxy provider.
acl backup_domains hdr_end(host) -i lumtest.com || -i .google.com
use_backend provider_1 if !backup_domains
use_backend provider_2 if backup_domains
######### Proxy Providers Configuration #########
backend provider_1
mode http
http-request del-header Proxy-Authorization
http-request set-header Proxy-Authorization "Basic BASE64_VALUE_1"
server proxy_provider_1 zproxy.lum-superproxy.io:22225 check
backend provider_2
mode http
http-request del-header Proxy-Authorization
http-request set-header Proxy-Authorization "Basic BASE64_VALUE_2"
server proxy_provider_2 zproxy.lum-superproxy.io:22225 check
@cturan said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn It's a very clever method, and the config file works great, thanks for the idea.
You need to know that in production usage, configuring HAProxy can be a more complex task because of authorization and many other factors
@sergerdn said in Best method(s) to minimize/compact traffic consumption of project:
@cturan said in Best method(s) to minimize/compact traffic consumption of project:
@sergerdn It's a very clever method, and the config file works great, thanks for the idea.
You need to know that in production usage, configuring HAProxy can be a more complex task because of authorization and many other factors
Yes, for security I just gave access to our vpn server only.
# allow only our server
acl network_allowed src {vpn_ip}
tcp-request connection reject if !network_allowed
It doesn't work on every site, but since the site I need uses cdn with different domain, I redirected all cdn traffic to a cheap proxy, it worked very well.
