Best method(s) to minimize/compact traffic consumption of project

[sergerdn]

@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:

@sergerdn Thank you for sharing. Can I get a copy of your config for reference? Thank you very much

I'm not sure if it will be of much help to you. Please note that I only copied and pasted a small part of the project, and you'll need some more information to make it work.

I apologize for any incorrect comments I may have provided as I do not remember specifically which line they referred to.

Here is snippet:

squid.conf:

# This ACL is used to exempt requests to any backup proxies from the other ACL rules.
# I do not remember why I did it.
acl no_backup_proxy_acl dstdomain .google.com

# These ACLs are used to match requests to specific domains that are associated with the backup proxy.
# For example, the following ACL matches requests that are destined for 'api64.ipify.org'.
acl proxy_backup_domain_acl dstdomain api64.ipify.org
acl proxy_backup_domain_acl dstdomain .mradx.net
acl proxy_backup_domain_acl dstdomain .yastatic.net
acl proxy_backup_domain_acl dstdomain .yandex.net

# This line sets up a port for Squid to listen on.
http_port 10.66.66.5:14199 name=port_14199 tcpkeepalive=60,30,3

# This ACL is used to match requests that are destined for the port set up above.
# For example, the following ACL matches requests that are destined for port 14199.
acl port_14199_acl myportname port_14199

# This line allows Squid to handle requests that match the above ACL.
never_direct allow port_14199_acl

# This line sets up a cache peer with the name 'proxy14199'.
cache_peer proxy_ip_of_proxy_provider_1 parent 9599 0 connect-fail-limit=100 connect-timeout=10 no-tproxy no-query proxy-only no-digest no-netdb-exchange name=proxy14199 login=my_login_of_proxy_provider_1

# This line sets up another cache peer with the name 'proxy14199_backup'.
cache_peer proxy_ip_of_proxy_provider_2 parent 22225 0 connect-fail-limit=100 connect-timeout=10 no-tproxy no-query proxy-only no-digest no-netdb-exchange name=proxy14199_backup login=my_login_of_proxy_provider_2


# This rule allows traffic that matches the specified ACLs to access the cache peer named 'proxy14199'.
cache_peer_access proxy14199 allow port_14199_acl no_backup_proxy_acl 

# This rule denies traffic that matches the specified ACLs from accessing the cache peer named 'proxy14199'.
cache_peer_access proxy14199 deny port_14199_acl proxy_backup_domain_acl
cache_peer_access proxy14199 allow port_14199_acl
cache_peer_access proxy14199 deny all

# This rule denies traffic that does not match the specified ACLs from accessing the cache peer named 'proxy14199_backup'.
cache_peer_access proxy14199_backup deny !port_14199_acl
cache_peer_access proxy14199_backup deny !proxy_backup_domain_acl

# This rule allows traffic that matches the specified ACLs to access the cache peer named 'proxy14199_backup'.
cache_peer_access proxy14199_backup allow port_14199_acl proxy_backup_domain_acl

# This rule denies all other traffic from accessing the cache peer named 'proxy14199_backup'.
# The 'deny all' directive at the end of this block sets the default behavior for requests that do not match the above rules.
cache_peer_access proxy14199_backup deny all

[Xxxxxx]

@sergerdn Thanks!!! It help me avoid a lot trouble!

[sergerdn]

@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:

@sergerdn Thanks!!! It help me avoid a lot trouble!

You are welcome. I understand that using Squid ACL rules can be a challenge for everyone.

[Xxxxxx]

@sergerdn Hello, I now try to work with your recommand way(squid), but meet one matter about forward https requests to specific proxy. I write one conf but can only forward http request. Pleasure if received from your reply.

[sergerdn]

@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:

@sergerdn Hello, I now try to work with your recommand way(squid), but meet one matter about forward https requests to specific proxy. I write one conf but can only forward http request. Pleasure if received from your reply.

Squid is overkill for many tasks because its ACL is not easy for everyone. The first time I had a task, I spent about a week making it work.

And please make sure to note that I don't know Squid very well and I am not a system administrator of Linux. I am just an ordinary user.

[Xxxxxx]

@sergerdn Okay! thanks a lot!

[sergerdn]

@Xxxxxx said in Best method(s) to minimize/compact traffic consumption of project:

@sergerdn Okay! thanks a lot!

I made another approach with HAProxy. HAProxy looks like a more stable solution and much simple.

Basic idea:

+-------------+
|  Browser    |
+-------------+
       |
       v
+-------------+
|  Haproxy    |
+-------------+
       |
       +-----------------------> Proxy Provider 1 (if not google.com)
       |
       +-----------------------> Proxy Provider 2 (if google.com)

######### Proxy Inbound Configuration #########
frontend http-in
    bind *:4000
    mode http
    option http-use-htx
    option http_proxy
    
    # An ACL is defined for backup domains, which are the domains that will be routed to the second proxy provider.
    acl backup_domains hdr_end(host) -i lumtest.com || -i .google.com

    use_backend provider_1 if !backup_domains
    use_backend provider_2 if backup_domains

######### Proxy Providers Configuration #########

backend provider_1
    mode http
    http-request del-header Proxy-Authorization
    http-request set-header Proxy-Authorization "Basic BASE64_VALUE_1"
    server proxy_provider_1 zproxy.lum-superproxy.io:22225 check

backend provider_2
    mode http
    http-request del-header Proxy-Authorization
    http-request set-header Proxy-Authorization "Basic BASE64_VALUE_2"
    server proxy_provider_2 zproxy.lum-superproxy.io:22225 check    

[DuckDuck]

This post is deleted!

[cturan]

@sergerdn It's a very clever method, and the config file works great, thanks for the idea.

[sergerdn]

@cturan said in Best method(s) to minimize/compact traffic consumption of project:

@sergerdn It's a very clever method, and the config file works great, thanks for the idea.

You need to know that in production usage, configuring HAProxy can be a more complex task because of authorization and many other factors

[cturan]

@sergerdn said in Best method(s) to minimize/compact traffic consumption of project:

@cturan said in Best method(s) to minimize/compact traffic consumption of project:

@sergerdn It's a very clever method, and the config file works great, thanks for the idea.

You need to know that in production usage, configuring HAProxy can be a more complex task because of authorization and many other factors

Yes, for security I just gave access to our vpn server only.

     # allow only our server
     acl network_allowed src {vpn_ip}
     tcp-request connection reject if !network_allowed

It doesn't work on every site, but since the site I need uses cdn with different domain, I redirected all cdn traffic to a cheap proxy, it worked very well.