Precaution 1: 3proxy was not initially developed for high load and is positioned as a SOHO product. The main reason is the "one connection - one thread" model 3proxy uses. 3proxy is known to work with over 200,000 connections under proper configuration, but use it in a production environment under high loads at your own risk and do not expect too much.
Precaution 2: This documentation is incomplete and insufficient. High loads may require very specific system tuning including, but not limited to, specific or customized kernels, builds, settings, sysctls, options, etc. All of this is not covered by this documentation.
maxconn 1000 proxy -p3129 proxy -p3128 socksmaxconn for every service is 1000, and there are 3 services running (2 proxy and 1 socks), so for all services there can be up to 3000 simultaneous connections to 3proxy.
Avoid setting 'maxconn' to an arbitrarily high value; it should be carefully chosen to protect the system and proxy from resource exhaustion. Setting maxconn above available resources can lead to denial of service conditions.
'maxconn' is not reduced automatically to fit the open file limit. If the limit is too low 3proxy only prints a warning at startup ("current open file ulimits are too low") and then fails to accept connections once the limit is reached, so check for this warning after changing 'maxconn'.
cat /proc/PID/limitswhere PID is the process ID. Validate that ulimits match your expectations, especially if you run 3proxy under a dedicated account by adding, e.g.:
system "ulimit -Ha >>/tmp/3proxy.ulim.hard" system "ulimit -Sa >>/tmp/3proxy.ulim.soft"at the beginning (before the first service is started) and at the end of the config file. Perform both a hard restart (i.e., kill and start the 3proxy process) and a soft restart by sending SIGUSR1 to the 3proxy process; check that the ulimits recorded to files match your expectations. In systemd-based distros (e.g., latest Debian/Ubuntu) changing limits.conf is not enough for a service: limits must be set in the unit file. Set them in the 3proxy unit itself rather than globally, so the rest of the system is unaffected. The shipped 3proxy.service already contains:
LimitNOFILE=1048576 LimitNPROC=infinity TasksMax=infinityTo change them on an installed system use an override instead of editing the unit:
systemctl edit 3proxy systemctl daemon-reload && systemctl restart 3proxy systemctl show 3proxy -p LimitNOFILE -p LimitNPROC -p TasksMaxTasksMax is the one that is easy to miss. It is the cgroup limit on the number of threads, and if it is not set the unit inherits DefaultTasksMax, which is 15% of kernel.threads-max (about 9000 on a typical host). Since 3proxy uses one thread per connection, that caps concurrent connections at that number regardless of LimitNPROC and maxconn, and the only symptom is "pthread_create()" errors in the log.
On systemd older than 227, which has no TasksMax, and for limits that must apply to several services, the same values can be set globally as DefaultLimitNOFILE / DefaultLimitNPROC in /etc/systemd/system.conf, but prefer the per-unit settings.
With SysV init the limits are not applied by limits.conf either, because start-stop-daemon does not open a PAM session, so the daemon simply inherits the limits of init. The shipped init script raises them itself before starting 3proxy:
ulimit -n 65536 ulimit -u 32768adjust these values in the script to match 'maxconn'.
On FreeBSD rc.subr applies limits(1) with the login class of the service (the "daemon" class by default), so the limits can be set either in /etc/login.conf for that class, or per service in rc.conf:
3proxy_limits="-n 65536"
To help with socket-based system-dependent settings, since 0.9-devel, 3proxy supports different socket options which can be set via the -ol option for the listening socket, -oc for the proxy-to-client socket, and -os for the proxy-to-server socket. Example:
proxy -olSO_REUSEADDR,SO_REUSEPORT -ocTCP_TIMESTAMPS,TCP_NODELAY -osTCP_NODELAYAvailable options are system-dependent.
File descriptors. 3proxy needs 2 descriptors per connection (4 for FTP), plus one per service, plus temporary ones for name resolution and RADIUS.
fs.nr_open = 1048576 # (1048576) upper bound for any process' RLIMIT_NOFILEulimit -n (RLIMIT_NOFILE) is the limit that actually applies and is commonly left at 1024; it must be raised for the 3proxy process itself, see "Setting ulimits" above. fs.file-max is effectively unlimited on 64-bit kernels and rarely needs changing.
Threads. Because of the "one connection - one thread" model these limits are reached earlier with 3proxy than with event-driven servers. Each thread also consumes one or two mappings, so vm.max_map_count matters too.
kernel.threads-max = 200000 # (~60000 on a 16G host, scales with RAM) kernel.pid_max = 4194304 # (4194304) vm.max_map_count = 1048576 # (1048576)RLIMIT_NPROC (ulimit -u) limits threads per user and must be raised as well. Check the actual thread count with grep Threads /proc/PID/status.
Listen queue. 3proxy uses a listen backlog of 1+(maxconn/8) unless the 'backlog' command is given, so a large 'maxconn' does not automatically give a large queue, and the kernel caps it at somaxconn:
net.core.somaxconn = 4096 # (4096) net.ipv4.tcp_max_syn_backlog = 4096 # (512) raise for bursty connection rates net.ipv4.tcp_syncookies = 1 # (1) keep enabled
Ephemeral ports and TIME_WAIT. See "Extending the Ephemeral Port Range" above for the multi-IP case. The range gives about 28000 outgoing connections per destination address by default:
net.ipv4.ip_local_port_range = 10240 65535 # (32768 60999) net.ipv4.tcp_tw_reuse = 2 # (2) reuse TIME_WAIT for outgoing connections net.ipv4.tcp_fin_timeout = 30 # (60)Do not enable tcp_tw_recycle; it was removed in kernel 4.12 and breaks NAT clients.
Socket buffers. Autotuning is usually right. Buffer memory is per connection, so raising the maximums with tens of thousands of connections costs a lot of RAM:
net.core.rmem_max = 4194304 # (212992) net.core.wmem_max = 4194304 # (212992) net.ipv4.tcp_rmem = 4096 131072 6291456 # (same) min default max net.ipv4.tcp_wmem = 4096 16384 4194304 # (same)Raise these only for high bandwidth-delay product links, and prefer raising the third (max) value and leaving the default alone.
Conntrack. Only relevant if netfilter/nftables tracks the proxy's traffic. If it does, the table is exhausted long before 3proxy's own limits, with "nf_conntrack: table full, dropping packet" in dmesg:
net.netfilter.nf_conntrack_max = 1048576 net.netfilter.nf_conntrack_buckets = 262144 net.netfilter.nf_conntrack_tcp_timeout_established = 3600 # (432000, i.e. 5 days) net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30 # (120)nf_conntrack_max defaults to nf_conntrack_buckets, which itself is derived from the amount of RAM, so it is often much lower than expected on small machines. Each connection takes two entries (one per direction). The default established timeout of 5 days matters more than the table size with high connection churn: entries for connections that are long gone keep occupying the table.
If no rules need conntrack, not loading it at all is faster: the modules are loaded on demand by the first rule that needs them ("-m state", "-m conntrack", any NAT rule), so a ruleset without such rules keeps the proxy traffic untracked. If conntrack is needed for other traffic but not for the proxy's, exempt the proxy's traffic explicitly in the raw table:
iptables -t raw -A PREROUTING -p tcp --dport 3128 -j CT --notrack iptables -t raw -A OUTPUT -p tcp -m owner --uid-owner proxy -j CT --notrack
Conntrack helpers (ALGs). The helper modules - nf_conntrack_ftp, nf_conntrack_sip, nf_conntrack_h323, nf_conntrack_pptp, nf_conntrack_irc, nf_conntrack_tftp - inspect the payload of every matching packet and create additional "expectation" entries, so they cost both CPU and table space, and they have a long history of security issues. Unload and blacklist the ones you do not actually need:
lsmod | grep nf_conntrack modprobe -r nf_conntrack_sip nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_pptp echo "blacklist nf_conntrack_sip" >> /etc/modprobe.d/no-alg.confOn current kernels a helper only acts when it is attached explicitly ("-j CT --helper ftp"), so simply not attaching it is enough; automatic helper assignment was deprecated and later removed. Older kernels, and most router firmware, still enable them by default.
Checking the result. ss -s for socket state totals, ss -lnt for listen queue overflow, nstat -az TcpExtListenOverflows TcpExtListenDrops for accept queue drops, and cat /proc/PID/limits for the limits actually applied to the running process.
Dynamic (ephemeral) port range. Since Windows Vista / Server 2008 the default range is 49152-65535, i.e. only 16384 outgoing connections per local address, which is reached quickly by a busy proxy. Show and change it with:
netsh int ipv4 show dynamicport tcp netsh int ipv4 set dynamicport tcp start=10000 num=55535The minimum start port is 1025, the minimum size of the range is 255, and the end of the range cannot exceed 65535. The range is set separately for TCP and UDP, and for IPv4 and IPv6. On pre-Vista systems the equivalent is the MaxUserPort registry value in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
Listening socket. 3proxy sets SO_REUSEADDR on the listening socket by default on Unix, but not on Windows: there it is not needed to rebind the port, and it only allows another local process to bind the same address and port, with undefined behaviour as to which of them receives the connections. If the machine is shared or untrusted, harden the listening socket instead:
proxy -olSO_EXCLUSIVEADDRUSENote that a socket with SO_EXCLUSIVEADDRUSE may not be immediately rebindable after a restart if accepted connections are still active, so test restarts before using it.
Port reuse. 3proxy always binds the outgoing socket before connecting, so Windows does not apply its automatic ephemeral port reuse (which it does only for connections with an implicit bind). Setting the option explicitly on the proxy-to-server socket therefore helps against port exhaustion:
proxy -osSO_REUSE_UNICASTPORTSO_REUSE_UNICASTPORT requires Windows 10 / Server 2019 or later. On older systems (Windows 7 / Server 2008 and later) use SO_PORT_SCALABILITY instead; where both are available Microsoft recommends SO_REUSE_UNICASTPORT. Note that SO_REUSEADDR has different, weaker semantics on Windows than on Unix and allows another socket to bind the same address and port, so do not use it on the listening socket as a substitute.
TIME_WAIT. Closed connections hold their port for the TcpTimedWaitDelay period, set in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters (DWORD, seconds). The effective default differs between Windows versions (2 to 4 minutes); check the current behaviour before changing it, and lower it only together with an extended port range. Count the connections in that state with:
netstat -ano -p tcp | find /c "TIME_WAIT"
Threads and address space. Windows has no ulimit equivalent, and the handle count is not normally the limit. On 32-bit builds the 2 GB of user address space is: each connection thread reserves its stack there, so a few thousand connections can exhaust the address space while physical memory is still free. Use a 64-bit build for high load, and see "Setting Stack Size" above.
Filter drivers. Antivirus, endpoint protection and other LSP/WFP filter drivers inspect every connection and are frequently the actual bottleneck on Windows, costing far more than any tuning above can recover. Exclude the 3proxy process and its ports, or test with the protection temporarily disabled to see the difference before tuning anything else.
radius secret 1.2.3.4 auth radius proxyor by using multiple services with different external interfaces, for example:
allow user1,user11,user111 proxy -p1111 -e1.1.1.1 flush allow user2,user22,user222 proxy -p2222 -e2.2.2.2 flush allow user3,user33,user333 proxy -p3333 -e3.3.3.3 flush allow user4,user44,user444 proxy -p4444 -e4.4.4.4 flushor via "parent extip" rotation, e.g.:
allow user1,user11,user111 parent 1000 extip 1.1.1.1 0 allow user2,user22,user222 parent 1000 extip 2.2.2.2 0 allow user3,user33,user333 parent 1000 extip 3.3.3.3 0 allow user4,user44,user444 parent 1000 extip 4.4.4.4 0 proxyor
allow * parent 250 extip 1.1.1.1 0 parent 250 extip 2.2.2.2 0 parent 250 extip 3.3.3.3 0 parent 250 extip 4.4.4.4 0 socks
Under the latest Linux versions, you can also start multiple services with different external addresses on a single port with SO_REUSEPORT on the listening socket to evenly distribute incoming connections between outgoing interfaces:
socks -olSO_REUSEPORT -p3128 -e1.1.1.1 socks -olSO_REUSEPORT -p3128 -e2.2.2.2 socks -olSO_REUSEPORT -p3128 -e3.3.3.3 socks -olSO_REUSEPORT -p3128 -e4.4.4.4For web browsing, the last two examples are not recommended because the same client can get a different external address for different requests; you should choose the external interface with user-based rules instead.
proxy -p3128 -e1.2.3.4 -osSO_REUSEADDRThe behavior for SO_REUSEADDR and SO_REUSEPORT is different between different systems, even between different kernel versions, and can lead to unexpected results. The specifics are described here. Use these options only if actually required and if you fully understand the possible consequences. For example, SO_REUSEPORT can help establish more connections than the number of client ports available, but it can also lead to situations where connections randomly fail due to IP+port pair collisions if the remote or local system doesn't support this trick.
On a Linux based router the same knobs apply and have to be raised there as well: nf_conntrack_max / nf_conntrack_buckets and the conntrack timeouts (see "Linux Tuning Hints" above), plus the port range used for translation, which is ip_local_port_range for MASQUERADE, or the explicit range if SNAT is configured with --to-ports. Note that the range is per translated address: with a single public IP, all clients share it.
Entry level and SOHO routers are the usual bottleneck here. They typically have a small fixed NAT/conntrack table (a few thousand entries), aggressive or non-adjustable timeouts, and no way to change either. Symptoms are seen on the proxy but caused by the router: connections that fail or hang at random under load while the proxy is far from its own limits, no error in the 3proxy log except a failed outgoing connect, and recovery after a pause or a router reboot. Before tuning 3proxy further, check the router's session/NAT table counters. For high load either give the proxy a public address without NAT in the path, or use a router where the table size and timeouts are configurable.
On the router, also turn off the application layer gateways that are not actually used - they usually appear in the web interface as "SIP ALG", "FTP ALG", "H.323 ALG", "PPTP passthrough", "IPsec/VPN passthrough". They are commonly enabled by default, they parse the payload of matching connections, and they consume additional session table entries for the connections they expect. If nothing behind the proxy uses FTP, VoIP or those VPN protocols, disabling them frees table space and CPU on exactly the device that is the bottleneck.
For 32-bit systems, address space can be a bottleneck you should consider. If you're short on address space, you can try using a negative stack size. The result is never lowered below the system minimum (PTHREAD_STACK_MIN), so a large negative value can not disable the thread stack. The base value the 'stacksize' is added to is 48K (64K on FreeBSD/NetBSD/OpenBSD/DragonFly, where libc uses more stack, e.g. in vfprintf() called by syslog()).
system "rm /some/path/3proxy/3proxy.lck"at the end of the config file to remove it after the configuration is successfully loaded
monitor "/some/path/3proxy/3proxy.ver"
proxy -osTCP_NODELAY -ocTCP_NODELAYsets TCP_NODELAY for client (oc) and server (os) connections.
Do not use TCP_NODELAY on slow connections with high delays when connection bandwidth is a bottleneck.
proxy -g8000,3,10The first parameter is the average read size we want to keep, the second parameter is the minimal number of packets in the same direction to apply the algorithm, and the last value is the delay added after polling and prior to reading data. The example above adds a 10-millisecond delay before reading data if the average polling size is below 8000 bytes and 3 read operations have been made in the same direction.
logdump 1 1is useful to see how grace delays work; choose a delay value to avoid filling the read buffer (typically 64K) but keep the request sizes close to the chosen average on large file uploads/downloads.