Precaution 1: 3proxy was not initially developed for high load and is positioned as a SOHO product. The main reason is the "one connection - one thread" model 3proxy uses. 3proxy is known to work with over 200,000 connections under proper configuration, but use it in a production environment under high loads at your own risk and do not expect too much.
Precaution 2: This documentation is incomplete and insufficient. High loads may require very specific system tuning including, but not limited to, specific or customized kernels, builds, settings, sysctls, options, etc. All of this is not covered by this documentation.
maxconn 1000 proxy -p3129 proxy -p3128 socksmaxconn for every service is 1000, and there are 3 services running (2 proxy and 1 socks), so for all services there can be up to 3000 simultaneous connections to 3proxy.
Avoid setting 'maxconn' to an arbitrarily high value; it should be carefully chosen to protect the system and proxy from resource exhaustion. Setting maxconn above available resources can lead to denial of service conditions.
'maxconn' is not reduced automatically to fit the open file limit. If the limit is too low 3proxy only prints a warning at startup ("current open file ulimits are too low") and then fails to accept connections once the limit is reached, so check for this warning after changing 'maxconn'.
cat /proc/PID/limitswhere PID is the process ID. Validate that ulimits match your expectations, especially if you run 3proxy under a dedicated account by adding, e.g.:
system "ulimit -Ha >>/tmp/3proxy.ulim.hard" system "ulimit -Sa >>/tmp/3proxy.ulim.soft"at the beginning (before the first service is started) and at the end of the config file. Perform both a hard restart (i.e., kill and start the 3proxy process) and a soft restart by sending SIGUSR1 to the 3proxy process; check that the ulimits recorded to files match your expectations. In systemd-based distros (e.g., latest Debian/Ubuntu) changing limits.conf is not enough for a service: limits must be set in the unit file. Set them in the 3proxy unit itself rather than globally, so the rest of the system is unaffected. The shipped 3proxy.service already contains:
LimitNOFILE=1048576 LimitNPROC=infinity TasksMax=infinityTo change them on an installed system use an override instead of editing the unit:
systemctl edit 3proxy systemctl daemon-reload && systemctl restart 3proxy systemctl show 3proxy -p LimitNOFILE -p LimitNPROC -p TasksMaxTasksMax is the one that is easy to miss. It is the cgroup limit on the number of threads, and if it is not set the unit inherits DefaultTasksMax, which is 15% of kernel.threads-max (about 9000 on a typical host). Since 3proxy uses one thread per connection, that caps concurrent connections at that number regardless of LimitNPROC and maxconn, and the only symptom is "pthread_create()" errors in the log.
On systemd older than 227, which has no TasksMax, and for limits that must apply to several services, the same values can be set globally as DefaultLimitNOFILE / DefaultLimitNPROC in /etc/systemd/system.conf, but prefer the per-unit settings.
With SysV init the limits are not applied by limits.conf either, because start-stop-daemon does not open a PAM session, so the daemon simply inherits the limits of init. The shipped init script raises them itself before starting 3proxy:
ulimit -n 65536 ulimit -u 32768adjust these values in the script to match 'maxconn'.
On FreeBSD rc.subr applies limits(1) with the login class of the service (the "daemon" class by default), so the limits can be set either in /etc/login.conf for that class, or per service in rc.conf:
3proxy_limits="-n 65536"
To help with socket-based system-dependent settings, since 0.9-devel, 3proxy supports different socket options which can be set via the -ol option for the listening socket, -oc for the proxy-to-client socket, and -os for the proxy-to-server socket. Example:
proxy -olSO_REUSEADDR,SO_REUSEPORT -ocTCP_TIMESTAMPS,TCP_NODELAY -osTCP_NODELAYAvailable options are system-dependent.
File descriptors. 3proxy needs 2 descriptors per connection (4 for FTP), plus one per service, plus temporary ones for name resolution and RADIUS.
fs.nr_open = 1048576 # (1048576) upper bound for any process' RLIMIT_NOFILEulimit -n (RLIMIT_NOFILE) is the limit that actually applies and is commonly left at 1024; it must be raised for the 3proxy process itself, see "Setting ulimits" above. fs.file-max is effectively unlimited on 64-bit kernels and rarely needs changing.
Threads. Because of the "one connection - one thread" model these limits are reached earlier with 3proxy than with event-driven servers. Each thread also consumes one or two mappings, so vm.max_map_count matters too.
kernel.threads-max = 200000 # (~60000 on a 16G host, scales with RAM) kernel.pid_max = 4194304 # (4194304) vm.max_map_count = 1048576 # (1048576)RLIMIT_NPROC (ulimit -u) limits threads per user and must be raised as well. Check the actual thread count with grep Threads /proc/PID/status.
Listen queue. 3proxy uses a listen backlog of 1+(maxconn/8) unless the 'backlog' command is given, so a large 'maxconn' does not automatically give a large queue, and the kernel caps it at somaxconn:
net.core.somaxconn = 4096 # (4096) net.ipv4.tcp_max_syn_backlog = 4096 # (512) raise for bursty connection rates net.ipv4.tcp_syncookies = 1 # (1) keep enabled
Ephemeral ports and TIME_WAIT. See "Extending the Ephemeral Port Range" above for the multi-IP case. The range gives about 28000 outgoing connections per destination address by default:
net.ipv4.ip_local_port_range = 10240 65535 # (32768 60999) net.ipv4.tcp_tw_reuse = 2 # (2) reuse TIME_WAIT for outgoing connections net.ipv4.tcp_fin_timeout = 30 # (60)Do not enable tcp_tw_recycle; it was removed in kernel 4.12 and breaks NAT clients.
Socket buffers. Autotuning is usually right. Buffer memory is per connection, so raising the maximums with tens of thousands of connections costs a lot of RAM:
net.core.rmem_max = 4194304 # (212992) net.core.wmem_max = 4194304 # (212992) net.ipv4.tcp_rmem = 4096 131072 6291456 # (same) min default max net.ipv4.tcp_wmem = 4096 16384 4194304 # (same)Raise these only for high bandwidth-delay product links, and prefer raising the third (max) value and leaving the default alone.
Conntrack. Only relevant if netfilter/nftables tracks the proxy's traffic. If it does, the table is exhausted long before 3proxy's own limits, with "nf_conntrack: table full, dropping packet" in dmesg:
net.netfilter.nf_conntrack_max = 1048576 net.netfilter.nf_conntrack_buckets = 262144 net.netfilter.nf_conntrack_tcp_timeout_established = 3600 # (432000, i.e. 5 days) net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30 # (120)nf_conntrack_max defaults to nf_conntrack_buckets, which itself is derived from the amount of RAM, so it is often much lower than expected on small machines. Each connection takes two entries (one per direction). The default established timeout of 5 days matters more than the table size with high connection churn: entries for connections that are long gone keep occupying the table.
If no rules need conntrack, not loading it at all is faster: the modules are loaded on demand by the first rule that needs them ("-m state", "-m conntrack", any NAT rule), so a ruleset without such rules keeps the proxy traffic untracked. If conntrack is needed for other traffic but not for the proxy's, exempt the proxy's traffic explicitly in the raw table:
iptables -t raw -A PREROUTING -p tcp --dport 3128 -j CT --notrack iptables -t raw -A OUTPUT -p tcp -m owner --uid-owner proxy -j CT --notrack
Conntrack helpers (ALGs). The helper modules - nf_conntrack_ftp, nf_conntrack_sip, nf_conntrack_h323, nf_conntrack_pptp, nf_conntrack_irc, nf_conntrack_tftp - inspect the payload of every matching packet and create additional "expectation" entries, so they cost both CPU and table space, and they have a long history of security issues. Unload and blacklist the ones you do not actually need:
lsmod | grep nf_conntrack modprobe -r nf_conntrack_sip nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_pptp echo "blacklist nf_conntrack_sip" >> /etc/modprobe.d/no-alg.confOn current kernels a helper only acts when it is attached explicitly ("-j CT --helper ftp"), so simply not attaching it is enough; automatic helper assignment was deprecated and later removed. Older kernels, and most router firmware, still enable them by default.
Checking the result. ss -s for socket state totals, ss -lnt for listen queue overflow, nstat -az TcpExtListenOverflows TcpExtListenDrops for accept queue drops, and cat /proc/PID/limits for the limits actually applied to the running process.
Dynamic (ephemeral) port range. Since Windows Vista / Server 2008 the default range is 49152-65535, i.e. only 16384 outgoing connections per local address, which is reached quickly by a busy proxy. Show and change it with:
netsh int ipv4 show dynamicport tcp netsh int ipv4 set dynamicport tcp start=10000 num=55535The minimum start port is 1025, the minimum size of the range is 255, and the end of the range cannot exceed 65535. The range is set separately for TCP and UDP, and for IPv4 and IPv6. On pre-Vista systems the equivalent is the MaxUserPort registry value in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
Listening socket. 3proxy sets SO_REUSEADDR on the listening socket by default on Unix, but not on Windows: there it is not needed to rebind the port, and it only allows another local process to bind the same address and port, with undefined behaviour as to which of them receives the connections. If the machine is shared or untrusted, harden the listening socket instead:
proxy -olSO_EXCLUSIVEADDRUSENote that a socket with SO_EXCLUSIVEADDRUSE may not be immediately rebindable after a restart if accepted connections are still active, so test restarts before using it.
Port reuse. 3proxy always binds the outgoing socket before connecting, so Windows does not apply its automatic ephemeral port reuse (which it does only for connections with an implicit bind). Setting the option explicitly on the proxy-to-server socket therefore helps against port exhaustion:
proxy -osSO_REUSE_UNICASTPORTSO_REUSE_UNICASTPORT requires Windows 10 / Server 2019 or later. On older systems (Windows 7 / Server 2008 and later) use SO_PORT_SCALABILITY instead; where both are available Microsoft recommends SO_REUSE_UNICASTPORT. Note that SO_REUSEADDR has different, weaker semantics on Windows than on Unix and allows another socket to bind the same address and port, so do not use it on the listening socket as a substitute.
TIME_WAIT. Closed connections hold their port for the TcpTimedWaitDelay period, set in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters (DWORD, seconds). The effective default differs between Windows versions (2 to 4 minutes); check the current behaviour before changing it, and lower it only together with an extended port range. Count the connections in that state with:
netstat -ano -p tcp | find /c "TIME_WAIT"
Threads and address space. Windows has no ulimit equivalent, and the handle count is not normally the limit. On 32-bit builds the 2 GB of user address space is: each connection thread reserves its stack there, so a few thousand connections can exhaust the address space while physical memory is still free. Use a 64-bit build for high load, and see "Setting Stack Size" above.
Filter drivers. Antivirus, endpoint protection and other LSP/WFP filter drivers inspect every connection and are frequently the actual bottleneck on Windows, costing far more than any tuning above can recover. Exclude the 3proxy process and its ports, or test with the protection temporarily disabled to see the difference before tuning anything else.
radius secret 1.2.3.4 auth radius proxyor by using multiple services with different external interfaces, for example:
allow user1,user11,user111 proxy -p1111 -e1.1.1.1 flush allow user2,user22,user222 proxy -p2222 -e2.2.2.2 flush allow user3,user33,user333 proxy -p3333 -e3.3.3.3 flush allow user4,user44,user444 proxy -p4444 -e4.4.4.4 flushor via "parent extip" rotation, e.g.:
allow user1,user11,user111 parent 1000 extip 1.1.1.1 0 allow user2,user22,user222 parent 1000 extip 2.2.2.2 0 allow user3,user33,user333 parent 1000 extip 3.3.3.3 0 allow user4,user44,user444 parent 1000 extip 4.4.4.4 0 proxyor
allow * parent 250 extip 1.1.1.1 0 parent 250 extip 2.2.2.2 0 parent 250 extip 3.3.3.3 0 parent 250 extip 4.4.4.4 0 socks
Under the latest Linux versions, you can also start multiple services with different external addresses on a single port with SO_REUSEPORT on the listening socket to evenly distribute incoming connections between outgoing interfaces:
socks -olSO_REUSEPORT -p3128 -e1.1.1.1 socks -olSO_REUSEPORT -p3128 -e2.2.2.2 socks -olSO_REUSEPORT -p3128 -e3.3.3.3 socks -olSO_REUSEPORT -p3128 -e4.4.4.4For web browsing, the last two examples are not recommended because the same client can get a different external address for different requests; you should choose the external interface with user-based rules instead.
proxy -p3128 -e1.2.3.4 -osSO_REUSEADDRThe behavior for SO_REUSEADDR and SO_REUSEPORT is different between different systems, even between different kernel versions, and can lead to unexpected results. The specifics are described here. Use these options only if actually required and if you fully understand the possible consequences. For example, SO_REUSEPORT can help establish more connections than the number of client ports available, but it can also lead to situations where connections randomly fail due to IP+port pair collisions if the remote or local system doesn't support this trick.
On a Linux based router the same knobs apply and have to be raised there as well: nf_conntrack_max / nf_conntrack_buckets and the conntrack timeouts (see "Linux Tuning Hints" above), plus the port range used for translation, which is ip_local_port_range for MASQUERADE, or the explicit range if SNAT is configured with --to-ports. Note that the range is per translated address: with a single public IP, all clients share it.
Entry level and SOHO routers are the usual bottleneck here. They typically have a small fixed NAT/conntrack table (a few thousand entries), aggressive or non-adjustable timeouts, and no way to change either. Symptoms are seen on the proxy but caused by the router: connections that fail or hang at random under load while the proxy is far from its own limits, no error in the 3proxy log except a failed outgoing connect, and recovery after a pause or a router reboot. Before tuning 3proxy further, check the router's session/NAT table counters. For high load either give the proxy a public address without NAT in the path, or use a router where the table size and timeouts are configurable.
On the router, also turn off the application layer gateways that are not actually used - they usually appear in the web interface as "SIP ALG", "FTP ALG", "H.323 ALG", "PPTP passthrough", "IPsec/VPN passthrough". They are commonly enabled by default, they parse the payload of matching connections, and they consume additional session table entries for the connections they expect. If nothing behind the proxy uses FTP, VoIP or those VPN protocols, disabling them frees table space and CPU on exactly the device that is the bottleneck.
For 32-bit systems, address space can be a bottleneck you should consider. If you're short on address space, you can try using a negative stack size. The result is never lowered below the system minimum (PTHREAD_STACK_MIN), so a large negative value can not disable the thread stack. The base value the 'stacksize' is added to is 48K (64K on FreeBSD/NetBSD/OpenBSD/DragonFly, where libc uses more stack, e.g. in vfprintf() called by syslog()).
authcache <cachetype> <cachetime> <cachesize>'cachetype' lists the fields the entry is keyed by, 'cachetime' is the lifetime in seconds, and the optional 'cachesize' limits the number of entries. The cached type must be placed before the real authentication type in the 'auth' command: a cache miss falls through to the next type in the chain, and a hit skips it.
There are two cached types:
'cacheacl' does not work with parent proxies. Redirections are applied while the ACL is walked, so on a 'cacheacl' hit no redirection happens at all: the request is sent directly instead of through the 'parent' proxy of the matching ACL entry, and nothing is reported. The first request for a given key uses the parent and the rest do not, which makes it look intermittent. Use 'cache' with parent proxies:
authcache user,pass 60 auth cache strong allow user1 parent 1000 socks5 10.0.0.1 1080 proxyThe one exception is 'parent ... extip': the external address is stored in the cache entry and restored on a hit, but only if 'ext' is a part of the key:
authcache user,pass,ext 60 auth cacheacl strong allow user1 parent 1000 extip 1.2.3.4 0 proxyWithout 'ext' in the key the external address falls back to the default one on every cache hit.
The key must match the ACL. On a 'cacheacl' hit the cached result is reused as is, so any field the ACL limits but the key does not contain makes the cached result apply to a request it was not computed for. Use 'user' and 'pass' for the userlist, 'ip' for the sourcelist, 'dstaddr' and 'dsthost' for the targetlist, 'dstport' for the targetportlist and 'dstoper' for the operationlist. For this ACL:
allow user1,user2 * * 80,443 deny *the key needs 'user', 'pass' and 'dstport':
users $/etc/3proxy/passwd authcache user,pass,dstport 60 65536 auth cacheacl strong allow user1,user2 * * 80,443 deny * proxy
With RADIUS the round trip is the expensive part, so cache by whatever identifies the client:
radius secret 1.2.3.4 authcache ip,user,pass 600 auth cache radius proxy'cache' rather than 'cacheacl' is used here because RADIUS may return per-user parameters such as the outgoing IP; add 'ext' to the key to cache the external address returned by RADIUS as well.
Choose 'cachetime' to balance the load against how quickly a password or ACL change has to take effect. Cache entries are not invalidated by a configuration reload, so a long lifetime keeps applying the old decision to existing keys after the new configuration is loaded.
Note that 'ip' alone, or 'ip,user' without 'pass', assigns every connection from the same address to the same user for the lifetime of the entry without checking the password. That is a deliberate trade-off for high load; do not use it where clients share addresses.
socks -U3 # default: log and authorize every new destination socks -U2 # authorize, do not log socks -U1 # log, do not authorize socks -U0 # neither (same as bare -U)The destination of the first datagram is always authorized and logged, so the association as a whole is still subject to ACLs, authentication and the choice of the parent proxy.
Before reaching for the option, note that the cost is already avoided when no ACL entry reachable by the client limits the destination: in that case the result can not depend on the destination and no re-authorization is performed whatever '-U' says. '-U' only matters for configurations that do limit the destination.
For high load:
The ACL is the exception and is still processed linearly. Every request walks the access list entry by entry until one matches, and within an entry the userlist, the sourcelist, the targetlist and the targetportlist are walked linearly as well, so the cost is roughly the number of entries multiplied by the size of the lists inside them. The 'bandlimin'/'bandlimout' and 'countin'/'countout' rules are separate lists of the same kind and are walked on every request in addition to the access list.
Two things follow from this:
With a large ACL, cache the authorization result instead of shortening the list. 'cacheacl' walks the ACL once per cache entry rather than once per request:
users $/etc/3proxy/passwd authcache user,pass,dstaddr,dstport 60 65536 auth cacheacl strongBoth limitations from "Use the Authentication Cache" above apply: the cache key must contain every field the ACL limits, and 'cacheacl' must not be used together with parent proxies. Where a parent proxy is required, plain 'cache' still removes the authentication cost but leaves the ACL walk on every request.
RADIUS moves the user list and the ACLs out of 3proxy entirely, and also allows you to set an outgoing IP on a per-user basis or to implement more sophisticated logic. RADIUS is a new beta feature; test it before using it in production.
system "rm /some/path/3proxy/3proxy.lck"at the end of the config file to remove it after the configuration is successfully loaded
monitor "/some/path/3proxy/3proxy.ver"
proxy -osTCP_NODELAY=0 -ocTCP_NODELAY=0
proxy -s0 tcppm -s3 8080 192.168.1.1 80-s alone, the default, splices both directions where it pays: an HTTP body of known length if at least 16K of it is left to read, a stream (CONNECT, SOCKS, tcppm) once it carries large reads, going back to copying when they turn small. -s1 always splices what the server sends to the client (downloads), -s2 what the client sends to the server (uploads), -s3 both; -s5, -s6 and -s7 (the same as -s) do it for those directions only where it pays. -s0 turns it off. Where it pays can be tuned after the mode, as grace delay is, with :MIN,STREAK,SMALL:
proxy -s:16384,2,4096 tcppm -s6:65536,3,8192 8080 192.168.1.1 80A direction is spliced after STREAK reads in a row of MIN bytes or more (SMALL from a TLS socket, which hands out a record at a time), and copied again after more than STREAK splices in a row of less than SMALL; a body of known length is spliced if at least MIN of it is left to read. The first line is the default. A larger MIN or STREAK splices only longer transfers. In builds without it (CMake option 3PROXY_USE_SPLICE=OFF, or -DWITHSPLICE removed from the Makefile) -s is accepted and ignored.
splice() pays off when most of the traffic is large transfers. Measured in the benchmarks on an 8 CPU KVM virtual machine (Intel Xeon Skylake, 2.0 GHz, Linux 6.12), with 3proxy confined to one CPU and the client and the origin on CPUs and in network namespaces of their own, linked by veth pairs (MTU 1500), default 16K buffers, the automatic mode (-s) against copying (-s0):
| load | -s0 | -s | CPU per gigabyte |
|---|---|---|---|
| TCP streams through tcppm, CONNECT or SOCKS | 9.4-10.2 Gbit/s | 28-34 Gbit/s | 65-70% less |
| HTTP proxy, 1 MB answers on kept-alive connections | 9.4 Gbit/s | 25.3 Gbit/s | 62% less |
| 2000 clients, 50 KB answers: tcppm | 11.0k req/s | 18.1k req/s | 39% less |
| 2000 clients, 50 KB answers: CONNECT, SOCKS | 10.4-10.5k req/s | 15.8k req/s | 33-35% less |
| 2000 clients, 50 KB answers: HTTP proxy | 8.1k req/s | 9.6k req/s | 16% less |
| 2000 clients, 2 KB POST and 50 KB answer: tcppm | 10.5k req/s | 16.4k req/s | 36% less |
| 2000 clients, 2 KB POST and 50 KB answer: CONNECT, SOCKS | 9.8-10.1k req/s | 14.6-14.7k req/s | 31-34% less |
| 2000 clients, 2 KB POST and 50 KB answer: HTTP proxy | 7.5k req/s | 9.4k req/s | 20% less |
| 1K requests, kept alive or a connection each | within 5% either way: the automatic mode does not splice them | ||
The mix closer to a proxy giving clients access to the Internet: 20000 connections, 3000 exchanges a second, web-like answers (median 12K, up to 8M), as a TCP relay, an HTTP proxy and through CONNECT, on the same machine, with the clients' links emulated as in the grace delay tests below. 3proxy's own CPU time per exchange with -s against -s0 (and the CPU's busy time, the kernel's network work included):
| client link | tcppm | HTTP proxy | CONNECT |
|---|---|---|---|
| no limit, no delay | 8% less (4%) | 9% less (6%) | 12% less (7%) |
| 100 Mbit/s, 2 ms, 0.1% loss | 7% less (+1%) | 1% less (+3%) | 6% less (+1%) |
| 20 Mbit/s, 5 ms, 0.5% loss | +1% (+3%) | 12% less (5%) | +7% (+8%) |
| 10 Mbit/s, 10 ms, 0.5% loss (two CPUs) | – | – | 9% less (2%) |
| 5 Mbit/s, 25 ms, 1% loss | 8% less (0%) | 10% less (9%) | 6% less (2%) |
Conclusion: for web traffic -s, the default, and -s0 show no significant difference: most connections are slow and their exchanges short, splice() applies to few of them, and the difference stays within 0-12% of 3proxy's CPU at no cost in latency. -s increases the speed and decreases CPU usage for high-speed large uploads and downloads, 3 times the speed at 60-70% less CPU per gigabyte here, especially for traffic inside high-speed local networks and virtual environments, where nothing but the CPU limits a connection.
Every spliced direction needs a pipe (two file descriptors) and more kernel buffer memory. splice() is not used where 3proxy has to see the data: on a connection while data filters apply to it (an HTTP body is spliced only once the filters are done with it: they saw it whole, or it is larger than filtermaxsize, which they skip), and by a service that ends or begins TLS (ssl_serv, ssl_cli) only for connections the kernel does TLS for both ways (see Kernel TLS).
TCP_NODELAY and splice() do not conflict and can be combined on high-speed connections.
proxy -g8000,2,2-gTRAFF,NUM,DELAY: once NUM reads in a row in one direction, within the same second, have averaged TRAFF bytes or less, 3proxy waits DELAY milliseconds before it polls and reads again, so that data gathers in the socket and is read in fewer, larger pieces. The example waits 2 milliseconds after two reads in a row averaging 8000 bytes or less. The counts start again every second and whenever the other direction is read, and the wait stops as soon as reads grow larger than TRAFF on average.
Fewer reads mean fewer system calls, wakeups and context switches: it decreases CPU usage on connections which carry data in small pieces, such as bulk transfers to and from clients on slow links. The cost is the throughput of a single connection. While 3proxy waits, it reads at most a buffer's worth after each wait: 16K copying (with the default buffer size; less with -b), up to 64K (the pipe) with splice(). This caps a connection's direction at one buffer per DELAY:
| DELAY | copying, 16K per read | splice(), 64K per read |
|---|---|---|
| 1 ms | 131 Mbit/s | 524 Mbit/s |
| 2 ms | 65 Mbit/s | 262 Mbit/s |
| 3 ms | 44 Mbit/s | 175 Mbit/s |
| 5 ms | 26 Mbit/s | 105 Mbit/s |
| 10 ms | 13 Mbit/s | 52 Mbit/s |
Measured on the same Intel machine as splice() above, the clients' links emulated for each connection (rate limit, delay and loss each way, with fq and netem on the client's side), 16 streams through tcppm and a mix of 20000 connections at a fixed 3000 exchanges a second (web-like answers, median 12K), against 3proxy without grace delay:
| client link | setting | 16 streams: throughput | 16 streams: 3proxy CPU per gigabyte | mix, HTTP proxy: CPU busy |
|---|---|---|---|---|
| 100 Mbit/s, 2 ms, 0.1% loss | -g8000,2,2 | the same (100 Mbit/s each) | the same: reads are large, waits do not apply | the same (73%) |
| -g6000,3,2 | ||||
| -g8000,2,3 | ||||
| 20 Mbit/s, 5 ms, 0.5% loss | -g8000,2,2 | 3% less | 41% less | about the same: 88-96% with and without, across two sessions |
| -g6000,3,2 | 3% less | 35% less | ||
| -g8000,2,3 | 7% less | 46% less | ||
| 5 Mbit/s, 25 ms, 1% loss | -g8000,2,2 | the same (held by the loss) | 21-23% more | 117-124% against 109-111% (of 200%: two CPUs) |
| -g8000,2,10 | 5-8% less | 21-29% less | about the same: 112-117%, within the runs' spread (two CPUs) | |
| -g8000,2,20 | 12-17% less | 31-32% less | about the same: 113% (two CPUs) |
Grace delay pays only where most of the traffic is steady streams to and from clients: large downloads and uploads, video. For web traffic, many short requests and answers, it reduced the CPU on none of these links (the mix column above, and -g8000,2,5 through CONNECT at 10 Mbit/s, 10 ms, 0.5% loss): such connections rarely give NUM small reads in a row, so the waits seldom apply. There leave it off, whatever the clients' speed.
For streams, set DELAY to the clients' speed: about the time TRAFF bytes take to arrive. A shorter wait gathers nothing and costs CPU; a longer one costs throughput. As a starting point for TRAFF 8000:
| a client connection's speed | web traffic | steady streams |
|---|---|---|
| 100 Mbit/s and more | none | none: reads are large, the wait does not apply |
| about 20 Mbit/s | none | -g8000,2,2: about 40% less CPU for 3% of the throughput |
| about 5 Mbit/s | none | -g8000,2,10: 21-34% less CPU for 3-8% of the throughput |
logdump 1 1is useful to see how grace delays work.
ssl_ktls ssl_server_cert /path/to/cert.pem ssl_server_key /path/to/key.pem ssl_serv tcppm 8443 192.168.1.1 80ssl_ktls applies to the services configured after it, ssl_noktls (the default) turns it off. Whether the kernel took sending and receiving depends on the library, the kernel and the negotiated cipher (AES-GCM and ChaCha20-Poly1305 are the usual ones); 3proxy logs it on the first TLS connection of a service:
kernel TLS: sending in the kernel, receiving in the kernelReceiving with TLS 1.3 needs a later OpenSSL than 3.0. Distributions differ in whether their OpenSSL is built with it (RHEL and AlmaLinux are, Ubuntu 24.04 is not). wolfSSL builds do not support it and ignore the command.
A connection the kernel does TLS for both ways is spliced as a plain one is (with -s, the default; not where data filters apply). Measured on the Intel machine above, ending or beginning TLS with ssl_ktls, 3proxy moved bulk data (1 MB answers, streams) 6-17% faster at 5-34% less CPU per gigabyte, while 1K requests, which are not spliced, were 0-11% slower: the kernel does the same encryption, with a layer more for every record. Without splice() kernel TLS does not pay at all, as for nginx, which was 0-9% slower with it. Turn it on where TLS connections carry large transfers.