3proxy/doc/html/highload.html

<h3>Optimizing 3proxy for High Load</h3>
<p>Precaution 1: 3proxy was not initially developed for high load and is positioned as a SOHO product. The main reason is the "one connection - one thread" model 3proxy uses. 3proxy is known to work with over 200,000 connections under proper configuration, but use it in a production environment under high loads at your own risk and do not expect too much.
<p>Precaution 2: This documentation is incomplete and insufficient. High loads may require very specific system tuning including, but not limited to, specific or customized kernels, builds, settings, sysctls, options, etc. All of this is not covered by this documentation.

<h4>Configuring 'maxconn'</h4>

The number of simultaneous connections per service is limited by the 'maxconn' option.
The default maxconn value is 500. You may want to set 'maxconn'
to a higher value; it must be set before the services it should apply to. Under this configuration:
<pre>
maxconn 1000
proxy -p3129
proxy -p3128
socks
</pre>
maxconn for every service is 1000, and there are 3 services running
(2 proxy and 1 socks), so for all services there can be up to 3000
simultaneous connections to 3proxy.
<p>Avoid setting 'maxconn' to an arbitrarily high value; it should be carefully
chosen to protect the system and proxy from resource exhaustion. Setting maxconn
above available resources can lead to denial of service conditions.
<p>'maxconn' is not reduced automatically to fit the open file limit. If the limit is
too low 3proxy only prints a warning at startup
("current open file ulimits are too low") and then fails to accept connections once
the limit is reached, so check for this warning after changing 'maxconn'.
<h4>Understanding Resource Requirements</h4>
Each running service requires:
<ul>
<li>1 thread (process)
<li>1 socket (file descriptor)
<li>1 stack memory segment + some heap memory, ~64K-128K depending on the system
</ul>
Each connected client requires:
<ul>
<li>1 thread (process)
<li>2 sockets (file descriptors). For FTP, 4 sockets are required.
<br>With splice() (Linux, -s), a pipe (2 file descriptors) for each spliced direction: up to 4 more file descriptors, 6 for FTP.
<br>Up to 128K (up to 256K with splice()) of kernel buffer memory. This is the theoretical maximum; actual numbers depend on connection quality and traffic amount.
<br>1 additional socket (file descriptor) during name resolution for non-cached names
<br>1 additional socket during authentication or logging for RADIUS authentication or logging.
<li>1 ephemeral port (3 ephemeral ports for FTP connections).
<li>1 stack memory segment of ~32K-128K depending on the system + at least 16K and up to a few MB (for 'proxy' and 'ftppr') of heap memory. If you are short on memory, prefer 'socks' over 'proxy' and 'ftppr'.
<li>Many system buffers, especially in the case of slow network connections.
</ul>
Also, additional resources like system buffers are required for network activity.

<h4>Setting ulimits</h4>

Hard and soft ulimits must be set above calculated requirements. Under Linux, you can
check the limits of a running process with
<pre>
cat /proc/PID/limits
</pre>
where PID is the process ID.
Validate that ulimits match your expectations, especially if you run 3proxy under a dedicated account
by adding, e.g.:
<pre>
system "ulimit -Ha >>/tmp/3proxy.ulim.hard"
system "ulimit -Sa >>/tmp/3proxy.ulim.soft"
</pre>
at the beginning (before the first service is started) and at the end of the config file.
Perform both a hard restart (i.e., kill and start the 3proxy process) and a soft restart
by sending SIGUSR1 to the 3proxy process; check that the ulimits recorded to files match your
expectations. In systemd-based distros (e.g., latest Debian/Ubuntu) changing limits.conf is not
enough for a service: limits must be set in the unit file. Set them in the 3proxy
unit itself rather than globally, so the rest of the system is unaffected. The
shipped 3proxy.service already contains:
<pre>
LimitNOFILE=1048576
LimitNPROC=infinity
TasksMax=infinity
</pre>
To change them on an installed system use an override instead of editing the unit:
<pre>
systemctl edit 3proxy
systemctl daemon-reload &amp;&amp; systemctl restart 3proxy
systemctl show 3proxy -p LimitNOFILE -p LimitNPROC -p TasksMax
</pre>
<b>TasksMax is the one that is easy to miss.</b> It is the cgroup limit on the number
of threads, and if it is not set the unit inherits DefaultTasksMax, which is 15% of
kernel.threads-max (about 9000 on a typical host). Since 3proxy uses one thread per
connection, that caps concurrent connections at that number regardless of LimitNPROC
and maxconn, and the only symptom is "pthread_create()" errors in the log.
<p>On systemd older than 227, which has no TasksMax, and for limits that must apply to
several services, the same values can be set globally as DefaultLimitNOFILE /
DefaultLimitNPROC in /etc/systemd/system.conf, but prefer the per-unit settings.

<p>With SysV init the limits are not applied by limits.conf either, because
start-stop-daemon does not open a PAM session, so the daemon simply inherits the limits
of init. The shipped init script raises them itself before starting 3proxy:
<pre>
ulimit -n 65536
ulimit -u 32768
</pre>
adjust these values in the script to match 'maxconn'.

<p>On FreeBSD rc.subr applies limits(1) with the login class of the service (the
"daemon" class by default), so the limits can be set either in /etc/login.conf for that
class, or per service in rc.conf:
<pre>
3proxy_limits="-n 65536"
</pre>

<h4>Extending System Limitations</h4>

Check the manuals/documentation for your system's limitations, e.g., the system-wide limit for the number of open files
(fs.file-max in Linux). You may need to change sysctls or even rebuild the kernel from source.
<p>
To help with socket-based system-dependent settings, since 0.9-devel, 3proxy supports different
socket options which can be set via the -ol option for the listening socket, -oc for the proxy-to-client
socket, and -os for the proxy-to-server socket. Example:
<pre>
proxy -olSO_REUSEADDR,SO_REUSEPORT -ocTCP_TIMESTAMPS,TCP_NODELAY -osTCP_NODELAY
</pre>
Available options are system-dependent.

<h4>Linux Tuning Hints</h4>

Values below are examples, not recommendations: check the current value first
(<tt>sysctl NAME</tt>), change only what your workload actually hits, and make changes
persistent in <tt>/etc/sysctl.d/</tt>. Defaults given in parentheses are from a recent
(6.x) kernel and vary between distributions and versions.

<p><b>File descriptors.</b> 3proxy needs 2 descriptors per connection (4 for FTP), plus
one per service, plus temporary ones for name resolution and RADIUS.
<pre>
fs.nr_open = 1048576          &#35; (1048576) upper bound for any process' RLIMIT_NOFILE
</pre>
<tt>ulimit -n</tt> (RLIMIT_NOFILE) is the limit that actually applies and is commonly
left at 1024; it must be raised for the 3proxy process itself, see "Setting ulimits"
above. <tt>fs.file-max</tt> is effectively unlimited on 64-bit kernels and rarely needs
changing.

<p><b>Threads.</b> Because of the "one connection - one thread" model these limits are
reached earlier with 3proxy than with event-driven servers. Each thread also consumes
one or two mappings, so <tt>vm.max_map_count</tt> matters too.
<pre>
kernel.threads-max = 200000   &#35; (~60000 on a 16G host, scales with RAM)
kernel.pid_max = 4194304      &#35; (4194304)
vm.max_map_count = 1048576    &#35; (1048576)
</pre>
RLIMIT_NPROC (<tt>ulimit -u</tt>) limits threads per user and must be raised as well.
Check the actual thread count with <tt>grep Threads /proc/PID/status</tt>.

<p><b>Listen queue.</b> 3proxy uses a listen backlog of 1+(maxconn/8) unless the
'backlog' command is given, so a large 'maxconn' does not automatically give a large
queue, and the kernel caps it at somaxconn:
<pre>
net.core.somaxconn = 4096          &#35; (4096)
net.ipv4.tcp_max_syn_backlog = 4096 &#35; (512) raise for bursty connection rates
net.ipv4.tcp_syncookies = 1        &#35; (1) keep enabled
</pre>

<p><b>Ephemeral ports and TIME_WAIT.</b> See "Extending the Ephemeral Port Range" above
for the multi-IP case. The range gives about 28000 outgoing connections per
destination address by default:
<pre>
net.ipv4.ip_local_port_range = 10240 65535   &#35; (32768 60999)
net.ipv4.tcp_tw_reuse = 2                    &#35; (2) reuse TIME_WAIT for outgoing connections
net.ipv4.tcp_fin_timeout = 30                &#35; (60)
</pre>
Do not enable tcp_tw_recycle; it was removed in kernel 4.12 and breaks NAT clients.

<p><b>Socket buffers.</b> Autotuning is usually right. Buffer memory is per connection,
so raising the maximums with tens of thousands of connections costs a lot of RAM:
<pre>
net.core.rmem_max = 4194304     &#35; (212992)
net.core.wmem_max = 4194304     &#35; (212992)
net.ipv4.tcp_rmem = 4096 131072 6291456   &#35; (same) min default max
net.ipv4.tcp_wmem = 4096 16384 4194304    &#35; (same)
</pre>
Raise these only for high bandwidth-delay product links, and prefer raising the third
(max) value and leaving the default alone.

<p><b>Conntrack.</b> Only relevant if netfilter/nftables tracks the proxy's traffic. If
it does, the table is exhausted long before 3proxy's own limits, with
"nf_conntrack: table full, dropping packet" in dmesg:
<pre>
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_buckets = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 3600   &#35; (432000, i.e. 5 days)
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30       &#35; (120)
</pre>
nf_conntrack_max defaults to nf_conntrack_buckets, which itself is derived from the
amount of RAM, so it is often much lower than expected on small machines. Each
connection takes two entries (one per direction). The default established timeout of
5 days matters more than the table size with high connection churn: entries for
connections that are long gone keep occupying the table.
<p>If no rules need conntrack, not loading it at all is faster: the modules are loaded
on demand by the first rule that needs them ("-m state", "-m conntrack", any NAT
rule), so a ruleset without such rules keeps the proxy traffic untracked. If conntrack
is needed for other traffic but not for the proxy's, exempt the proxy's traffic
explicitly in the raw table:
<pre>
iptables -t raw -A PREROUTING -p tcp --dport 3128 -j CT --notrack
iptables -t raw -A OUTPUT -p tcp -m owner --uid-owner proxy -j CT --notrack
</pre>

<p><b>Conntrack helpers (ALGs).</b> The helper modules - nf_conntrack_ftp,
nf_conntrack_sip, nf_conntrack_h323, nf_conntrack_pptp, nf_conntrack_irc,
nf_conntrack_tftp - inspect the payload of every matching packet and create additional
"expectation" entries, so they cost both CPU and table space, and they have a long
history of security issues. Unload and blacklist the ones you do not actually need:
<pre>
lsmod | grep nf_conntrack
modprobe -r nf_conntrack_sip nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_pptp
echo "blacklist nf_conntrack_sip" >> /etc/modprobe.d/no-alg.conf
</pre>
On current kernels a helper only acts when it is attached explicitly
("-j CT --helper ftp"), so simply not attaching it is enough; automatic helper
assignment was deprecated and later removed. Older kernels, and most router firmware,
still enable them by default.

<p><b>Checking the result.</b> <tt>ss -s</tt> for socket state totals,
<tt>ss -lnt</tt> for listen queue overflow, <tt>nstat -az TcpExtListenOverflows
TcpExtListenDrops</tt> for accept queue drops, and
<tt>cat /proc/PID/limits</tt> for the limits actually applied to the running process.

<h4>Windows Tuning Hints</h4>

<p><b>Dynamic (ephemeral) port range.</b> Since Windows Vista / Server 2008 the default
range is 49152-65535, i.e. only 16384 outgoing connections per local address, which is
reached quickly by a busy proxy. Show and change it with:
<pre>
netsh int ipv4 show dynamicport tcp
netsh int ipv4 set dynamicport tcp start=10000 num=55535
</pre>
The minimum start port is 1025, the minimum size of the range is 255, and the end of
the range cannot exceed 65535. The range is set separately for TCP and UDP, and for
IPv4 and IPv6. On pre-Vista systems the equivalent is the MaxUserPort registry value
in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.

<p><b>Listening socket.</b> 3proxy sets SO_REUSEADDR on the listening socket by
default on Unix, but not on Windows: there it is not needed to rebind the port, and it
only allows another local process to bind the same address and port, with undefined
behaviour as to which of them receives the connections. If the machine is shared or
untrusted, harden the listening socket instead:
<pre>
proxy -olSO_EXCLUSIVEADDRUSE
</pre>
Note that a socket with SO_EXCLUSIVEADDRUSE may not be immediately rebindable after a
restart if accepted connections are still active, so test restarts before using it.

<p><b>Port reuse.</b> 3proxy always binds the outgoing socket before connecting, so
Windows does not apply its automatic ephemeral port reuse (which it does only for
connections with an implicit bind). Setting the option explicitly on the
proxy-to-server socket therefore helps against port exhaustion:
<pre>
proxy -osSO_REUSE_UNICASTPORT
</pre>
SO_REUSE_UNICASTPORT requires Windows 10 / Server 2019 or later. On older systems
(Windows 7 / Server 2008 and later) use SO_PORT_SCALABILITY instead; where both are
available Microsoft recommends SO_REUSE_UNICASTPORT. Note that SO_REUSEADDR has
different, weaker semantics on Windows than on Unix and allows another socket to bind
the same address and port, so do not use it on the listening socket as a substitute.

<p><b>TIME_WAIT.</b> Closed connections hold their port for the TcpTimedWaitDelay
period, set in
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters (DWORD, seconds). The
effective default differs between Windows versions (2 to 4 minutes); check the current
behaviour before changing it, and lower it only together with an extended port range.
Count the connections in that state with:
<pre>
netstat -ano -p tcp | find /c "TIME_WAIT"
</pre>

<p><b>Threads and address space.</b> Windows has no ulimit equivalent, and the handle
count is not normally the limit. On 32-bit builds the 2 GB of user address space is:
each connection thread reserves its stack there, so a few thousand connections can
exhaust the address space while physical memory is still free. Use a 64-bit build for high load, and
see "Setting Stack Size" above.

<p><b>Filter drivers.</b> Antivirus, endpoint protection and other LSP/WFP filter
drivers inspect every connection and are frequently the actual bottleneck on Windows,
costing far more than any tuning above can recover. Exclude the 3proxy process and its
ports, or test with the protection temporarily disabled to see the difference before
tuning anything else.

<h4>Using 3proxy in a Virtual Environment</h4>

If 3proxy is used in a VPS environment, there can be additional limitations.
For example, kernel resources, system CPU usage, and IOCTLs can be limited differently, and this can become a bottleneck.
splice() (see below) moves work into the kernel, so where the hosting limits kernel resources
(seen as nearly 100% CPU usage without any real activity of the application) do not enable it.

<h4>Extending the Ephemeral Port Range</h4>

Check the ephemeral port range for your system and extend it to the number of
ports required.
The ephemeral range is always limited to the maximum number of ports (64K). To extend the
number of outgoing connections above this limit, extending the ephemeral port range
is not enough; you need additional actions:
<ol>
<li> Configure multiple outgoing IPs
<li> Make sure 3proxy is configured to use a different outgoing IP by either setting
the external IP via RADIUS:
<pre>
radius secret 1.2.3.4
auth radius
proxy
</pre>
or by using multiple services with different external
interfaces, for example:
<pre>
allow user1,user11,user111
proxy -p1111 -e1.1.1.1
flush
allow user2,user22,user222
proxy -p2222 -e2.2.2.2
flush
allow user3,user33,user333
proxy -p3333 -e3.3.3.3
flush
allow user4,user44,user444
proxy -p4444 -e4.4.4.4
flush
</pre>
or via "parent extip" rotation,
e.g.:
<pre>
allow user1,user11,user111
parent 1000 extip 1.1.1.1 0
allow user2,user22,user222
parent 1000 extip 2.2.2.2 0
allow user3,user33,user333
parent 1000 extip 3.3.3.3 0
allow user4,user44,user444
parent 1000 extip 4.4.4.4 0
proxy
</pre>
or
<pre>
allow *
parent 250 extip 1.1.1.1 0
parent 250 extip 2.2.2.2 0
parent 250 extip 3.3.3.3 0
parent 250 extip 4.4.4.4 0
socks
</pre>
<pre>
</pre>
Under the latest Linux versions, you can also start multiple services with different
external addresses on a single port with SO_REUSEPORT on the listening socket to
evenly distribute incoming connections between outgoing interfaces:
<pre>
socks -olSO_REUSEPORT -p3128 -e1.1.1.1
socks -olSO_REUSEPORT -p3128 -e2.2.2.2
socks -olSO_REUSEPORT -p3128 -e3.3.3.3
socks -olSO_REUSEPORT -p3128 -e4.4.4.4
</pre>
For web browsing, the last two examples are not recommended because the same client can get
a different external address for different requests; you should choose the external
interface with user-based rules instead.
<li> You may need additional system-dependent actions to use the same port on different IPs,
usually by adding the SO_REUSEADDR (SO_PORT_SCALABILITY for Windows) socket option to
the external socket. This option can be set (since 0.9-devel) with the -os option:
<pre>
proxy -p3128 -e1.2.3.4 -osSO_REUSEADDR
</pre>
The behavior for SO_REUSEADDR and SO_REUSEPORT is different between different systems,
even between different kernel versions, and can lead to unexpected results.
The specifics are described <a href="https://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t">here</a>.
Use these options only if actually required and if you fully understand the possible
consequences. For example, SO_REUSEPORT can help establish more connections than the
number of client ports available, but it can also lead to situations where connections
randomly fail due to IP+port pair collisions if the remote or local system
doesn't support this trick.
</ol>

<h4>NAT on the Path Must Be Tuned Too</h4>

Everything above tunes the machine 3proxy runs on. If the outgoing traffic passes
through NAT - a router, a firewall, a CGNAT of the provider, or a cloud NAT gateway -
that device keeps its own translation table and its own pool of source ports, and it
limits the number of connections independently of the proxy. Extending
ip_local_port_range on the 3proxy host changes nothing if the NAT device rewrites the
source port from its own, smaller pool.

<p>On a Linux based router the same knobs apply and have to be raised there as well:
nf_conntrack_max / nf_conntrack_buckets and the conntrack timeouts (see "Linux Tuning
Hints" above), plus the port range used for translation, which is ip_local_port_range
for MASQUERADE, or the explicit range if SNAT is configured with --to-ports. Note that
the range is per translated address: with a single public IP, all clients share it.

<p>Entry level and SOHO routers are the usual bottleneck here. They typically have a
small fixed NAT/conntrack table (a few thousand entries), aggressive or non-adjustable
timeouts, and no way to change either. Symptoms are seen on the proxy but caused by the
router: connections that fail or hang at random under load while the proxy is far from
its own limits, no error in the 3proxy log except a failed outgoing connect, and
recovery after a pause or a router reboot. Before tuning 3proxy further, check the
router's session/NAT table counters. For high load either give the proxy a public
address without NAT in the path, or use a router where the table size and timeouts are
configurable.

<p>On the router, also turn off the application layer gateways that are not actually
used - they usually appear in the web interface as "SIP ALG", "FTP ALG", "H.323 ALG",
"PPTP passthrough", "IPsec/VPN passthrough". They are commonly enabled by default, they
parse the payload of matching connections, and they consume additional session table
entries for the connections they expect. If nothing behind the proxy uses FTP, VoIP or
those VPN protocols, disabling them frees table space and CPU on exactly the device
that is the bottleneck.

<h4>Setting Stack Size</h4>

'stacksize' is a size added to all stack allocations and can be both positive and
negative. Stack is required for function calls. 3proxy itself doesn't require a large
stack, but it can be required if some
poorly written libc, 3rd party libraries, or system functions are called. There is known
dirty code in Unix ODBC
implementations and built-in DNS resolvers, especially in the case of IPv6 and a large
number of interfaces. Under most 64-bit systems, extending stacksize will lead
to additional memory space usage but does not require actual committed memory,
so you can increase stacksize to a relatively large value (e.g., 1024000) without
the need to add additional physical memory,
but it's system/libc dependent and requires additional testing under your
installation. Don't forget about memory-related ulimits.
<p>For 32-bit systems, address space can be a bottleneck you should consider. If
you're short on address space, you can try using a negative stack size. The result is
never lowered below the system minimum (PTHREAD_STACK_MIN), so a large negative value
can not disable the thread stack. The base value the 'stacksize' is added to is 48K
(64K on FreeBSD/NetBSD/OpenBSD/DragonFly, where libc uses more stack, e.g. in
vfprintf() called by syslog()).

<h4>Known System Issues</h4>

There are known race condition issues in the Linux/glibc resolver. The probability
of a race condition arises under configuration with IPv6, a large number of interfaces
or IP addresses, or with resolvers configured. In this case, install a local recursor and
use 3proxy's built-in resolver (nserver / nscache / nscache6).
<h4>Do Not Use Public Resolvers</h4>
Public resolvers like those from Google have rate limits. For a large number of
requests, install a local caching recursor (ISC bind named, PowerDNS recursor, etc).

<h4>Use the Authentication Cache</h4>

Authentication and authorization are performed for every request. Password
lookup itself is a hash lookup and is cheap, but the ACL is walked linearly,
and with RADIUS or plugin authentication every request costs a network round
trip. 'authcache' keeps the result of a successful authentication for a number
of seconds:
<pre>
authcache &lt;cachetype&gt; &lt;cachetime&gt; &lt;cachesize&gt;
</pre>
'cachetype' lists the fields the entry is keyed by, 'cachetime' is the lifetime
in seconds, and the optional 'cachesize' limits the number of entries. The
cached type must be placed before the real authentication type in the 'auth'
command: a cache miss falls through to the next type in the chain, and a hit
skips it.

<p>There are two cached types:
<ul>
<li>'cache' caches the authentication only; the ACL is still checked on every
request. Use it when ACLs must take effect immediately, or when the
authentication method returns per-user parameters that the authorization step
has to apply.
<li>'cacheacl' also caches the authorization result, so the ACL is not walked
at all on a hit. This is the one that helps with long ACLs, but the key must
cover every field the ACL limits, and it can not be used with redirections.
</ul>

<p><b>'cacheacl' does not work with parent proxies.</b> Redirections are applied
while the ACL is walked, so on a 'cacheacl' hit no redirection happens at all:
the request is sent directly instead of through the 'parent' proxy of the
matching ACL entry, and nothing is reported. The first request for a given key
uses the parent and the rest do not, which makes it look intermittent. Use
'cache' with parent proxies:
<pre>
authcache user,pass 60
auth cache strong
allow user1
parent 1000 socks5 10.0.0.1 1080
proxy
</pre>
The one exception is 'parent ... extip': the external address is stored in the
cache entry and restored on a hit, but only if 'ext' is a part of the key:
<pre>
authcache user,pass,ext 60
auth cacheacl strong
allow user1
parent 1000 extip 1.2.3.4 0
proxy
</pre>
Without 'ext' in the key the external address falls back to the default one on
every cache hit.

<p><b>The key must match the ACL.</b> On a 'cacheacl' hit the cached result is
reused as is, so any field the ACL limits but the key does not contain makes
the cached result apply to a request it was not computed for. Use 'user' and
'pass' for the userlist, 'ip' for the sourcelist, 'dstaddr' and 'dsthost' for
the targetlist, 'dstport' for the targetportlist and 'dstoper' for the
operationlist. For this ACL:
<pre>
allow user1,user2 * * 80,443
deny *
</pre>
the key needs 'user', 'pass' and 'dstport':
<pre>
users $/etc/3proxy/passwd
authcache user,pass,dstport 60 65536
auth cacheacl strong
allow user1,user2 * * 80,443
deny *
proxy
</pre>

<p>With RADIUS the round trip is the expensive part, so cache by whatever
identifies the client:
<pre>
radius secret 1.2.3.4
authcache ip,user,pass 600
auth cache radius
proxy
</pre>
'cache' rather than 'cacheacl' is used here because RADIUS may return per-user
parameters such as the outgoing IP; add 'ext' to the key to cache the external
address returned by RADIUS as well.

<p>Choose 'cachetime' to balance the load against how quickly a password or ACL
change has to take effect. Cache entries are <b>not</b> invalidated by a
configuration reload, so a long lifetime keeps applying the old decision to
existing keys after the new configuration is loaded.

<p>Note that 'ip' alone, or 'ip,user' without 'pass', assigns every connection
from the same address to the same user for the lifetime of the entry without
checking the password. That is a deliberate trade-off for high load; do not use
it where clients share addresses.

<h4>Tune UDP Destination Handling with '-U'</h4>

Within a SOCKSv5 UDP association the destination is taken from every datagram,
so 3proxy authorizes and logs every new destination. One association can reach
any number of destinations: a DNS forwarder produces one record, but a
peer-to-peer or QUIC client can produce hundreds, each costing an authorization
(a network round trip with RADIUS or plugin authentication) and a log record.
The 'socks' service option '-U' controls this:
<pre>
socks -U3    &#35; default: log and authorize every new destination
socks -U2    &#35; authorize, do not log
socks -U1    &#35; log, do not authorize
socks -U0    &#35; neither (same as bare -U)
</pre>
The destination of the first datagram is always authorized and logged, so the
association as a whole is still subject to ACLs, authentication and the choice
of the parent proxy.

<p>Before reaching for the option, note that the cost is already avoided when no
ACL entry reachable by the client limits the destination: in that case the
result can not depend on the destination and no re-authorization is performed
whatever '-U' says. '-U' only matters for configurations that do limit the
destination.

<p>For high load:
<ul>
<li><b>'-U2'</b> is the safe one. Destination ACLs keep working, only the
per-destination log records are dropped. Use it when log volume, not CPU, is
the problem.
<li><b>'-U' ('-U0')</b> removes both costs and is the fastest, but destination
ACLs then apply only to the first destination of each association, and the
parent proxy and external address chosen for it are kept for the whole
association. Use it when UDP destinations are not restricted anyway, or when
the restriction is enforced elsewhere (a firewall, or the parent proxy).
<li>Keep the default '-U3' when destination ACLs are a security boundary, and
make the authorization itself cheap with 'authcache' instead, see "Use the
Authentication Cache" above.
</ul>

<h4>Avoid Large ACLs</h4>

Most of the large data structures are hash tables and scale to a large number of
records: the user list loaded by 'users' (including the '$/path/to/file' form),
the authentication cache ('authcache'), the DNS cache ('nscache'/'nscache6') and
the client table of the UDP portmapper. Their size is not a concern.

<p>The ACL is the exception and is still processed linearly. Every request walks
the access list entry by entry until one matches, and within an entry the
userlist, the sourcelist, the targetlist and the targetportlist are walked
linearly as well, so the cost is roughly the number of entries multiplied by the
size of the lists inside them. The 'bandlimin'/'bandlimout' and
'countin'/'countout' rules are separate lists of the same kind and are walked on
every request in addition to the access list.

<p>Two things follow from this:
<ul>
<li>Order matters. The walk stops at the first matching entry, so put the
entries that match most of the traffic first.
<li>Prefer fewer, wider entries over many narrow ones. One entry with a long
userlist is cheaper than one entry per user, because the outer walk is repeated
for every entry while the inner list is only walked for entries that got that
far.
</ul>

<p>With a large ACL, cache the authorization result instead of shortening the
list. 'cacheacl' walks the ACL once per cache entry rather than once per
request:
<pre>
users $/etc/3proxy/passwd
authcache user,pass,dstaddr,dstport 60 65536
auth cacheacl strong
</pre>
Both limitations from "Use the Authentication Cache" above apply: the cache key
must contain every field the ACL limits, and 'cacheacl' must not be used
together with parent proxies. Where a parent proxy is required, plain 'cache'
still removes the authentication cost but leaves the ACL walk on every request.

<p>RADIUS moves the user list and the ACLs out of 3proxy entirely, and also
allows you to set an outgoing IP on a per-user basis or to implement more
sophisticated logic. RADIUS is a new beta feature; test it before using it in
production.

<h4>Avoid Changing Configuration Too Often</h4>

Every configuration reload requires additional resources. Do not make frequent
changes, such as user addition/deletion via configuration; use alternative
authentication methods instead, like RADIUS.

<h4>Consider Using 'noforce'</h4>

The 'force' behavior (default) re-authenticates all connections after
configuration reload; it may be resource-consuming with a large number of
connections. Consider adding the 'noforce' command before services are started
to prevent connection re-authentication.

<h4>Do Not Monitor Configuration Files Directly</h4>

Using a configuration file directly in 'monitor' can lead to a race condition where
the configuration is reloaded while the file is being written.
To avoid race conditions:
<ol>
<li> Update config files only if there is no lock file
<li> Create a lock file when the 3proxy configuration is updated, e.g., with
"touch /some/path/3proxy/3proxy.lck". If you generate config files
asynchronously, e.g., by a user's request via web, you should consider
implementing existence checking and file creation as an atomic operation.
<li> Add
<pre>
system "rm /some/path/3proxy/3proxy.lck"
</pre>
at the end of the config file to remove it after the configuration is successfully loaded
<li> Use a dedicated version file to monitor, e.g.:
<pre>
monitor "/some/path/3proxy/3proxy.ver"
</pre>
<li> After the config is updated, change the version file for 3proxy to reload the configuration,
e.g., with "touch /some/path/3proxy/3proxy.ver".
</ol>

<h4>TCP_NODELAY for Connections with Small Amounts of Data</h4>

TCP_NODELAY is set by default on client (oc) and server (os)
connections: without it, Nagle's algorithm holds back a short write until the
previous one is acknowledged, and a peer delays its acknowledgement, so a reply
written in more than one part (e.g. HTTP headers, then a short body) waits tens
of milliseconds. On slow connections with high delays, where the bandwidth is the
bottleneck and fewer packets matter more than latency, it can be turned off:
<pre>
proxy -osTCP_NODELAY=0 -ocTCP_NODELAY=0
</pre>

<h4>Use Splice to Speed Up Large Data Transfers</h4>

On Linux, splice() passes data between the client and the server connections
through a kernel pipe, without copying it into the 3proxy process. Linux builds
include it, and services use it by default where it pays; -s sets the mode:
<pre>
proxy -s0
tcppm -s3 8080 192.168.1.1 80
</pre>
-s alone, the default, splices both directions where it pays: an HTTP body of known length if at
least 16K of it is left to read, a stream (CONNECT, SOCKS, tcppm) once it carries
large reads, going back to copying when they turn small. -s1 always splices what the
server sends to the client (downloads), -s2 what the client sends to the server
(uploads), -s3 both; -s5, -s6 and -s7 (the same as -s) do it for those directions
only where it pays. -s0 turns it off. Where it pays can be tuned after the mode,
as grace delay is, with :MIN,STREAK,SMALL:
<pre>
proxy -s:16384,2,4096
tcppm -s6:65536,3,8192 8080 192.168.1.1 80
</pre>
A direction is spliced after STREAK reads in a row of MIN bytes or more (SMALL
from a TLS socket, which hands out a record at a time), and copied again after
more than STREAK splices in a row of less than SMALL; a body of known length is
spliced if at least MIN of it is left to read. The first line is the default.
A larger MIN or STREAK splices only longer transfers. In builds without it (CMake option 3PROXY_USE_SPLICE=OFF,
or -DWITHSPLICE removed from the Makefile) -s is accepted and ignored.
<p>splice() pays off when most of the traffic is large transfers. Measured in
the <a href="https://github.com/3proxy/3proxy/blob/devel/bench/RESULTS.md">benchmarks</a>
on an 8 CPU KVM virtual machine (Intel Xeon Skylake, 2.0 GHz, Linux 6.12), with
3proxy confined to one CPU and the client and the origin on CPUs and in network
namespaces of their own, linked by veth pairs (MTU 1500), default 16K buffers,
the automatic mode (-s) against copying (-s0):
<table border="1" cellpadding="3">
<tr><th>load</th><th>-s0</th><th>-s</th><th>CPU per gigabyte</th></tr>
<tr><td>TCP streams through tcppm, CONNECT or SOCKS</td><td>9.4-10.2 Gbit/s</td><td>28-34 Gbit/s</td><td>65-70% less</td></tr>
<tr><td>HTTP proxy, 1 MB answers on kept-alive connections</td><td>9.4 Gbit/s</td><td>25.3 Gbit/s</td><td>62% less</td></tr>
<tr><td>2000 clients, 50 KB answers: tcppm</td><td>11.0k req/s</td><td>18.1k req/s</td><td>39% less</td></tr>
<tr><td>2000 clients, 50 KB answers: CONNECT, SOCKS</td><td>10.4-10.5k req/s</td><td>15.8k req/s</td><td>33-35% less</td></tr>
<tr><td>2000 clients, 50 KB answers: HTTP proxy</td><td>8.1k req/s</td><td>9.6k req/s</td><td>16% less</td></tr>
<tr><td>2000 clients, 2 KB POST and 50 KB answer: tcppm</td><td>10.5k req/s</td><td>16.4k req/s</td><td>36% less</td></tr>
<tr><td>2000 clients, 2 KB POST and 50 KB answer: CONNECT, SOCKS</td><td>9.8-10.1k req/s</td><td>14.6-14.7k req/s</td><td>31-34% less</td></tr>
<tr><td>2000 clients, 2 KB POST and 50 KB answer: HTTP proxy</td><td>7.5k req/s</td><td>9.4k req/s</td><td>20% less</td></tr>
<tr><td>1K requests, kept alive or a connection each</td><td colspan="3">within 5% either way: the automatic mode does not splice them</td></tr>
</table>
These loads are synthetic: the network between the client, 3proxy and the
origin has no bandwidth limit and no delay, so every connection goes as fast
as the CPUs allow, which favours splice() more than real clients would. The
mix scenarios, many connections with web-like exchanges, over links emulated
as below, are closer to a proxy giving clients access to the Internet.
<p>The mix closer to a proxy giving clients access to the Internet: 20000
connections, 3000 exchanges a second, web-like answers (median 12K, up to
8M), as a TCP relay, an HTTP proxy and through CONNECT, on the same machine,
with the clients' links emulated as in the grace delay tests below. 3proxy's
own CPU time per exchange with -s against -s0 (and the CPU's busy time, the
kernel's network work included):
<table border="1" cellpadding="3">
<tr><th>client link</th><th>tcppm</th><th>HTTP proxy</th><th>CONNECT</th></tr>
<tr><td>no limit, no delay</td><td>8% less (4%)</td><td>9% less (6%)</td><td>12% less (7%)</td></tr>
<tr><td>100 Mbit/s, 2 ms, 0.1% loss</td><td>7% less (+1%)</td><td>1% less (+3%)</td><td>6% less (+1%)</td></tr>
<tr><td>20 Mbit/s, 5 ms, 0.5% loss</td><td>+1% (+3%)</td><td>12% less (5%)</td><td>+7% (+8%)</td></tr>
<tr><td>10 Mbit/s, 10 ms, 0.5% loss (two CPUs)</td><td>–</td><td>–</td><td>9% less (2%)</td></tr>
<tr><td>5 Mbit/s, 25 ms, 1% loss</td><td>8% less (0%)</td><td>10% less (9%)</td><td>6% less (2%)</td></tr>
</table>
<p><b>Conclusion:</b> for web traffic -s, the default, and -s0 show no
significant difference: most connections are slow and their exchanges
short, splice() applies to few of them, and the difference stays within
0-12% of 3proxy's CPU at no cost in latency. -s increases the speed and
decreases CPU usage for high-speed large uploads and downloads, 3 times the
speed at 60-70% less CPU per gigabyte here, especially for traffic inside
high-speed local networks and virtual environments, where nothing but the
CPU limits a connection.
<p>Every spliced direction needs a pipe (two file descriptors) and more kernel
buffer memory. splice() is not used where 3proxy has to see the data: on a connection while
data filters apply to it (an HTTP body is spliced only once the filters are
done with it: they saw it whole, or it is larger than filtermaxsize, which
they skip), and by a service that ends or begins TLS (ssl_serv, ssl_cli) only for
connections the kernel does TLS for both ways (see <a href="#ktls">Kernel
TLS</a>).
<p>TCP_NODELAY and splice() do not conflict and can be combined on high-speed connections.

<h4>Add Grace Delay to Reduce System Calls</h4>

<pre>proxy -g8000,2,2</pre>
-gTRAFF,NUM,DELAY: once NUM reads in a row in one direction, within the same
second, have averaged TRAFF bytes or less, 3proxy waits DELAY milliseconds
before it polls and reads again, so that data gathers in the socket and is read
in fewer, larger pieces. The example waits 2 milliseconds after two reads in a
row averaging 8000 bytes or less. The counts start again every second and
whenever the other direction is read, and the wait stops as soon as reads grow
larger than TRAFF on average.
<p>Fewer reads mean fewer system calls, wakeups and context switches: it
decreases CPU usage on connections which carry data in small pieces, such as
bulk transfers to and from clients on slow links. The cost is the throughput of
a single connection. While 3proxy waits, it reads at most a buffer's worth
after each wait: 16K copying (with the default buffer size; less with -b), up
to 64K (the pipe) with splice(). This caps a connection's direction at one
buffer per DELAY:
<table border="1" cellpadding="3">
<tr><th>DELAY</th><th>copying, 16K per read</th><th>splice(), 64K per read</th></tr>
<tr><td>1 ms</td><td>131 Mbit/s</td><td>524 Mbit/s</td></tr>
<tr><td>2 ms</td><td>65 Mbit/s</td><td>262 Mbit/s</td></tr>
<tr><td>3 ms</td><td>44 Mbit/s</td><td>175 Mbit/s</td></tr>
<tr><td>5 ms</td><td>26 Mbit/s</td><td>105 Mbit/s</td></tr>
<tr><td>10 ms</td><td>13 Mbit/s</td><td>52 Mbit/s</td></tr>
</table>
These are the limits while the wait applies; the wait stops once reads grow
larger than TRAFF, so a fast connection escapes it as soon as data arrives
faster than TRAFF per DELAY.
<p>Measured on the same Intel machine as splice() above, the clients' links
emulated for each connection (rate limit, delay and loss each way, with fq and
netem on the client's side), 16 streams through tcppm and a mix of 20000
connections at a fixed 3000 exchanges a second (web-like answers, median 12K),
against 3proxy without grace delay:
<table border="1" cellpadding="3">
<tr><th>client link</th><th>setting</th><th>16 streams: throughput</th><th>16 streams: 3proxy CPU per gigabyte</th><th>mix, HTTP proxy: CPU busy</th></tr>
<tr><td rowspan="3">100 Mbit/s, 2 ms, 0.1% loss</td><td>-g8000,2,2</td><td rowspan="3">the same (100 Mbit/s each)</td><td rowspan="3">the same: reads are large, waits do not apply</td><td rowspan="3">the same (73%)</td></tr>
<tr><td>-g6000,3,2</td></tr>
<tr><td>-g8000,2,3</td></tr>
<tr><td rowspan="3">20 Mbit/s, 5 ms, 0.5% loss</td><td>-g8000,2,2</td><td>3% less</td><td>41% less</td><td rowspan="3">about the same: 88-96% with and without, across two sessions</td></tr>
<tr><td>-g6000,3,2</td><td>3% less</td><td>35% less</td></tr>
<tr><td>-g8000,2,3</td><td>7% less</td><td>46% less</td></tr>
<tr><td rowspan="3">5 Mbit/s, 25 ms, 1% loss</td><td>-g8000,2,2</td><td>the same (held by the loss)</td><td>21-23% more</td><td>117-124% against 109-111% (of 200%: two CPUs)</td></tr>
<tr><td>-g8000,2,10</td><td>5-8% less</td><td>21-29% less</td><td>about the same: 112-117%, within the runs' spread (two CPUs)</td></tr>
<tr><td>-g8000,2,20</td><td>12-17% less</td><td>31-32% less</td><td>about the same: 113% (two CPUs)</td></tr>
</table>
On the 5 Mbit/s link one CPU was not enough for the HTTP proxy at 3000
exchanges a second with any setting, so its column is from a run with a second
CPU for 3proxy, where it kept up with every one. The settings ranked the same
there: 10 ms took 27-34% off the CPU
per gigabyte of the streams for 3-8% of their throughput, 20 ms 33-37% for
10-14%, 2 ms added 12-19%.
<p>Grace delay pays only where most of the traffic is steady streams to and
from clients: large downloads and uploads, video. For web traffic, many short
requests and answers, it reduced the CPU on none of these links (the mix
column above, and -g8000,2,5 through CONNECT at 10 Mbit/s, 10 ms, 0.5% loss): such connections rarely give NUM small reads in a row, so the
waits seldom apply. There leave it off, whatever the clients' speed.
<p>For streams, set DELAY to the clients' speed: about the time TRAFF bytes
take to arrive. A shorter wait gathers nothing and costs CPU; a longer one
costs throughput. As a starting point for TRAFF 8000:
<table border="1" cellpadding="3">
<tr><th>a client connection's speed</th><th>web traffic</th><th>steady streams</th></tr>
<tr><td>100 Mbit/s and more</td><td>none</td><td>none: reads are large, the wait does not apply</td></tr>
<tr><td>about 20 Mbit/s</td><td>none</td><td>-g8000,2,2: about 40% less CPU for 3% of the throughput</td></tr>
<tr><td>about 5 Mbit/s</td><td>none</td><td>-g8000,2,10: 21-34% less CPU for 3-8% of the throughput</td></tr>
</table>
Choose DELAY so that its cap (above) stays over the speed of a single
connection you want to serve in full. Nor does grace delay help once 3proxy's
CPU is exhausted: the waits come on top of the queueing, and it degrades
slightly, with a little more CPU per exchange and longer delays. With grace
delay on streams splice() saves a little more than copying. <pre>logdump 1 1</pre> is useful to see how grace delays work.

<h4><a name="ktls">Kernel TLS</a></h4>

A service that ends or begins TLS (ssl_serv, ssl_cli, MITM) encrypts and
decrypts in 3proxy and does not splice(). With OpenSSL 3.0 or later built with
kernel TLS (enable-ktls), and the tls module loaded on Linux, the kernel can do
the encryption instead:
<pre>
ssl_ktls
ssl_server_cert /path/to/cert.pem
ssl_server_key /path/to/key.pem
ssl_serv
tcppm 8443 192.168.1.1 80
</pre>
ssl_ktls applies to the services configured after it, ssl_noktls (the
default) turns it off. Whether the kernel took sending and receiving depends on
the library, the kernel and the negotiated cipher (AES-GCM and ChaCha20-Poly1305
are the usual ones); 3proxy logs it on the first TLS connection of a service:
<pre>kernel TLS: sending in the kernel, receiving in the kernel</pre>
Receiving with TLS 1.3 needs a later OpenSSL than 3.0. Distributions differ in
whether their OpenSSL is built with it (RHEL and AlmaLinux are, Ubuntu 24.04 is
not). wolfSSL builds do not support it and ignore the command.
<p>A connection the kernel does TLS for both ways is spliced as a plain one
is (with -s, the default; not where data filters apply). Measured on the Intel
machine above, ending or beginning TLS with ssl_ktls, 3proxy moved bulk data
(1 MB answers, streams) 6-17% faster at 5-34% less CPU per gigabyte, while 1K
requests, which are not spliced, were 0-11% slower: the kernel does the same
encryption, with a layer more for every record. Without splice() kernel TLS
does not pay at all, as for nginx, which was 0-9% slower with it. Turn it on
where TLS connections carry large transfers.