+proxy -s:16384,2,4096 +tcppm -s6:65536,3,8192 8080 192.168.1.1 80 ++A direction is spliced after STREAK reads in a row of MIN bytes or more (SMALL +from a TLS socket, which hands out a record at a time), and copied again after +more than STREAK splices in a row of less than SMALL; a body of known length is +spliced if at least MIN of it is left to read. The first line is the default. +A larger MIN or STREAK splices only longer transfers. In builds without it (CMake option 3PROXY_USE_SPLICE=OFF, or -DWITHSPLICE removed from the Makefile) -s is accepted and ignored.
splice() pays off when most of the traffic is large transfers. Measured in the benchmarks @@ -706,6 +716,7 @@ kernel's network work included):
Conclusion: for web traffic -s, the default, and -s0 show no @@ -720,8 +731,9 @@ CPU limits a connection. buffer memory. splice() is not used where 3proxy has to see the data: on a connection while data filters apply to it (an HTTP body is spliced only once the filters are done with it: they saw it whole, or it is larger than filtermaxsize, which -they skip), and not at all by a service that ends or begins TLS (ssl_serv, -ssl_cli). +they skip), and by a service that ends or begins TLS (ssl_serv, ssl_cli) only for +connections the kernel does TLS for both ways (see Kernel +TLS).
TCP_NODELAY and splice() do not conflict and can be combined on high-speed connections.
logdump 1 1is useful to see how grace -delays work. +On the 5 Mbit/s link one CPU was not enough for the HTTP proxy at 3000 +exchanges a second with any setting, so its column is from a run with a second +CPU for 3proxy, where it kept up with every one. The settings ranked the same +there: 10 ms took 27-34% off the CPU +per gigabyte of the streams for 3-8% of their throughput, 20 ms 33-37% for +10-14%, 2 ms added 12-19%. +
Grace delay pays only where most of the traffic is steady streams to and +from clients: large downloads and uploads, video. For web traffic, many short +requests and answers, it reduced the CPU on none of these links (the mix +column above, and -g8000,2,5 through CONNECT at 10 Mbit/s, 10 ms, 0.5% loss): such connections rarely give NUM small reads in a row, so the +waits seldom apply. There leave it off, whatever the clients' speed. +
For streams, set DELAY to the clients' speed: about the time TRAFF bytes +take to arrive. A shorter wait gathers nothing and costs CPU; a longer one +costs throughput. As a starting point for TRAFF 8000: +
| a client connection's speed | web traffic | steady streams |
|---|---|---|
| 100 Mbit/s and more | none | none: reads are large, the wait does not apply |
| about 20 Mbit/s | none | -g8000,2,2: about 40% less CPU for 3% of the throughput |
| about 5 Mbit/s | none | -g8000,2,10: 21-34% less CPU for 3-8% of the throughput |
logdump 1 1is useful to see how grace delays work. + +
+ssl_ktls +ssl_server_cert /path/to/cert.pem +ssl_server_key /path/to/key.pem +ssl_serv +tcppm 8443 192.168.1.1 80 ++ssl_ktls applies to the services configured after it, ssl_noktls (the +default) turns it off. Whether the kernel took sending and receiving depends on +the library, the kernel and the negotiated cipher (AES-GCM and ChaCha20-Poly1305 +are the usual ones); 3proxy logs it on the first TLS connection of a service: +
kernel TLS: sending in the kernel, receiving in the kernel+Receiving with TLS 1.3 needs a later OpenSSL than 3.0. Distributions differ in +whether their OpenSSL is built with it (RHEL and AlmaLinux are, Ubuntu 24.04 is +not). wolfSSL builds do not support it and ignore the command. +
A connection the kernel does TLS for both ways is spliced as a plain one +is (with -s, the default; not where data filters apply). Measured on the Intel +machine above, ending or beginning TLS with ssl_ktls, 3proxy moved bulk data +(1 MB answers, streams) 6-17% faster at 5-34% less CPU per gigabyte, while 1K +requests, which are not spliced, were 0-11% slower: the kernel does the same +encryption, with a layer more for every record. Without splice() kernel TLS +does not pay at all, as for nginx, which was 0-9% slower with it. Turn it on +where TLS connections carry large transfers.