highload.html: kernel TLS, splice() heuristics, grace delay by traffic

A section on ssl_ktls: what it needs, how 3proxy reports it, and that
it pays with splice() for TLS connections carrying large transfers.
The splice() section gets the :MIN,STREAK,SMALL tuning and says TLS
services splice where the kernel does TLS. Grace delay is recommended
by the clients' speed for steady streams only: for web traffic it
saved nothing on any of the links measured, nor with the second CPU
the HTTP proxy needed on the 5 Mbit/s one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

doc/html/highload.html

@@ -670,7 +670,17 @@ least 16K of it is left to read, a stream (CONNECT, SOCKS, tcppm) once it carrie
 large reads, going back to copying when they turn small. -s1 always splices what the
 server sends to the client (downloads), -s2 what the client sends to the server
 (uploads), -s3 both; -s5, -s6 and -s7 (the same as -s) do it for those directions
-only where it pays. -s0 turns it off. In builds without it (CMake option 3PROXY_USE_SPLICE=OFF,
+only where it pays. -s0 turns it off. Where it pays can be tuned after the mode,
+as grace delay is, with :MIN,STREAK,SMALL:
+<pre>
+proxy -s:16384,2,4096
+tcppm -s6:65536,3,8192 8080 192.168.1.1 80
+</pre>
+A direction is spliced after STREAK reads in a row of MIN bytes or more (SMALL
+from a TLS socket, which hands out a record at a time), and copied again after
+more than STREAK splices in a row of less than SMALL; a body of known length is
+spliced if at least MIN of it is left to read. The first line is the default.
+A larger MIN or STREAK splices only longer transfers. In builds without it (CMake option 3PROXY_USE_SPLICE=OFF,
 or -DWITHSPLICE removed from the Makefile) -s is accepted and ignored.
 <p>splice() pays off when most of the traffic is large transfers. Measured in
 the <a href="https://github.com/3proxy/3proxy/blob/devel/bench/RESULTS.md">benchmarks</a>
@@ -706,6 +716,7 @@ kernel's network work included):
 <tr><td>no limit, no delay</td><td>8% less (4%)</td><td>9% less (6%)</td><td>12% less (7%)</td></tr>
 <tr><td>100 Mbit/s, 2 ms, 0.1% loss</td><td>7% less (+1%)</td><td>1% less (+3%)</td><td>6% less (+1%)</td></tr>
 <tr><td>20 Mbit/s, 5 ms, 0.5% loss</td><td>+1% (+3%)</td><td>12% less (5%)</td><td>+7% (+8%)</td></tr>
+<tr><td>10 Mbit/s, 10 ms, 0.5% loss (two CPUs)</td><td>–</td><td>–</td><td>9% less (2%)</td></tr>
 <tr><td>5 Mbit/s, 25 ms, 1% loss</td><td>8% less (0%)</td><td>10% less (9%)</td><td>6% less (2%)</td></tr>
 </table>
 <p><b>Conclusion:</b> for web traffic -s, the default, and -s0 show no
@@ -720,8 +731,9 @@ CPU limits a connection.
 buffer memory. splice() is not used where 3proxy has to see the data: on a connection while
 data filters apply to it (an HTTP body is spliced only once the filters are
 done with it: they saw it whole, or it is larger than filtermaxsize, which
-they skip), and not at all by a service that ends or begins TLS (ssl_serv,
-ssl_cli).
+they skip), and by a service that ends or begins TLS (ssl_serv, ssl_cli) only for
+connections the kernel does TLS for both ways (see <a href="#ktls">Kernel
+TLS</a>).
 <p>TCP_NODELAY and splice() do not conflict and can be combined on high-speed connections.
 
 <h4>Add Grace Delay to Reduce System Calls</h4>
@@ -762,25 +774,65 @@ against 3proxy without grace delay:
 <tr><td rowspan="3">100 Mbit/s, 2 ms, 0.1% loss</td><td>-g8000,2,2</td><td rowspan="3">the same (100 Mbit/s each)</td><td rowspan="3">the same: reads are large, waits do not apply</td><td rowspan="3">the same (73%)</td></tr>
 <tr><td>-g6000,3,2</td></tr>
 <tr><td>-g8000,2,3</td></tr>
-<tr><td rowspan="3">20 Mbit/s, 5 ms, 0.5% loss</td><td>-g8000,2,2</td><td>3% less</td><td>41% less</td><td>88% instead of 97%</td></tr>
-<tr><td>-g6000,3,2</td><td>3% less</td><td>35% less</td><td>92%</td></tr>
-<tr><td>-g8000,2,3</td><td>7% less</td><td>46% less</td><td>88%</td></tr>
-<tr><td rowspan="3">5 Mbit/s, 25 ms, 1% loss</td><td>-g8000,2,2</td><td>the same (held by the loss)</td><td>21-23% more</td><td rowspan="3">CPU-bound with or without</td></tr>
-<tr><td>-g8000,2,10</td><td>5-8% less</td><td>21-29% less</td></tr>
-<tr><td>-g8000,2,20</td><td>12-17% less</td><td>31-32% less</td></tr>
+<tr><td rowspan="3">20 Mbit/s, 5 ms, 0.5% loss</td><td>-g8000,2,2</td><td>3% less</td><td>41% less</td><td rowspan="3">about the same: 88-96% with and without, across two sessions</td></tr>
+<tr><td>-g6000,3,2</td><td>3% less</td><td>35% less</td></tr>
+<tr><td>-g8000,2,3</td><td>7% less</td><td>46% less</td></tr>
+<tr><td rowspan="3">5 Mbit/s, 25 ms, 1% loss</td><td>-g8000,2,2</td><td>the same (held by the loss)</td><td>21-23% more</td><td>117-124% against 109-111% (of 200%: two CPUs)</td></tr>
+<tr><td>-g8000,2,10</td><td>5-8% less</td><td>21-29% less</td><td>about the same: 112-117%, within the runs' spread (two CPUs)</td></tr>
+<tr><td>-g8000,2,20</td><td>12-17% less</td><td>31-32% less</td><td>about the same: 113% (two CPUs)</td></tr>
 </table>
-Grace delay pays where clients receive data steadily at a few tens of Mbit/s,
-in small pieces, with DELAY matched to the speed: about the time TRAFF bytes
-take to arrive. At 20 Mbit/s -g8000,2,2 takes about 40% off the CPU per
-gigabyte for 3% of the throughput; at 5 Mbit/s 2 ms is too short and costs
-CPU, while 10 ms takes a quarter off for 5-8%. On fast links it does nothing.
-In these tests splice() and copying cost about the same once clients' links
-are emulated; with grace delay on such streams splice() saves a little more. Many short exchanges gain
-little either way. Nor does grace delay help once 3proxy's CPU is exhausted:
-the waits then come on top of the queueing, and it degrades slightly, with a
-little more CPU per exchange and longer delays. Choose DELAY so that the cap is above the speed
-of a single client connection you want to serve in full, and TRAFF below the
-read size you expect from a fast one. Grace delay does not help many small
-requests and answers: they are too short to give NUM small reads in a row, and
-the waits rarely apply. <pre>logdump 1 1</pre> is useful to see how grace
-delays work.
+On the 5 Mbit/s link one CPU was not enough for the HTTP proxy at 3000
+exchanges a second with any setting, so its column is from a run with a second
+CPU for 3proxy, where it kept up with every one. The settings ranked the same
+there: 10 ms took 27-34% off the CPU
+per gigabyte of the streams for 3-8% of their throughput, 20 ms 33-37% for
+10-14%, 2 ms added 12-19%.
+<p>Grace delay pays only where most of the traffic is steady streams to and
+from clients: large downloads and uploads, video. For web traffic, many short
+requests and answers, it reduced the CPU on none of these links (the mix
+column above, and -g8000,2,5 through CONNECT at 10 Mbit/s, 10 ms, 0.5% loss): such connections rarely give NUM small reads in a row, so the
+waits seldom apply. There leave it off, whatever the clients' speed.
+<p>For streams, set DELAY to the clients' speed: about the time TRAFF bytes
+take to arrive. A shorter wait gathers nothing and costs CPU; a longer one
+costs throughput. As a starting point for TRAFF 8000:
+<table border="1" cellpadding="3">
+<tr><th>a client connection's speed</th><th>web traffic</th><th>steady streams</th></tr>
+<tr><td>100 Mbit/s and more</td><td>none</td><td>none: reads are large, the wait does not apply</td></tr>
+<tr><td>about 20 Mbit/s</td><td>none</td><td>-g8000,2,2: about 40% less CPU for 3% of the throughput</td></tr>
+<tr><td>about 5 Mbit/s</td><td>none</td><td>-g8000,2,10: 21-34% less CPU for 3-8% of the throughput</td></tr>
+</table>
+Choose DELAY so that its cap (above) stays over the speed of a single
+connection you want to serve in full. Nor does grace delay help once 3proxy's
+CPU is exhausted: the waits come on top of the queueing, and it degrades
+slightly, with a little more CPU per exchange and longer delays. With grace
+delay on streams splice() saves a little more than copying. <pre>logdump 1 1</pre> is useful to see how grace delays work.
+
+<h4><a name="ktls">Kernel TLS</a></h4>
+
+A service that ends or begins TLS (ssl_serv, ssl_cli, MITM) encrypts and
+decrypts in 3proxy and does not splice(). With OpenSSL 3.0 or later built with
+kernel TLS (enable-ktls), and the tls module loaded on Linux, the kernel can do
+the encryption instead:
+<pre>
+ssl_ktls
+ssl_server_cert /path/to/cert.pem
+ssl_server_key /path/to/key.pem
+ssl_serv
+tcppm 8443 192.168.1.1 80
+</pre>
+ssl_ktls applies to the services configured after it, ssl_noktls (the
+default) turns it off. Whether the kernel took sending and receiving depends on
+the library, the kernel and the negotiated cipher (AES-GCM and ChaCha20-Poly1305
+are the usual ones); 3proxy logs it on the first TLS connection of a service:
+<pre>kernel TLS: sending in the kernel, receiving in the kernel</pre>
+Receiving with TLS 1.3 needs a later OpenSSL than 3.0. Distributions differ in
+whether their OpenSSL is built with it (RHEL and AlmaLinux are, Ubuntu 24.04 is
+not). wolfSSL builds do not support it and ignore the command.
+<p>A connection the kernel does TLS for both ways is spliced as a plain one
+is (with -s, the default; not where data filters apply). Measured on the Intel
+machine above, ending or beginning TLS with ssl_ktls, 3proxy moved bulk data
+(1 MB answers, streams) 6-17% faster at 5-34% less CPU per gigabyte, while 1K
+requests, which are not spliced, were 0-11% slower: the kernel does the same
+encryption, with a layer more for every record. Without splice() kernel TLS
+does not pay at all, as for nginx, which was 0-9% slower with it. Turn it on
+where TLS connections carry large transfers.