A section on ssl_ktls: what it needs, how 3proxy reports it, and that
it pays with splice() for TLS connections carrying large transfers.
The splice() section gets the :MIN,STREAK,SMALL tuning and says TLS
services splice where the kernel does TLS. Grace delay is recommended
by the clients' speed for steady streams only: for web traffic it
saved nothing on any of the links measured, nor with the second CPU
the HTTP proxy needed on the 5 Mbit/s one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grace delay is described as it works: NUM small reads in a row in one
direction within a second start the wait, larger reads end it. It
saves CPU on connections carrying data in small pieces and caps a
connection at one read per DELAY, 16K copying and 64K (the pipe) with
splice(); a table gives the caps for 1-10 ms, another the measured
gains per client link, and DELAY is to be matched to the clients'
speed. It does not help once the CPU is exhausted. The example becomes
-g8000,2,2: the old -g8000,3,10 capped copying at 13 Mbit/s.
splice() gets measured figures: 3 times the speed at 60-70% less CPU
for large transfers on a fast network, within 0-12% for web traffic
in the mix over emulated links. -s, the automatic mode, is the
default; the paragraph still said -s0 was. splice() is not used where
3proxy has to see the data: while data filters apply, and by services
ending or beginning TLS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmarks live on the devel branch only, so link to their results
there; master has no bench/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s takes the directions to splice: -s1 what the server sends (downloads),
-s2 what the client sends (uploads), -s3 both. -s alone (-s7) splices both
only where it pays, as -s5 and -s6 do for one direction:
- a body of known length is spliced if at least 16K of it is left to read
once what is already buffered is counted, so an answer read whole with
its headers is just copied;
- a stream of unknown length starts copied, is spliced after 2 reads in a
row of at least 16K, and goes back to copying after 3 splices in a row
of less than 4K, once its pipe is empty.
Pipes are made when a direction first splices and kept with the
connection for its next mappings, so a kept-alive connection no longer
makes and closes two pipes per request; one left holding data by an
error or a timeout is closed. The direction not spliced keeps its buffer.
Before, -s spliced both directions and every mapping made both pipes,
which made small answers slower than without splice: with kept-alive
50K answers 2 pipe2() and 4 close() per request, now none.
Also corrects the default buffer size in the -b help (65536 for TCP).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- the proxy gets one CPU, the client four, the origin three (8 CPUs), so
the proxy is what runs out first
- CPU time is the busy time of a role's CPUs from /proc/stat, softirqs
included: with IRQ time accounting the cgroup's own figure leaves out
much of the network work
- every run records which side set its pace (proxy, client, origin), by
the busy share of the role's CPUs and of its busiest thread
- a 50 KB keep-alive scenario with 2000 clients
- haproxy with splice: haproxy::splice-auto and haproxy::splice
- wrk timeouts counted apart from errors; tools found by absolute path;
leftovers of an interrupted run are cleaned up
report.py turns runs into Markdown tables: throughput, the proxy's CPU
per gigabyte or per request, its peak memory, splice and copying proxies
in groups of their own, the best of each group marked.
RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy,
with and without splice, on an Apple M4 and an Intel Skylake virtual
machine, and how it was measured; data/ has the measurements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`proxy -g8000,3,10`
First parameter is average read size we want to keep, second parameter is
minimal number of packets in the same direction to apply algorythm,
last value is delay added after polling and prior to reading data.
An example above adds 10 millisecond delay before reading data if average
polling size is below 8000 bytes and 3 read operations are made in the same
direction. It's specially usefule with splice. `logdump 1 1` is useful
to see how grace delays work, choose delay value to avoid filling the read
pipe/buffer (typically 64K) but keep the request sizes close to chosen average
on large file upload/download.
