bench.py takes ktls among 3proxy's options (ssl_ktls before the TLS
services) and for nginx (nginx::ktls, nginx::halfclose+ktls: Options
KTLS on both sides).
On Intel, whose OpenSSL and nginx have kernel TLS: 3proxy splicing over
it moves bulk data 6-17% faster at 5-34% less CPU per gigabyte, with 1K
requests 0-11% slower; before splice() over kernel TLS, and for nginx,
which relays through user space either way, kernel TLS was slower.
CONNECT at 10 Mbit/s, 10 ms, 0.5% loss: -s takes about 9% off the CPU
per exchange, grace delay matched to the rate changes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grace delay section, measured on plain veth links, is replaced by
one over an emulated client link: 20 Mbit/s per connection, 5 ms and
0.5% loss each way, devel with four -g settings and nginx, streams
through a relay and the fixed-rate mix scenarios. The README says where
the mix profile's shape comes from and that its parameters are chosen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wrk's and mix's 90th percentile is recorded (p90_ms) and shown in the
report.py --latency tables when present.
--flow-rate limits each connection between the client and the proxy
(fq maxrate), under --netem when both are given, now with a large netem
queue. Both directions are shaped in the client's namespace, the
incoming one through an ifb, so the emulation runs on the client's CPUs;
segmentation offload and GRO are off on that link so that netem drops
single packets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-stream* and tls-init-stream* run iperf3 streams through the TLS
relays. iperf3 has no TLS of its own, so bench/mix gets a TLS wrapper
mode, run on the client to begin TLS towards a proxy ending it and at the
origin to end the TLS a proxy begins.
--proxy envoy measures envoy where nginx and haproxy are measured: as an
HTTP proxy for the origin, as a TCP relay and in the TLS scenarios, its
circuit breakers lifted for the scenarios opening more than 1024
connections.
report.py --machine with a single machine gives its medians alone, with
the client- or origin-limited ones marked; --overall marks only the best
of each row. The mix client's -errlog prints every failed exchange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-* and tls-init-* measure TLS ended or begun by the proxy on a
TCP relay: 3proxy tcppm under ssl_serv/ssl_cli, nginx stream, haproxy
mode tcp and stunnel, with a certificate made for each run and an HTTPS
listener on the origin. nginx::halfclose turns on proxy_half_close.
bench/mix is the Go load generator and origin for the mix-* scenarios.
A run stops when the origin exits (e.g. killed by the OOM killer)
instead of recording failures for everything after it.
RESULTS.md gets the HTTPS and mix tables; data/ the full suite with 16
KB buffers and -s by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RESULTS.md now leads with devel as of the automatic splice mode (lazy
pipes kept per connection): the HTTP proxy, CONNECT tunnels (against
squid and tinyproxy), SOCKSv5 tunnels (against dante) and TCP relays
(tcppm against nginx stream and haproxy mode tcp), each a copying and a
splice table, on the M4 and Skylake hosts; 16 KB buffers against 64 KB.
The earlier sessions follow.
data/: *-auto-http.csv (HTTP proxy) and *-tunnels.csv (CONNECT, SOCKS,
relay) for both machines.
README.md: which proxies take part in which scenarios, reworded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- 2000 clients posting 2 KB and getting 50 KB back, next to the GET one
- haproxy with splice-response and splice-request, like -s1 and -s2
- CONNECT and SOCKSv5 scenarios: wrk and iperf3 run under proxychains-ng,
so every connection asks the proxy for a tunnel; and the same loads
through a TCP relay to a fixed destination
- tinyproxy and squid (HTTP proxy, CONNECT), dante (SOCKS); each proxy
takes part only in the scenarios it supports
- 3proxy also gets its socks service and a tcppm to the web server
report.py: one throughput table for several machines (--machine), each
cell their medians and their average; bold for the best of a group, * for
the best of the row; --split puts the splice proxies in a table of their
own, --prefix picks scenarios.
RESULTS.md: the comparison as one table for both machines, splice by
direction (devel -s0/-s1/-s3 against haproxy's settings) and grace delay
with splice, with their data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- the proxy gets one CPU, the client four, the origin three (8 CPUs), so
the proxy is what runs out first
- CPU time is the busy time of a role's CPUs from /proc/stat, softirqs
included: with IRQ time accounting the cgroup's own figure leaves out
much of the network work
- every run records which side set its pace (proxy, client, origin), by
the busy share of the role's CPUs and of its busiest thread
- a 50 KB keep-alive scenario with 2000 clients
- haproxy with splice: haproxy::splice-auto and haproxy::splice
- wrk timeouts counted apart from errors; tools found by absolute path;
leftovers of an interrupted run are cleaned up
report.py turns runs into Markdown tables: throughput, the proxy's CPU
per gigabyte or per request, its peak memory, splice and copying proxies
in groups of their own, the best of each group marked.
RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy,
with and without splice, on an Apple M4 and an Intel Skylake virtual
machine, and how it was measured; data/ has the measurements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench/build.sh builds git refs or the working tree, each with the
Makefile it shipped with. bench/bench.py runs the client, the proxy and
the origin in network namespaces of their own, linked by veth pairs,
and in cgroups pinned to separate CPUs, then measures TCP relaying
(iperf3) and HTTP proxying (wrk) for 3proxy builds, nginx, haproxy and
the kernel forwarding alone, reporting throughput, latency and the
proxy's CPU time and peak memory per unit of work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
