bench.py takes ktls among 3proxy's options (ssl_ktls before the TLS
services) and for nginx (nginx::ktls, nginx::halfclose+ktls: Options
KTLS on both sides).
On Intel, whose OpenSSL and nginx have kernel TLS: 3proxy splicing over
it moves bulk data 6-17% faster at 5-34% less CPU per gigabyte, with 1K
requests 0-11% slower; before splice() over kernel TLS, and for nginx,
which relays through user space either way, kernel TLS was slower.
CONNECT at 10 Mbit/s, 10 ms, 0.5% loss: -s takes about 9% off the CPU
per exchange, grace delay matched to the rate changes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 5 Mbit/s, 25 ms, 1% loss run again with a second proxy CPU, so
the HTTP proxy is not held by its CPU: it then keeps 3000 exchanges a
second in every configuration, and the grace delay settings rank as
with one CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results on both machines over per-connection emulated links: seven
grace delay settings with a fifth client CPU; the promising ones over
three link qualities (100 Mbit/s 2 ms 0.1%, 20 Mbit/s 5 ms 0.5%,
5 Mbit/s 25 ms 1%); copying against splice() with and without grace
delay, the CONNECT mix included; and the slow link with waits matched
to its rate (10 and 20 ms).
Grace delay saves 35-46% of 3proxy's CPU per gigabyte on streams at
20 Mbit/s and 21-29% at 5 Mbit/s with a 10 ms wait, does nothing on
fast links, and costs CPU with waits too short for the rate or once
the CPU is exhausted. Over these links splice() and copying cost about
the same in the mix scenarios. A caveat says that every scenario but
the mix ones is synthetic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grace delay section, measured on plain veth links, is replaced by
one over an emulated client link: 20 Mbit/s per connection, 5 ms and
0.5% loss each way, devel with four -g settings and nginx, streams
through a relay and the fixed-rate mix scenarios. The README says where
the mix profile's shape comes from and that its parameters are chosen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wrk's and mix's 90th percentile is recorded (p90_ms) and shown in the
report.py --latency tables when present.
--flow-rate limits each connection between the client and the proxy
(fq maxrate), under --netem when both are given, now with a large netem
queue. Both directions are shaped in the client's namespace, the
incoming one through an ifb, so the emulation runs on the client's CPUs;
segmentation offload and GRO are off on that link so that netem drops
single packets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
report.py --latency makes a table of the runs' 50th and 99th percentile
latency for one CSV, ranked by the 99th. RESULTS gets one for each
machine in the HTTP, CONNECT, SOCKS, relay and mix sections, with how to
read them: wrk runs a closed loop, so medians follow throughput and the
99th percentile against the median shows how evenly a proxy serves its
connections; the mix -3k runs are the open-loop ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP, CONNECT, SOCKS, TCP relay and mix tables rebuilt from the full run
on both machines with devel at 98fd0f3 (half-close fix, idle buffer
release, TLS changes), 1.0.1 with TCP_NODELAY, nginx relays with
proxy_half_close on, and envoy added. The morning session moves to the
earlier sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every table is now for one machine, ARM or Intel, with no averages of
the two, the copying proxies and those using splice() in one table:
bold the best of each group and * the best of the row for the HTTP proxy
and the relay, only the best of the row for CONNECT and SOCKS. The
tables for 13 October replace those of 11-12 October, whose columns for
splice by direction and 16 KB buffers are gone with them.
The HTTPS section is measured again with today's devel, with the TLS
streams and envoy. report.py takes the TLS streams and 1 MB answers for
bulk data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-stream* and tls-init-stream* run iperf3 streams through the TLS
relays. iperf3 has no TLS of its own, so bench/mix gets a TLS wrapper
mode, run on the client to begin TLS towards a proxy ending it and at the
origin to end the TLS a proxy begins.
--proxy envoy measures envoy where nginx and haproxy are measured: as an
HTTP proxy for the origin, as a TCP relay and in the TLS scenarios, its
circuit breakers lifted for the scenarios opening more than 1024
connections.
report.py --machine with a single machine gives its medians alone, with
the client- or origin-limited ones marked; --overall marks only the best
of each row. The mix client's -errlog prints every failed exchange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-* and tls-init-* measure TLS ended or begun by the proxy on a
TCP relay: 3proxy tcppm under ssl_serv/ssl_cli, nginx stream, haproxy
mode tcp and stunnel, with a certificate made for each run and an HTTPS
listener on the origin. nginx::halfclose turns on proxy_half_close.
bench/mix is the Go load generator and origin for the mix-* scenarios.
A run stops when the origin exits (e.g. killed by the OOM killer)
instead of recording failures for everything after it.
RESULTS.md gets the HTTPS and mix tables; data/ the full suite with 16
KB buffers and -s by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RESULTS.md now leads with devel as of the automatic splice mode (lazy
pipes kept per connection): the HTTP proxy, CONNECT tunnels (against
squid and tinyproxy), SOCKSv5 tunnels (against dante) and TCP relays
(tcppm against nginx stream and haproxy mode tcp), each a copying and a
splice table, on the M4 and Skylake hosts; 16 KB buffers against 64 KB.
The earlier sessions follow.
data/: *-auto-http.csv (HTTP proxy) and *-tunnels.csv (CONNECT, SOCKS,
relay) for both machines.
README.md: which proxies take part in which scenarios, reworded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- 2000 clients posting 2 KB and getting 50 KB back, next to the GET one
- haproxy with splice-response and splice-request, like -s1 and -s2
- CONNECT and SOCKSv5 scenarios: wrk and iperf3 run under proxychains-ng,
so every connection asks the proxy for a tunnel; and the same loads
through a TCP relay to a fixed destination
- tinyproxy and squid (HTTP proxy, CONNECT), dante (SOCKS); each proxy
takes part only in the scenarios it supports
- 3proxy also gets its socks service and a tcppm to the web server
report.py: one throughput table for several machines (--machine), each
cell their medians and their average; bold for the best of a group, * for
the best of the row; --split puts the splice proxies in a table of their
own, --prefix picks scenarios.
RESULTS.md: the comparison as one table for both machines, splice by
direction (devel -s0/-s1/-s3 against haproxy's settings) and grace delay
with splice, with their data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- the proxy gets one CPU, the client four, the origin three (8 CPUs), so
the proxy is what runs out first
- CPU time is the busy time of a role's CPUs from /proc/stat, softirqs
included: with IRQ time accounting the cgroup's own figure leaves out
much of the network work
- every run records which side set its pace (proxy, client, origin), by
the busy share of the role's CPUs and of its busiest thread
- a 50 KB keep-alive scenario with 2000 clients
- haproxy with splice: haproxy::splice-auto and haproxy::splice
- wrk timeouts counted apart from errors; tools found by absolute path;
leftovers of an interrupted run are cleaned up
report.py turns runs into Markdown tables: throughput, the proxy's CPU
per gigabyte or per request, its peak memory, splice and copying proxies
in groups of their own, the best of each group marked.
RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy,
with and without splice, on an Apple M4 and an Intel Skylake virtual
machine, and how it was measured; data/ has the measurements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench/build.sh builds git refs or the working tree, each with the
Makefile it shipped with. bench/bench.py runs the client, the proxy and
the origin in network namespaces of their own, linked by veth pairs,
and in cgroups pinned to separate CPUs, then measures TCP relaying
(iperf3) and HTTP proxying (wrk) for 3proxy builds, nginx, haproxy and
the kernel forwarding alone, reporting throughput, latency and the
proxy's CPU time and peak memory per unit of work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
