mirror of
https://github.com/3proxy/3proxy.git
synced 2026-10-16 00:25:48 +08:00
c13c048f3f
bench.py: - the proxy gets one CPU, the client four, the origin three (8 CPUs), so the proxy is what runs out first - CPU time is the busy time of a role's CPUs from /proc/stat, softirqs included: with IRQ time accounting the cgroup's own figure leaves out much of the network work - every run records which side set its pace (proxy, client, origin), by the busy share of the role's CPUs and of its busiest thread - a 50 KB keep-alive scenario with 2000 clients - haproxy with splice: haproxy::splice-auto and haproxy::splice - wrk timeouts counted apart from errors; tools found by absolute path; leftovers of an interrupted run are cleaned up report.py turns runs into Markdown tables: throughput, the proxy's CPU per gigabyte or per request, its peak memory, splice and copying proxies in groups of their own, the best of each group marked. RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy, with and without splice, on an Apple M4 and an Intel Skylake virtual machine, and how it was measured; data/ has the measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
134 lines
6.3 KiB
Markdown
134 lines
6.3 KiB
Markdown
# Benchmarks
|
|
|
|
Compares the speed and the cost of 3proxy versions, and of other proxies doing
|
|
the same job, on Linux.
|
|
|
|
Results of the latest comparison, with the setup they were measured on, are
|
|
in [RESULTS.md](RESULTS.md); the measurements are in [data/](data/).
|
|
|
|
## How the measurement is isolated
|
|
|
|
The client, the proxy and the origin each run in a network namespace of
|
|
their own, linked by veth pairs:
|
|
|
|
```
|
|
[bcli] 10.0.1.2 --veth-- 10.0.1.1 [bprx] 10.0.2.1 --veth-- 10.0.2.2 [bsrv]
|
|
wrk, iperf3 the proxy nginx (files), iperf3 -s
|
|
```
|
|
|
|
so they share no loopback, and a packet crosses the same path whichever proxy
|
|
is measured. Each role also runs in a cgroup of its own pinned to separate
|
|
CPUs (`--cpus`, default `cli=0-3,prx=4,srv=5-7` for 8 CPUs), so one does not
|
|
take CPU from another, and RPS steers what each veth receives to the
|
|
receiver's CPUs, so the kernel's network work for a role is done on its own
|
|
CPUs too. The proxy's cgroup is created anew for every run, which is what
|
|
makes its peak memory readable on its own.
|
|
|
|
The proxy gets the fewest CPUs on purpose: the measurement is about the proxy,
|
|
so it has to be what runs out first. Every proxy is given the same CPUs: nginx
|
|
gets as many workers and haproxy as many threads as the proxy has CPUs;
|
|
3proxy runs a thread per connection inside the same cpuset. Logging is off
|
|
everywhere.
|
|
|
|
CPU time is the busy time of a role's CPUs read from `/proc/stat`: its
|
|
processes and the interrupts and softirqs handled there. The cgroup's own CPU
|
|
time is not enough: a kernel with `CONFIG_IRQ_TIME_ACCOUNTING` leaves the
|
|
softirqs out of it, and that is where much of the network work is done (a
|
|
quarter of a proxy's CPU in these tests).
|
|
|
|
Every run records which side set its pace (`bound`): the role whose CPUs were
|
|
busy at least 90% of the time, or which had a thread doing so (an iperf3
|
|
stream is one thread, so one busy core is its limit however many CPUs the
|
|
role has). A run bound by the client or the origin says only that the proxy
|
|
can do at least that much.
|
|
|
|
`none` is no proxy at all: the client reaches the origin through the proxy
|
|
namespace with the kernel forwarding, the upper bound for every scenario.
|
|
|
|
## What is reported
|
|
|
|
For every run, in a CSV under `bench/results/`:
|
|
|
|
| column | meaning |
|
|
|---|---|
|
|
| `gbit`, `rate` | throughput, requests per second for HTTP |
|
|
| `p50_ms`, `p99_ms` | latency from wrk (closed loop: compare at equal load only) |
|
|
| `prx_cpu_s`, `prx_task_s` | busy CPU seconds of the proxy's CPUs, network work included; the CPU time of its processes alone |
|
|
| `cpu_s_per_gb`, `cpu_us_per_req` | the proxy's cost per unit of work: the fairest figure, least affected by which side was the limit |
|
|
| `peak_mb` | the proxy's peak memory |
|
|
| `cli_util`, `prx_util`, `srv_util` | how busy each role's CPUs were |
|
|
| `cli_thread`, `srv_thread` | the busiest client and origin thread, as a share of one CPU |
|
|
| `bound` | `proxy`, `client`, `origin`, or `none` when nothing was busy all the time |
|
|
| `errors`, `timeouts`, `retrans` | socket errors and non-2xx answers; requests slower than wrk's 2 seconds; TCP retransmissions in the stream scenarios |
|
|
|
|
Repetitions rotate the order of the proxies, so drift over time (heat, the
|
|
host) falls on all of them alike; the summary gives medians.
|
|
|
|
## Scenarios
|
|
|
|
| name | load |
|
|
|---|---|
|
|
| `stream1`, `stream1-down` | one iperf3 stream through a TCP relay (3proxy `tcppm`, nginx `stream`, haproxy `mode tcp`), upload and download |
|
|
| `stream16` | sixteen streams |
|
|
| `http-ka-1k`, `http-ka-1m` | HTTP over kept-alive connections through an HTTP proxy (absolute URIs), 1 KB and 1 MB answers |
|
|
| `http-ka-50k-2000` | the same with 2000 clients at once, 50 KB answers |
|
|
| `http-close-1k` | a new connection for every request |
|
|
|
|
nginx is not a forward proxy: it answers absolute URIs by passing them to its
|
|
upstream, which is the origin here, and it has no CONNECT or SOCKS, so only
|
|
these scenarios compare like with like.
|
|
|
|
## Running
|
|
|
|
Root, `iproute2`, `iperf3`, `wrk`, `nginx` (with the stream module:
|
|
`libnginx-mod-stream` on Debian and Ubuntu, `nginx-mod-stream` on
|
|
RHEL-like systems) and, to compare with it,
|
|
`haproxy`.
|
|
|
|
```sh
|
|
bench/build.sh 0.9.9.1 1.0.1 # tags into bench/bin/
|
|
NAME=devel bench/build.sh . # the working tree
|
|
CC="gcc -fcommon" bench/build.sh 0.8.13 # before 0.9, newer gcc
|
|
CC="gcc -fpermissive" bench/build.sh 0.9.4 # 0.9.x with gcc 14 and later
|
|
|
|
sudo bench/bench.py \
|
|
--proxy none \
|
|
--proxy 3proxy:bench/bin/3proxy-1.0.1 \
|
|
--proxy 3proxy:bench/bin/3proxy-devel \
|
|
--proxy "3proxy:bench/bin/3proxy-0.9.4:-s0" \
|
|
--proxy nginx --proxy haproxy
|
|
```
|
|
|
|
`bench/report.py <csv>` turns a run into Markdown tables: throughput, the
|
|
proxy's CPU per gigabyte or per request, and its peak memory.
|
|
|
|
After the binary, a 3proxy entry can carry options for both its services:
|
|
`-s0` keeps 0.9.0 to 0.9.8, which use splice() by default, from using it, and
|
|
`-ocTCP_NODELAY -osTCP_NODELAY` turns Nagle's algorithm off on the client and
|
|
server sockets.
|
|
|
|
haproxy can be measured with splice() too: `haproxy::splice-auto` lets its
|
|
heuristic splice a direction once reads fill the buffer three times in a row,
|
|
`haproxy::splice` splices both directions always (`option splice-request` and
|
|
`option splice-response`). nginx has no splice() support, so it has no such
|
|
variant.
|
|
|
|
`--scenario` picks scenarios, `--reps` and `--duration` set the repetitions
|
|
and their length, `--netem "delay 1ms rate 10gbit"` puts a slower, longer link
|
|
between the client and the proxy, `--mtu` changes the links' MTU.
|
|
|
|
## Caveats
|
|
|
|
* A virtual machine cannot pin its virtual CPUs to physical ones, so its
|
|
absolute numbers move between sessions: compare within one session, and
|
|
keep absolute figures for real hardware.
|
|
* wrk measures latency in a closed loop, so a slower proxy is also offered
|
|
less load; compare latencies only between runs of similar throughput.
|
|
* The roles have CPUs of their own but share the memory: memory bandwidth,
|
|
which copying proxies use most, cannot be split, and on some hosts it,
|
|
rather than the CPU, sets the limit. A CPU waiting for memory counts as
|
|
busy, so such a limit shows as less work per busy second.
|
|
* Versions are built with the Makefile they shipped with, so compiler flags
|
|
differ between them (0.8 builds with `-O2`, 0.9.9 and later with
|
|
`-O3 -flto`) as they did for their users.
|