mirror of
https://github.com/3proxy/3proxy.git
synced 2026-11-22 23:59:41 +08:00
bench: RESULTS from the evening run of 13 October
HTTP, CONNECT, SOCKS, TCP relay and mix tables rebuilt from the full run
on both machines with devel at 98fd0f3 (half-close fix, idle buffer
release, TLS changes), 1.0.1 with TCP_NODELAY, nginx relays with
proxy_half_close on, and envoy added. The morning session moves to the
earlier sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7ec05ff372
commit
a3fc088d15
11 files changed
+1499
-199
No files matched your search
+223
-199
@@ -1,45 +1,49 @@
|
||||
# Benchmark results
|
||||
|
||||
3proxy against nginx, haproxy, squid, tinyproxy, dante and stunnel on two
|
||||
3proxy against nginx, haproxy, envoy, squid, tinyproxy, dante and stunnel on two
|
||||
machines, as an HTTP proxy, through CONNECT and SOCKSv5 tunnels, as a TCP
|
||||
relay, ending or beginning TLS, and with 20000 connections open at once; and
|
||||
devel against 1.0.1. The current figures are from 13 October 2026, for devel
|
||||
with 16 KB buffers, the automatic splice() mode by default and the thread
|
||||
pool; the sessions before them are at the end. How the benchmark works is in
|
||||
[README.md](README.md); the measurements are in [data/](data/), and
|
||||
`bench/report.py` turns them into the tables below.
|
||||
devel against 1.0.1. The current figures are from the evening of 13 October
|
||||
2026, for devel with 16 KB buffers, the automatic splice() mode by default,
|
||||
the thread pool and the half-close fix; the sessions before them are at the
|
||||
end. How the benchmark works is in [README.md](README.md); the measurements
|
||||
are in [data/](data/), and `bench/report.py` turns them into the tables
|
||||
below.
|
||||
|
||||
## In short
|
||||
|
||||
Ranges are over the two machines, ARM and Intel; the tables give each.
|
||||
|
||||
* **Bulk data:** devel splices 29-42 Gbit/s through every service (HTTP,
|
||||
CONNECT, SOCKS, `tcppm`), the fastest everywhere: 2-2.9 times squid, 2-4.2
|
||||
times dante, and 2.6-5.6 times nginx and haproxy copying 1 MB HTTP answers.
|
||||
haproxy `splice-auto` is level with it as a relay on Intel and behind on
|
||||
ARM. Copying with its 16 KB buffers (`-s0`) devel does 9.4-13.4 Gbit/s,
|
||||
level with nginx and haproxy; the automatic mode, the default, is what
|
||||
makes 16 KB enough.
|
||||
* **Small requests:** nginx and haproxy answer 1.5-2.2 times more 1 KB
|
||||
requests as relays and 1.9-2.8 times as HTTP proxies; squid 1.2-2 times
|
||||
* **Bulk data:** devel splices 32-48 Gbit/s through CONNECT, SOCKS and
|
||||
`tcppm` and 25-35 in 1 MB HTTP answers, the fastest everywhere: 2.2-3.1
|
||||
times squid, 2-4.3 times dante, 2.7-6.1 times nginx and haproxy copying 1 MB
|
||||
answers and 1.5-2.2 times envoy, the fastest copying proxy. haproxy
|
||||
`splice-auto` is level with it as a relay on Intel and behind on ARM.
|
||||
Copying with its 16 KB buffers (`-s0`) devel does 9.4-14.4 Gbit/s, level
|
||||
with nginx and haproxy; the automatic mode, the default, is what makes 16
|
||||
KB enough.
|
||||
* **Small requests:** nginx and haproxy answer 1.5-2.3 times more 1 KB
|
||||
requests as relays and 1.8-2.9 times as HTTP proxies; squid 1.2-1.85 times
|
||||
more through CONNECT; through SOCKS dante leads on ARM, devel on Intel.
|
||||
3proxy runs a thread per connection; splice() cannot change that.
|
||||
* **A new connection per request:** nginx and haproxy do 1.2-1.5 times more
|
||||
as relays, 2.2-2.9 times as HTTP proxies, where they keep their connections
|
||||
* **A new connection per request:** nginx and haproxy do 1.35-1.5 times more
|
||||
as relays, 2.3-2.8 times as HTTP proxies, where they keep their connections
|
||||
to the origin and a forward proxy cannot.
|
||||
* **2000 clients, 50 KB answers:** on Intel devel leads as an HTTP proxy,
|
||||
through CONNECT and through SOCKS; on ARM nginx, squid and dante are 13-47%
|
||||
ahead of it. As a relay haproxy `splice-auto` leads on both.
|
||||
through CONNECT and through SOCKS; on ARM nginx and envoy are 12% ahead of
|
||||
it as HTTP proxies, squid 41% through CONNECT, dante 29% through SOCKS. As
|
||||
a relay haproxy `splice-auto` leads on both.
|
||||
* **HTTPS:** with TLS ended or begun in the proxy devel moves bulk data as
|
||||
fast as nginx, ahead of haproxy and stunnel, with the least memory; envoy
|
||||
(BoringSSL) leads, by 30-50% on ARM. devel answers half to two thirds of
|
||||
nginx's 1 KB requests and does 60-95% of its handshakes. See
|
||||
[HTTPS](#https-tls-in-the-proxy).
|
||||
* **20000 connections:** devel keeps up with nginx and haproxy within 5-25% in
|
||||
throughput and CPU, but needs 65-130 KB per connection against their 8-30
|
||||
KB. See [the mix scenarios](#many-connections-the-mix-scenarios).
|
||||
* **Memory with few connections:** 3proxy needs the least, 5-20 MB against
|
||||
30-50 MB for nginx and haproxy.
|
||||
* **20000 connections:** devel does 63-95% of the best as an HTTP proxy and
|
||||
a relay and leads through CONNECT and SOCKS on Intel, with no failed
|
||||
exchange; it needs 65-90 KB per connection against 8-36 KB for nginx,
|
||||
haproxy and envoy. See [the mix scenarios](#many-connections-the-mix-scenarios).
|
||||
* **Memory with few connections:** 3proxy needs 4-23 MB, envoy 16-31 MB,
|
||||
nginx and haproxy 28-44 MB.
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -55,6 +59,7 @@ Ranges are over the two machines, ARM and Intel; the tables give each.
|
||||
| dante | 1.4.3 | 1.4.4 |
|
||||
| wrk, iperf3 | 4.1.0, 3.16 | 4.2.0, 3.17.1 |
|
||||
| proxychains-ng | 4.17 | 4.17 |
|
||||
| envoy | 1.34.1 (BoringSSL) | 1.34.1 (BoringSSL) |
|
||||
| stunnel, OpenSSL | 5.72, 3.0.13 | 5.72, 3.5.8 |
|
||||
|
||||
### How the client, the proxy and the origin were kept apart
|
||||
@@ -88,19 +93,20 @@ Ranges are over the two machines, ARM and Intel; the tables give each.
|
||||
|
||||
| column | configuration |
|
||||
|---|---|
|
||||
| devel | devel as of 13 October: 16 KB buffers, the thread pool, splice() in its automatic mode, the default: each direction switched to splice() once two reads in a row fill 16 KB and back to copying after three small ones; an HTTP body of known length is spliced when 16 KB or more of it remain |
|
||||
| devel | devel as of the evening of 13 October (98fd0f3): 16 KB buffers, the thread pool, splice() in its automatic mode, the default: each direction switched to splice() once two reads in a row fill 16 KB and back to copying after three small ones; an HTTP body of known length is spliced when 16 KB or more of it remain; a half-closed connection relays the rest of the answer; an idle kept-alive connection gives up its request buffer |
|
||||
| devel -s0 | devel, no splice (TCP_NODELAY is on by default) |
|
||||
| 1.0.1 | 1.0.1 as it starts by default: built without splice, 64 KB buffers, Nagle's algorithm on (no TCP_NODELAY, which devel sets by default) |
|
||||
| nginx | `proxy_pass` to an upstream with `keepalive 1024`, `stream` for the TCP relay, `reuseport` |
|
||||
| 1.0.1 | 1.0.1 built without splice, 64 KB buffers, with TCP_NODELAY on both sides (`-ocTCP_NODELAY -osTCP_NODELAY`) as devel sets it by default |
|
||||
| nginx | `proxy_pass` to an upstream with `keepalive 1024`, `stream` with `proxy_half_close on` for the TCP relay, `reuseport` |
|
||||
| haproxy | `mode http` with `http-reuse always`, `mode tcp` for the TCP relay |
|
||||
| haproxy splice-auto | `option splice-auto`: splices a direction once reads fill the buffer three times in a row |
|
||||
| envoy | one worker (`--concurrency 1`), `http_connection_manager` with the router filter as an HTTP proxy for the origin, `tcp_proxy` for the TCP relay |
|
||||
| squid | no cache, no logs, `via off` |
|
||||
| tinyproxy | `DisableViaHeader Yes`, critical messages only |
|
||||
| dante | no authentication, one `sockd` process tree (`-N 1`) |
|
||||
| stunnel | TLS scenarios only |
|
||||
|
||||
nginx has no splice() support. Neither nginx nor haproxy has CONNECT or SOCKS,
|
||||
so they are measured as HTTP proxies and TCP relays only; squid and tinyproxy
|
||||
nginx and envoy have no splice() support. None of nginx, haproxy and envoy has
|
||||
CONNECT or SOCKS, so they are measured as HTTP proxies and TCP relays only; squid and tinyproxy
|
||||
only as HTTP proxies and through CONNECT, dante only through SOCKS. 3proxy
|
||||
versions are built with the Makefile they shipped with (`-O3 -flto`). Logging
|
||||
is off everywhere.
|
||||
@@ -146,46 +152,50 @@ a TCP relay to the origin (3proxy `tcppm`, nginx `stream`, haproxy
|
||||
|
||||
## HTTP proxy
|
||||
|
||||
Measured on 13 October 2026, as were the CONNECT, SOCKS and relay tables
|
||||
([data/](data/) `*-16k-*.csv`).
|
||||
Measured on the evening of 13 October 2026, as were the CONNECT, SOCKS and
|
||||
relay tables ([data/](data/) `*-full-*.csv`).
|
||||
|
||||
**ARM (M4)**
|
||||
|
||||
| scenario (ARM) | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| ka-1k, k req/s | 14.5 | 1.5 | **40.3** * | 33.1 | 3.9 | 19.4 | ‖ | 14.5 | **40.1** * |
|
||||
| ka-1m, Gbit/s | **12.4** | 8.1 | 5.4 ‡ | 11.8 | 1.0 | 3.0 ‡ | ‖ | **30.2 ‡** * | 5.4 ‡ |
|
||||
| ka-50k-2000, k req/s | 8.6 | 8.1 | 8.6 | **10.9** * | 1.6 | 4.1 | ‖ | **9.6** | 8.5 |
|
||||
| ka-2k-50k-2000, k req/s | 7.8 | 6.7 | 7.9 | **9.6** * | 1.5 | 3.1 | ‖ | **8.7** | 8.1 |
|
||||
| close-1k, k req/s | 5.4 | 4.1 | **15.6** * | 12.2 | 3.9 | 6.8 | ‖ | 5.4 | **15.7** * |
|
||||
| scenario (ARM) | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| ka-1k, k req/s | 15.3 | 5.9 | **45.1** * | 37.3 | 24.1 | 4.1 | 21.3 | ‖ | 15.4 | **44.8** * |
|
||||
| ka-1m, Gbit/s | 13.5 | 12.3 | 5.7 ‡ | 12.8 | **23.9 ‡** | 1.1 | 3.2 ‡ | ‖ | **35.0 ‡** * | 5.9 ‡ |
|
||||
| ka-50k-2000, k req/s | 8.9 | 9.2 | 9.6 | **11.3** * | **11.2** * | 1.7 | 4.6 | ‖ | **10.0** | 9.8 |
|
||||
| ka-2k-50k-2000, k req/s | 8.1 | 7.8 | 8.8 | 10.3 | **11.3** * | 1.6 | 3.5 | ‖ | **9.0** | 8.8 |
|
||||
| close-1k, k req/s | 5.9 | 4.5 | **16.8** * | 13.4 | 15.3 | 4.0 | 7.2 | ‖ | 5.9 | **16.7** * |
|
||||
|
||||
**Intel (Skylake)**
|
||||
|
||||
| scenario (Intel) | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| ka-1k, k req/s | 18.9 | 1.6 | **36.2** * | **35.6** * | 3.9 | 9.3 | ‖ | 18.6 | **35.8** * |
|
||||
| ka-1m, Gbit/s | **9.4** | 7.8 | 7.9 | 5.9 | 1.1 | 2.2 | ‖ | **26.3 ‡** * | 24.8 ‡ |
|
||||
| ka-50k-2000, k req/s | 8.6 | **9.6** | 7.6 | 9.1 | 1.6 | 2.1 | ‖ | **10.4** * | 7.4 |
|
||||
| ka-2k-50k-2000, k req/s | 7.8 | **8.0** | 7.3 | **8.1** | 1.5 | 1.8 | ‖ | **9.5** * | 6.9 |
|
||||
| close-1k, k req/s | 6.0 | 3.9 | **15.8** * | 13.3 | 3.7 | 4.0 | ‖ | 6.1 | **15.5** * |
|
||||
| scenario (Intel) | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| ka-1k, k req/s | 17.2 | 6.1 | **32.7** * | **31.6** * | 9.9 | 3.8 | 8.8 | ‖ | 17.9 | **33.9** * |
|
||||
| ka-1m, Gbit/s | 9.4 | **12.4** | 7.3 | 5.8 | **11.8** | 1.1 | 2.1 | ‖ | **25.3 ‡** * | **22.5 ‡** * |
|
||||
| ka-50k-2000, k req/s | 8.1 | **9.4** * | 7.1 | **8.9** * | 6.1 | 1.6 | 2.1 | ‖ | **9.6** * | 7.1 |
|
||||
| ka-2k-50k-2000, k req/s | **7.5** | **8.8** * | 7.3 | **8.3** | 4.9 | 1.6 | 1.8 | ‖ | **9.4** * | 7.1 |
|
||||
| close-1k, k req/s | 5.8 | 3.9 | **14.8** * | 12.9 | 5.8 | 3.6 | 3.9 | ‖ | 5.5 | **15.0** * |
|
||||
|
||||
* 1 MB answers: devel splices them at 30 Gbit/s on ARM and 26 on Intel, 2.6
|
||||
to 5.6 times nginx and haproxy copying. haproxy `splice-auto` keeps up on
|
||||
Intel (25) but hardly splices on ARM (5.4). Copying (`-s0`), devel is ahead
|
||||
of the other copying proxies too (12.4 and 9.4 Gbit/s).
|
||||
* 2000 clients with 50 KB answers: nginx leads on ARM (10.9k requests a
|
||||
second against devel's 9.6k), devel on Intel (10.4k against 9.1k); haproxy
|
||||
is behind both on both. With a 2 KB POST each, the same order.
|
||||
* nginx and haproxy answer 2.3-2.8 times more 1 KB requests over kept-alive
|
||||
connections on ARM, 1.9 times on Intel, and 2.2-2.9 times more with a new
|
||||
connection for each. That last gap is not the thread pool's: nginx
|
||||
* 1 MB answers: devel splices them at 35 Gbit/s on ARM and 25 on Intel, 2.7
|
||||
to 6.1 times nginx and haproxy copying and 1.5-2.1 times envoy, the fastest
|
||||
of the copying proxies here. haproxy `splice-auto` keeps up on Intel (22.5)
|
||||
but hardly splices on ARM (5.9).
|
||||
* 2000 clients with 50 KB answers: nginx and envoy lead on ARM (11.3k
|
||||
requests a second against devel's 10.0k), devel on Intel (9.6k against
|
||||
nginx's 8.9k); haproxy is behind both on both. With a 2 KB POST each, the
|
||||
same order, envoy first on ARM.
|
||||
* nginx and haproxy answer 2.4-2.9 times more 1 KB requests over kept-alive
|
||||
connections on ARM, 1.8-1.9 times on Intel, and 2.3-2.8 times more with a
|
||||
new connection for each. That last gap is not the thread pool's: nginx
|
||||
(`keepalive 1024`) and haproxy (`http-reuse always`) keep their connections
|
||||
to the origin and only accept and close the client's, while a forward proxy
|
||||
such as 3proxy also opens and closes one to the server for every client
|
||||
connection. As a TCP relay, where every proxy does, the gap is 1.35-1.5
|
||||
times on ARM and 1.2-1.3 on Intel.
|
||||
* 1.0.1 answers 1.5k 1 KB requests a second over kept-alive connections, held
|
||||
by a 10 ms sleep removed in devel.
|
||||
connection. As a TCP relay, where every proxy does, the gap is 1.5 times on
|
||||
ARM and 1.35 on Intel.
|
||||
* envoy is close to nginx on ARM and well behind it on Intel (9.9k 1 KB
|
||||
requests against 31.6k, 5.8k new connections a second against 12.9k).
|
||||
* 1.0.1, with TCP_NODELAY set this time, answers 6k 1 KB requests a second
|
||||
over kept-alive connections (1.5k without it): one every 10.6 ms on each of
|
||||
the 64 connections, held by a 10 ms sleep removed in devel.
|
||||
|
||||
## CONNECT tunnel
|
||||
|
||||
@@ -193,32 +203,32 @@ Measured on 13 October 2026, as were the CONNECT, SOCKS and relay tables
|
||||
|
||||
| scenario (ARM) | devel -s0 | 1.0.1 | tinyproxy | squid | ‖ | devel |
|
||||
|---|---:|---:|---:|---:|:-:|---:|
|
||||
| stream1-down, Gbit/s | 13.3 | 20.5 | 1.2 | 14.3 | ‖ | **41.7 ‡** * |
|
||||
| stream16, Gbit/s | 12.6 | 19.0 | 1.2 | 16.9 | ‖ | **33.6 ‡** * |
|
||||
| ka-1k, k req/s | 17.4 | 17.5 | 15.4 | **35.0 ‡** * | ‖ | 17.5 |
|
||||
| ka-50k-2000, k req/s | 8.4 | 10.0 | 2.1 | **15.0** * | ‖ | 10.2 |
|
||||
| ka-2k-50k-2000, k req/s | 7.9 | 9.4 | 2.0 | **14.0** * | ‖ | 9.5 |
|
||||
| close-1k, k req/s | 4.2 ‡ | 3.5 ‡ | 3.7 ‡ | **4.9 ‡** * | ‖ | 4.2 ‡ |
|
||||
| stream1-down, Gbit/s | 14.4 | 22.7 | 1.3 | 15.2 | ‖ | **47.3 ‡** * |
|
||||
| stream16, Gbit/s | 13.7 | 21.1 | 1.3 | 17.7 | ‖ | **38.5 ‡** * |
|
||||
| ka-1k, k req/s | 19.7 | 19.7 | 17.1 | **36.4 ‡** * | ‖ | 19.7 |
|
||||
| ka-50k-2000, k req/s | 9.5 | 11.2 | 2.2 | **16.1** * | ‖ | 11.4 |
|
||||
| ka-2k-50k-2000, k req/s | 8.7 | 10.4 | 2.2 | **15.1** * | ‖ | 10.6 |
|
||||
| close-1k, k req/s | 4.6 ‡ | 3.7 ‡ | 4.1 ‡ | **5.3 ‡** * | ‖ | 4.6 ‡ |
|
||||
|
||||
**Intel (Skylake)**
|
||||
|
||||
| scenario (Intel) | devel -s0 | 1.0.1 | tinyproxy | squid | ‖ | devel |
|
||||
|---|---:|---:|---:|---:|:-:|---:|
|
||||
| stream1-down, Gbit/s | 9.6 | 19.1 | 1.2 | 13.4 | ‖ | **31.0** * |
|
||||
| stream16, Gbit/s | 9.7 | 18.7 | 1.3 | 13.1 | ‖ | **29.5** * |
|
||||
| ka-1k, k req/s | 30.7 | 30.3 | 29.0 | **36.3** * | ‖ | 30.4 |
|
||||
| ka-50k-2000, k req/s | 10.0 | 13.1 | 2.5 | 11.3 | ‖ | **15.4** * |
|
||||
| ka-2k-50k-2000, k req/s | 9.7 | 12.3 | 2.5 | 11.0 | ‖ | **14.1** * |
|
||||
| close-1k, k req/s | **5.2 ‡** * | 3.7 | 4.9 ‡ | 3.6 | ‖ | **5.2 ‡** * |
|
||||
| stream1-down, Gbit/s | 10.0 | 20.3 | 1.2 | 13.9 | ‖ | **33.9** * |
|
||||
| stream16, Gbit/s | 10.1 | 19.6 | 1.3 | 13.2 | ‖ | **32.4** * |
|
||||
| ka-1k, k req/s | 31.6 | 31.8 | 30.2 | **37.0** * | ‖ | 30.8 |
|
||||
| ka-50k-2000, k req/s | 10.5 | 13.3 | 2.5 | 11.7 | ‖ | **15.8** * |
|
||||
| ka-2k-50k-2000, k req/s | 9.8 | 12.8 | 2.4 | 11.4 | ‖ | **14.6** * |
|
||||
| close-1k, k req/s | 5.3 ‡ | 3.7 | 5.1 ‡ | 3.7 | ‖ | **5.3 ‡** * |
|
||||
|
||||
* With splice() devel moves 31-42 Gbit/s through a tunnel, 2-2.9 times squid;
|
||||
copying with 16 KB buffers (`-s0`) it does 10-13, below 1.0.1's 19-20 with
|
||||
its 64 KB buffers, which is what the automatic mode is for.
|
||||
* squid answers 1 KB requests twice as fast as devel on ARM, 19% faster on
|
||||
Intel; with 2000 clients squid leads on ARM (15.0k against 10.2k), devel on
|
||||
Intel (15.4k against 11.3k).
|
||||
* With splice() devel moves 32-47 Gbit/s through a tunnel, 2.2-3.1 times
|
||||
squid; copying with 16 KB buffers (`-s0`) it does 10-14, below 1.0.1's
|
||||
20-23 with its 64 KB buffers, which is what the automatic mode is for.
|
||||
* squid answers 1 KB requests 1.85 times as fast as devel on ARM, 20% faster
|
||||
on Intel; with 2000 clients squid leads on ARM (16.1k against 11.4k), devel
|
||||
on Intel (15.8k against 11.7k).
|
||||
* With a new tunnel per request proxychains on the client sets the pace on
|
||||
ARM; on Intel devel does 5.2k a second, squid 3.6k.
|
||||
ARM; on Intel devel does 5.3k a second, squid 3.7k.
|
||||
|
||||
## SOCKSv5 tunnel
|
||||
|
||||
@@ -226,61 +236,65 @@ Measured on 13 October 2026, as were the CONNECT, SOCKS and relay tables
|
||||
|
||||
| scenario (ARM) | devel -s0 | 1.0.1 | dante | ‖ | devel |
|
||||
|---|---:|---:|---:|:-:|---:|
|
||||
| stream1-down, Gbit/s | 12.9 | 19.7 | 10.6 | ‖ | **42.3 ‡** * |
|
||||
| stream16, Gbit/s | 12.8 | 19.1 | 13.5 | ‖ | **33.9** * |
|
||||
| ka-1k, k req/s | 17.5 | 17.6 | **25.1** * | ‖ | 17.5 |
|
||||
| ka-50k-2000, k req/s | 8.5 | 10.1 | **13.3** * | ‖ | 10.2 |
|
||||
| ka-2k-50k-2000, k req/s | 7.8 | 9.4 | **12.4** * | ‖ | 9.5 |
|
||||
| close-1k, k req/s | **4.0** * | 3.4 | 3.6 | ‖ | **4.0** * |
|
||||
| stream1-down, Gbit/s | 13.9 | 22.8 | 11.2 | ‖ | **48.1 ‡** * |
|
||||
| stream16, Gbit/s | 14.2 | 21.0 | 14.6 | ‖ | **38.5 ‡** * |
|
||||
| ka-1k, k req/s | 19.7 | 19.7 | **28.5** * | ‖ | 19.7 |
|
||||
| ka-50k-2000, k req/s | 9.4 | 11.1 | **14.7** * | ‖ | 11.4 |
|
||||
| ka-2k-50k-2000, k req/s | 8.7 | 10.3 | **13.9** * | ‖ | 10.5 |
|
||||
| close-1k, k req/s | **4.4** * | 3.6 | 3.8 | ‖ | **4.4** * |
|
||||
|
||||
**Intel (Skylake)**
|
||||
|
||||
| scenario (Intel) | devel -s0 | 1.0.1 | dante | ‖ | devel |
|
||||
|---|---:|---:|---:|:-:|---:|
|
||||
| stream1-down, Gbit/s | 9.7 | 19.4 | 8.0 | ‖ | **33.7** * |
|
||||
| stream16, Gbit/s | 9.7 | 18.5 | 15.8 | ‖ | **31.2** * |
|
||||
| ka-1k, k req/s | **29.6** * | **30.5** * | 25.0 | ‖ | **30.9** * |
|
||||
| ka-50k-2000, k req/s | 10.0 | 12.9 | 11.7 | ‖ | **15.0** * |
|
||||
| ka-2k-50k-2000, k req/s | 9.2 | 12.3 | 11.1 | ‖ | **14.5** * |
|
||||
| close-1k, k req/s | **4.7** * | 3.9 | 1.5 | ‖ | **5.1** * |
|
||||
| stream1-down, Gbit/s | 10.2 | 20.7 | 8.2 | ‖ | **33.3** * |
|
||||
| stream16, Gbit/s | 10.2 | 19.3 | 15.9 | ‖ | **31.9** * |
|
||||
| ka-1k, k req/s | **31.4** * | **31.2** * | 25.6 | ‖ | **30.4** * |
|
||||
| ka-50k-2000, k req/s | 10.4 | 13.4 | 12.5 | ‖ | **15.8** * |
|
||||
| ka-2k-50k-2000, k req/s | 10.1 | 12.7 | 11.5 | ‖ | **14.7** * |
|
||||
| close-1k, k req/s | **5.3** * | 3.9 | 1.6 | ‖ | **5.1** * |
|
||||
|
||||
* devel moves 31-42 Gbit/s with splice(), 2-4.2 times dante.
|
||||
* On ARM dante answers 43% more 1 KB requests and 30% more with 2000 clients;
|
||||
on Intel devel leads both (by 24% and 28%).
|
||||
* With a new tunnel per request devel manages 4.0-5.1k a second, dante 3.6k
|
||||
on ARM and 1.5k on Intel. With 2000 clients dante needs 0.8-1.4 GB, devel
|
||||
200 MB.
|
||||
* devel moves 32-48 Gbit/s with splice(), 2-4.3 times dante.
|
||||
* On ARM dante answers 45% more 1 KB requests and 29% more with 2000 clients;
|
||||
on Intel devel leads both (by 19% and 26%).
|
||||
* With a new tunnel per request devel manages 4.4-5.3k a second, dante 3.8k
|
||||
on ARM and 1.6k on Intel.
|
||||
|
||||
## TCP relay
|
||||
|
||||
**ARM (M4)**
|
||||
|
||||
| scenario (ARM) | devel -s0 | 1.0.1 | haproxy | nginx | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| stream1-down, Gbit/s | 13.4 | **20.3** | 13.1 | 12.7 | ‖ | **41.0** * | 26.2 |
|
||||
| stream16, Gbit/s | 12.8 | **19.2** | 8.1 ‡ | 12.7 | ‖ | **34.1 ‡** * | 14.3 ‡ |
|
||||
| ka-1k, k req/s | 17.9 | 17.9 | **39.0** * | **37.5** | ‖ | 17.9 | **39.8** * |
|
||||
| ka-50k-2000, k req/s | 9.4 | 11.1 | 10.3 | **12.8** | ‖ | 11.3 | **15.8** * |
|
||||
| ka-2k-50k-2000, k req/s | 8.7 | 10.3 | 10.1 | **11.9** | ‖ | 10.3 | **15.5** * |
|
||||
| close-1k, k req/s | 5.9 | 4.7 | **8.9** * | 8.0 | ‖ | 5.9 | **8.9** * |
|
||||
| scenario (ARM) | devel -s0 | 1.0.1 | haproxy | nginx | envoy | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| stream1-down, Gbit/s | 14.0 | 22.5 | 13.4 | 13.4 | **29.2** | ‖ | **48.0 ‡** * | 30.4 |
|
||||
| stream16, Gbit/s | 13.8 | **21.0** | 9.7 ‡ | 13.7 | 20.3 | ‖ | **38.3 ‡** * | 16.4 ‡ |
|
||||
| ka-1k, k req/s | 20.3 | 20.1 | **46.2** * | 44.2 | 35.5 | ‖ | 20.2 | **43.9** * |
|
||||
| ka-50k-2000, k req/s | 10.4 | 12.2 | 11.2 | 14.5 | **15.1** | ‖ | 12.6 | **17.4** * |
|
||||
| ka-2k-50k-2000, k req/s | 9.7 | 11.3 | 11.3 | 13.4 | **13.9** | ‖ | 11.6 | **17.1** * |
|
||||
| close-1k, k req/s | 6.4 | 5.1 | **9.7** * | 8.8 | **9.6** * | ‖ | 6.4 | **9.7** * |
|
||||
|
||||
**Intel (Skylake)**
|
||||
|
||||
| scenario (Intel) | devel -s0 | 1.0.1 | haproxy | nginx | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| stream1-down, Gbit/s | 9.6 | **19.0** | 9.4 | 9.6 | ‖ | **32.9** * | 31.5 |
|
||||
| stream16, Gbit/s | 9.9 | **18.7** | 9.1 | 9.8 | ‖ | **31.5** * | **31.0 ‡** * |
|
||||
| ka-1k, k req/s | 29.5 | 29.5 | 45.1 | **48.7** * | ‖ | 29.8 | **47.1** * |
|
||||
| ka-50k-2000, k req/s | 11.0 | **14.4** | 9.8 | 11.9 | ‖ | 17.5 | **18.8** * |
|
||||
| ka-2k-50k-2000, k req/s | 10.4 | **13.3** | 9.5 | 11.6 | ‖ | 15.3 | **17.9** * |
|
||||
| close-1k, k req/s | 6.4 | 4.8 | 7.9 | **8.6** * | ‖ | 6.5 | **8.3** |
|
||||
| scenario (Intel) | devel -s0 | 1.0.1 | haproxy | nginx | envoy | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| stream1-down, Gbit/s | 10.1 | **19.3** | 9.8 | 9.4 | 18.2 | ‖ | **33.2** * | **32.7** * |
|
||||
| stream16, Gbit/s | 9.7 | **19.2** | 9.2 | 9.7 | 14.7 | ‖ | **32.3** * | 30.7 ‡ |
|
||||
| ka-1k, k req/s | 29.8 | 30.9 | **48.3** * | **49.3** * | 36.6 | ‖ | 30.6 | **46.8** |
|
||||
| ka-50k-2000, k req/s | 11.0 | **15.0** | 10.2 | 12.3 | 12.8 | ‖ | 18.1 | **19.9** * |
|
||||
| ka-2k-50k-2000, k req/s | 10.5 | **13.8** | 9.7 | 12.1 | 12.4 | ‖ | 16.4 | **18.6** * |
|
||||
| close-1k, k req/s | 6.8 | 4.8 | **8.8** * | **8.6** * | 5.4 | ‖ | 6.5 | **8.7** * |
|
||||
|
||||
* On streams devel splices 32-41 Gbit/s; haproxy `splice-auto` is level on
|
||||
Intel with one stream (31.5) and sixteen (31.0), behind on ARM (26 and 14).
|
||||
* nginx and haproxy answer 2.1-2.2 times more 1 KB requests on ARM, 1.5-1.6
|
||||
* On streams devel splices 32-48 Gbit/s; haproxy `splice-auto` is level on
|
||||
Intel (32.7 and 30.7), behind on ARM (30 and 16). Of the copying proxies
|
||||
envoy is level with 1.0.1 and its 64 KB buffers on one stream (18-29),
|
||||
devel splicing 1.6-2.2 times either.
|
||||
* nginx and haproxy answer 2.2-2.3 times more 1 KB requests on ARM, 1.5-1.6
|
||||
times on Intel.
|
||||
* With 2000 clients and 50 KB answers haproxy `splice-auto` leads, 40% ahead
|
||||
of devel on ARM and 7% on Intel.
|
||||
* With 2000 clients and 50 KB answers haproxy `splice-auto` leads, 38% ahead
|
||||
of devel on ARM and 10% on Intel.
|
||||
* nginx runs with `proxy_half_close on` (`nginx::halfclose`), off by default,
|
||||
without which its relays cut short the exchanges whose client shuts down
|
||||
its sending side first.
|
||||
|
||||
## HTTPS: TLS in the proxy
|
||||
|
||||
@@ -412,114 +426,124 @@ program on one core, can set the pace (‡).
|
||||
|
||||
## Many connections: the mix scenarios
|
||||
|
||||
Measured on 13 October 2026 with the same devel ([data/](data/) `*-mix.csv`),
|
||||
20000 connections open through the proxy, `--mix-active` 2000; `-3k` at a
|
||||
fixed 3000 exchanges a second. dante is left out on Intel: with 20000
|
||||
connections it needs more memory than the machine has (6.6 GB on ARM) and the
|
||||
OOM killer ended the origin. ‡: limited by the client.
|
||||
Measured on the evening of 13 October 2026 with the same devel
|
||||
([data/](data/) `*-full-mix.csv`), 20000 connections open through the proxy,
|
||||
`--mix-active` 2000; `-3k` at a fixed 3000 exchanges a second. dante is left
|
||||
out on Intel: with 20000 connections it needs more memory than the machine
|
||||
has (6.5 GB on ARM) and the OOM killer ended the origin. ‡: limited by the
|
||||
client.
|
||||
|
||||
### ARM (M4)
|
||||
|
||||
**throughput, Gbit/s (thousands of requests per second for HTTP)**
|
||||
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | dante | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 1.80 (4.9k/s) | 1.42 (3.9k/s) | 1.83 (6.5k/s) ‡ | **2.76 (7.5k/s) ‡** | 0.59 (1.5k/s) | 0.92 (2.7k/s) ‡ | – (–k/s) | ‖ | **1.84 (5.0k/s)** | **1.81 (6.5k/s) ‡** |
|
||||
| mix-connect | 2.08 (5.5k/s) | 2.07 (5.5k/s) ‡ | – (–k/s) | – (–k/s) | 0.74 (2.0k/s) | **2.76 (7.3k/s) ‡** | – (–k/s) | ‖ | **2.16 (5.7k/s) ‡** | – (–k/s) |
|
||||
| mix-socks | 2.03 (5.5k/s) | **2.17 (5.8k/s) ‡** | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | 1.31 (3.5k/s) | ‖ | **2.13 (5.7k/s) ‡** | – (–k/s) |
|
||||
| mix-relay | 2.07 (5.7k/s) | 2.12 (5.8k/s) ‡ | 2.31 (6.8k/s) ‡ | **2.72 (7.5k/s) ‡** | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | 2.16 (5.9k/s) ‡ | **2.50 (7.1k/s) ‡** |
|
||||
| mix-http-3k | 1.09 (3.0k/s) | 1.08 (3.0k/s) | 1.08 (3.0k/s) | **1.09 (3.0k/s)** | 0.58 (1.5k/s) | **1.09 (3.0k/s) ‡** | – (–k/s) | ‖ | 1.08 (3.0k/s) | **1.08 (3.0k/s)** |
|
||||
| mix-connect-3k | **1.09 (3.0k/s)** | 1.08 (3.0k/s) | – (–k/s) | – (–k/s) | 0.74 (2.0k/s) | 1.08 (3.0k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-socks-3k | 1.08 (3.0k/s) | 1.08 (3.0k/s) | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | **1.13 (3.1k/s)** | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-relay-3k | **1.08 (3.0k/s)** | 1.08 (3.0k/s) | **1.09 (3.0k/s)** | 1.08 (3.0k/s) | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | **1.09 (3.0k/s)** |
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | dante | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 1.72 (4.7k/s) | 1.48 (4.1k/s) | 2.02 (6.8k/s) ‡ | **2.92 (8.0k/s) ‡** | 2.52 (6.9k/s) ‡ | 0.62 (1.6k/s) | 0.99 (2.9k/s) ‡ | – (–k/s) | ‖ | 1.84 (5.0k/s) | **2.06 (7.0k/s) ‡** |
|
||||
| mix-connect | 2.25 (6.0k/s) | 2.02 (5.4k/s) | – (–k/s) | – (–k/s) | – (–k/s) | 0.78 (2.2k/s) | **2.78 (7.4k/s) ‡** | – (–k/s) | ‖ | **2.13 (5.7k/s) ‡** | – (–k/s) |
|
||||
| mix-socks | **2.11 (5.7k/s)** | **2.07 (5.5k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | 1.34 (3.6k/s) | ‖ | **2.21 (6.0k/s) ‡** | – (–k/s) |
|
||||
| mix-relay | 2.17 (6.0k/s) | 2.15 (5.9k/s) | 2.49 (7.2k/s) ‡ | **2.86 (7.8k/s) ‡** | 2.77 (7.6k/s) ‡ | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | 2.28 (6.3k/s) ‡ | **2.66 (7.6k/s) ‡** |
|
||||
| mix-http-3k | 1.08 (3.0k/s) | 1.08 (3.0k/s) | 1.08 (3.0k/s) | **1.09 (3.0k/s)** | **1.09 (3.0k/s)** | 0.63 (1.6k/s) | **1.09 (3.0k/s)** | – (–k/s) | ‖ | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** |
|
||||
| mix-connect-3k | **1.08 (3.0k/s)** | **1.09 (3.0k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | 0.78 (2.1k/s) | 1.08 (3.0k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-socks-3k | 1.08 (3.0k/s) | 1.08 (3.0k/s) | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | **1.14 (3.0k/s)** | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-relay-3k | **1.09 (3.0k/s)** | 1.09 (3.0k/s) | 1.08 (3.0k/s) | **1.09 (3.0k/s)** | **1.09 (3.0k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** |
|
||||
|
||||
**proxy CPU time per gigabyte or per request**
|
||||
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | dante | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 205 µs/req | 272 µs/req | 156 µs/req | **134 µs/req** | 675 µs/req | 375 µs/req | – µs/req | ‖ | 211 µs/req | **157 µs/req** |
|
||||
| mix-connect | 190 µs/req | 187 µs/req | – µs/req | – µs/req | 507 µs/req | **139 µs/req** | – µs/req | ‖ | **182 µs/req** | – µs/req |
|
||||
| mix-socks | 187 µs/req | **179 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | 298 µs/req | ‖ | **179 µs/req** | – µs/req |
|
||||
| mix-relay | 186 µs/req | 182 µs/req | 151 µs/req | **136 µs/req** | – µs/req | – µs/req | – µs/req | ‖ | 177 µs/req | **143 µs/req** |
|
||||
| mix-http-3k | 236 µs/req | 278 µs/req | **162 µs/req** | 169 µs/req | 678 µs/req | 326 µs/req | – µs/req | ‖ | 236 µs/req | **162 µs/req** |
|
||||
| mix-connect-3k | 218 µs/req | 216 µs/req | – µs/req | – µs/req | 509 µs/req | **194 µs/req** | – µs/req | ‖ | **212 µs/req** | – µs/req |
|
||||
| mix-socks-3k | **215 µs/req** | **211 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | 334 µs/req | ‖ | **210 µs/req** | – µs/req |
|
||||
| mix-relay-3k | 215 µs/req | 212 µs/req | **179 µs/req** | **178 µs/req** | – µs/req | – µs/req | – µs/req | ‖ | 209 µs/req | **178 µs/req** |
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | dante | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 215 µs/req | 257 µs/req | 150 µs/req | **127 µs/req** | 147 µs/req | 660 µs/req | 351 µs/req | – µs/req | ‖ | 210 µs/req | **146 µs/req** |
|
||||
| mix-connect | 173 µs/req | 189 µs/req | – µs/req | – µs/req | – µs/req | 477 µs/req | **139 µs/req** | – µs/req | ‖ | **184 µs/req** | – µs/req |
|
||||
| mix-socks | **180 µs/req** | **191 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | – µs/req | 294 µs/req | ‖ | **172 µs/req** | – µs/req |
|
||||
| mix-relay | 172 µs/req | 179 µs/req | 140 µs/req | **130 µs/req** | **139 µs/req** | – µs/req | – µs/req | – µs/req | ‖ | 164 µs/req | **134 µs/req** |
|
||||
| mix-http-3k | 224 µs/req | 255 µs/req | **154 µs/req** | 162 µs/req | 184 µs/req | 631 µs/req | 295 µs/req | – µs/req | ‖ | 230 µs/req | **155 µs/req** |
|
||||
| mix-connect-3k | **201 µs/req** | **211 µs/req** | – µs/req | – µs/req | – µs/req | 482 µs/req | **193 µs/req** | – µs/req | ‖ | **202 µs/req** | – µs/req |
|
||||
| mix-socks-3k | **212 µs/req** | **206 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | – µs/req | 343 µs/req | ‖ | **201 µs/req** | – µs/req |
|
||||
| mix-relay-3k | 205 µs/req | 207 µs/req | 170 µs/req | **167 µs/req** | 174 µs/req | – µs/req | – µs/req | – µs/req | ‖ | 206 µs/req | **167 µs/req** |
|
||||
|
||||
**proxy peak memory, MB**
|
||||
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | dante | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 2,407 | 2,729 | 529 | **153** | 817 | 839 | – | ‖ | 2,438 | **556** |
|
||||
| mix-connect | 2,725 | 3,220 | – | – | **1,033** | 1,768 | – | ‖ | **2,760** | – |
|
||||
| mix-socks | **1,482** | 2,007 | – | – | – | – | 6,580 | ‖ | **1,506** | – |
|
||||
| mix-relay | 1,246 | 1,767 | **293** | 608 | – | – | – | ‖ | 1,296 | **263** |
|
||||
| mix-http-3k | 2,614 | 2,752 | 150 | **139** | 803 | 603 | – | ‖ | 2,627 | **154** |
|
||||
| mix-connect-3k | 2,653 | 2,831 | – | – | **1,025** | 1,289 | – | ‖ | **2,670** | – |
|
||||
| mix-socks-3k | **1,404** | 1,559 | – | – | – | – | 6,747 | ‖ | **1,422** | – |
|
||||
| mix-relay-3k | 1,291 | 1,456 | **267** | 574 | – | – | – | ‖ | 1,306 | **264** |
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | dante | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 1,613 | 2,730 | 531 | **151** | 587 | 819 | 839 | – | ‖ | 1,681 | **548** |
|
||||
| mix-connect | 1,494 | 3,220 | – | – | – | **1,007** | 1,774 | – | ‖ | **1,774** | – |
|
||||
| mix-socks | **1,462** | 1,948 | – | – | – | – | – | 6,518 | ‖ | **1,496** | – |
|
||||
| mix-relay | 1,248 | 1,789 | **292** | 617 | 645 | – | – | – | ‖ | 1,284 | **269** |
|
||||
| mix-http-3k | 1,564 | 2,756 | 150 | **144** | 386 | 800 | 609 | – | ‖ | 1,592 | **156** |
|
||||
| mix-connect-3k | 1,412 | 2,835 | – | – | – | **1,027** | 1,292 | – | ‖ | **1,428** | – |
|
||||
| mix-socks-3k | **1,405** | 1,559 | – | – | – | – | – | 6,627 | ‖ | **1,422** | – |
|
||||
| mix-relay-3k | 1,293 | 1,462 | **268** | 576 | 598 | – | – | – | ‖ | 1,308 | **268** |
|
||||
|
||||
### Intel (Skylake)
|
||||
|
||||
**throughput, Gbit/s (thousands of requests per second for HTTP)**
|
||||
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 2.39 (6.5k/s) | 1.98 (5.4k/s) | 2.29 (7.1k/s) | **2.87 (7.9k/s)** | 0.65 (1.7k/s) | 0.80 (2.2k/s) | ‖ | 2.54 (7.0k/s) | **3.02 (8.6k/s)** |
|
||||
| mix-connect | **3.08 (8.2k/s)** | 2.72 (7.3k/s) | – (–k/s) | – (–k/s) | 1.02 (2.8k/s) | 2.72 (7.2k/s) | ‖ | **3.39 (9.1k/s) ‡** | – (–k/s) |
|
||||
| mix-socks | 2.98 (8.0k/s) | **3.12 (8.4k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | **3.37 (9.0k/s)** | – (–k/s) |
|
||||
| mix-relay | 2.97 (8.1k/s) | 3.11 (8.6k/s) | 2.54 (7.9k/s) ‡ | **3.61 (9.9k/s) ‡** | – (–k/s) | – (–k/s) | ‖ | **3.35 (9.2k/s)** | 3.00 (8.5k/s) |
|
||||
| mix-http-3k | **1.10 (3.0k/s)** | **1.09 (3.0k/s)** | 1.08 (3.0k/s) | 1.09 (3.0k/s) | 0.64 (1.6k/s) | 0.63 (2.1k/s) | ‖ | **1.09 (3.0k/s)** | **1.09 (3.0k/s)** |
|
||||
| mix-connect-3k | **1.08 (3.0k/s)** | 1.08 (3.0k/s) | – (–k/s) | – (–k/s) | 1.01 (2.8k/s) | 1.08 (3.0k/s) | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-socks-3k | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-relay-3k | 1.08 (3.0k/s) | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** | 1.08 (3.0k/s) | – (–k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** |
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 2.14 (5.9k/s) | 2.04 (5.6k/s) | 2.42 (7.4k/s) | **2.94 (8.0k/s)** | 2.04 (5.6k/s) | 0.65 (1.7k/s) | 0.82 (2.4k/s) | ‖ | 2.43 (6.7k/s) | **3.19 (9.1k/s)** |
|
||||
| mix-connect | **3.02 (8.1k/s)** | **3.03 (8.1k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | 1.04 (2.9k/s) | **2.86 (7.6k/s)** | ‖ | **3.48 (9.3k/s)** | – (–k/s) |
|
||||
| mix-socks | 3.08 (8.4k/s) | **3.25 (8.7k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | **3.41 (9.2k/s)** | – (–k/s) |
|
||||
| mix-relay | 3.09 (8.5k/s) | 3.26 (9.0k/s) | 2.53 (7.9k/s) | **3.66 (10.0k/s) ‡** | 3.24 (8.9k/s) | – (–k/s) | – (–k/s) | ‖ | **3.46 (9.5k/s)** | 3.27 (9.2k/s) |
|
||||
| mix-http-3k | 1.08 (3.0k/s) | **1.08 (3.0k/s)** | 1.08 (3.0k/s) | **1.09 (3.0k/s)** | **1.09 (3.0k/s)** | 0.66 (1.7k/s) | 0.79 (2.3k/s) | ‖ | **1.09 (3.0k/s)** | 1.08 (3.0k/s) |
|
||||
| mix-connect-3k | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | 1.06 (2.9k/s) | 1.08 (3.0k/s) | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-socks-3k | **1.08 (3.0k/s)** | **1.08 (3.0k/s)** | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | – (–k/s) | ‖ | **1.08 (3.0k/s)** | – (–k/s) |
|
||||
| mix-relay-3k | 1.08 (3.0k/s) | 1.08 (3.0k/s) | 1.08 (3.0k/s) | 1.08 (3.0k/s) | **1.22 (3.3k/s)** | – (–k/s) | – (–k/s) | ‖ | **1.09 (3.0k/s)** | **1.08 (3.0k/s)** |
|
||||
|
||||
**proxy CPU time per gigabyte or per request**
|
||||
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 158 µs/req | 192 µs/req | 146 µs/req | **131 µs/req** | 623 µs/req | 462 µs/req | ‖ | 148 µs/req | **121 µs/req** |
|
||||
| mix-connect | **125 µs/req** | **137 µs/req** | – µs/req | – µs/req | 372 µs/req | 145 µs/req | ‖ | **114 µs/req** | – µs/req |
|
||||
| mix-socks | 129 µs/req | **123 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | ‖ | **115 µs/req** | – µs/req |
|
||||
| mix-relay | 128 µs/req | **122 µs/req** | 131 µs/req | **105 µs/req** | – µs/req | – µs/req | ‖ | **113 µs/req** | 123 µs/req |
|
||||
| mix-http-3k | 217 µs/req | 238 µs/req | **168 µs/req** | 187 µs/req | 647 µs/req | 489 µs/req | ‖ | 201 µs/req | **160 µs/req** |
|
||||
| mix-connect-3k | **175 µs/req** | **172 µs/req** | – µs/req | – µs/req | 358 µs/req | 189 µs/req | ‖ | **171 µs/req** | – µs/req |
|
||||
| mix-socks-3k | **181 µs/req** | **171 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | ‖ | **176 µs/req** | – µs/req |
|
||||
| mix-relay-3k | 174 µs/req | 170 µs/req | 166 µs/req | **162 µs/req** | – µs/req | – µs/req | ‖ | 171 µs/req | **159 µs/req** |
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 178 µs/req | 186 µs/req | 140 µs/req | **129 µs/req** | 188 µs/req | 630 µs/req | 436 µs/req | ‖ | 155 µs/req | **115 µs/req** |
|
||||
| mix-connect | **127 µs/req** | **126 µs/req** | – µs/req | – µs/req | – µs/req | 362 µs/req | **137 µs/req** | ‖ | **113 µs/req** | – µs/req |
|
||||
| mix-socks | **127 µs/req** | **122 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | – µs/req | ‖ | **115 µs/req** | – µs/req |
|
||||
| mix-relay | 122 µs/req | 116 µs/req | 133 µs/req | **104 µs/req** | 117 µs/req | – µs/req | – µs/req | ‖ | **110 µs/req** | 113 µs/req |
|
||||
| mix-http-3k | 217 µs/req | 223 µs/req | **166 µs/req** | 181 µs/req | 236 µs/req | 626 µs/req | 459 µs/req | ‖ | 204 µs/req | **154 µs/req** |
|
||||
| mix-connect-3k | **171 µs/req** | **167 µs/req** | – µs/req | – µs/req | – µs/req | 358 µs/req | 185 µs/req | ‖ | **160 µs/req** | – µs/req |
|
||||
| mix-socks-3k | 179 µs/req | **165 µs/req** | – µs/req | – µs/req | – µs/req | – µs/req | – µs/req | ‖ | **167 µs/req** | – µs/req |
|
||||
| mix-relay-3k | 173 µs/req | 167 µs/req | **162 µs/req** | **159 µs/req** | **158 µs/req** | – µs/req | – µs/req | ‖ | 166 µs/req | **154 µs/req** |
|
||||
|
||||
**proxy peak memory, MB**
|
||||
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 2,432 | 2,828 | 486 | **222** | 786 | 639 | ‖ | 2,440 | **428** |
|
||||
| mix-connect | 2,657 | 3,293 | – | – | **1,066** | 1,812 | ‖ | **2,692** | – |
|
||||
| mix-socks | **1,412** | 1,985 | – | – | – | – | ‖ | **1,445** | – |
|
||||
| mix-relay | 1,298 | 1,794 | **479** | 648 | – | – | ‖ | 1,328 | **334** |
|
||||
| mix-http-3k | 2,644 | 2,820 | 179 | **154** | 765 | 868 | ‖ | 2,659 | **180** |
|
||||
| mix-connect-3k | 2,648 | 2,834 | – | – | **1,038** | 1,371 | ‖ | **2,660** | – |
|
||||
| mix-socks-3k | **1,398** | 1,574 | – | – | – | – | ‖ | **1,410** | – |
|
||||
| mix-relay-3k | 1,358 | 1,531 | **314** | 604 | – | – | ‖ | 1,376 | **318** |
|
||||
| scenario | devel -s0 | 1.0.1 | haproxy | nginx | envoy | tinyproxy | squid | ‖ | devel | haproxy splice-auto |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|:-:|---:|---:|
|
||||
| mix-http | 1,765 | 2,851 | 489 | **220** | 609 | 786 | 650 | ‖ | 1,813 | **434** |
|
||||
| mix-connect | 1,411 | 3,280 | – | – | – | **1,062** | 1,839 | ‖ | **1,442** | – |
|
||||
| mix-socks | **1,408** | 2,157 | – | – | – | – | – | ‖ | **1,444** | – |
|
||||
| mix-relay | 1,289 | 1,809 | **477** | 662 | 723 | – | – | ‖ | 1,324 | **340** |
|
||||
| mix-http-3k | 1,622 | 2,817 | 172 | **153** | 414 | 765 | 613 | ‖ | 1,625 | **179** |
|
||||
| mix-connect-3k | 1,403 | 2,837 | – | – | – | **1,034** | 1,378 | ‖ | **1,416** | – |
|
||||
| mix-socks-3k | **1,400** | 1,573 | – | – | – | – | – | ‖ | **1,413** | – |
|
||||
| mix-relay-3k | 1,357 | 1,529 | **315** | 625 | 614 | – | – | ‖ | 1,372 | **321** |
|
||||
|
||||
* Copying many small exchanges devel does 75-80% of nginx and haproxy on ARM
|
||||
and 85-95% on Intel, where it is first through CONNECT and SOCKS; at a fixed rate it takes 3-8% more CPU than they do as a relay, 7-25%
|
||||
more as an HTTP proxy, with the same p99.
|
||||
* Memory is where a thread per connection shows: about 65 KB a connection for
|
||||
a relay or a SOCKS tunnel (the thread's stack and two 16 KB buffers), about
|
||||
130 KB for an HTTP proxy or CONNECT connection, against 8-30 KB for nginx and
|
||||
haproxy.
|
||||
* nginx relays fail exchanges whose client shuts down its sending side first
|
||||
(`proxy_half_close` is off by default), 350-470 a run; with it on
|
||||
(`nginx::halfclose`, [data/](data/) `*-halfclose.csv`) there are none and
|
||||
nothing else changes: 7.47k against 7.44k on ARM, 9.83k against 9.71k on
|
||||
Intel.
|
||||
* devel itself lost 2-11 exchanges a run in the tunnels and relays, every
|
||||
one half-closed by the client with a large answer, cut short. Once the
|
||||
server had answered and closed, `sockmap()` took the hang-up poll() reported
|
||||
on its socket for the end of it while the rest of the answer was still
|
||||
unread there, if its buffer was full at the time; fixed since.
|
||||
* Many small exchanges: devel does 63% of nginx's as an HTTP proxy on ARM,
|
||||
74% of haproxy `splice-auto`'s on Intel, and 81-95% of nginx's as a relay;
|
||||
through CONNECT and SOCKS it leads on Intel. At a fixed rate it takes 8-23%
|
||||
more CPU than the best as a relay, 32-49% more as an HTTP proxy; through
|
||||
CONNECT 5% more than squid on ARM and 14% less on Intel, through SOCKS 41%
|
||||
less than dante.
|
||||
* Memory is where a thread per connection shows: 65-75 KB a connection for a
|
||||
relay or a SOCKS tunnel (the thread's stack and two 16 KB buffers), 70-90 KB
|
||||
for an HTTP proxy or CONNECT connection, against 8-36 KB for nginx, haproxy
|
||||
and envoy. That is 30-45% less than in the morning session (120-140 KB
|
||||
for HTTP and CONNECT): devel now frees the request buffer of a kept-alive
|
||||
connection idle for a second, and before relaying a CONNECT tunnel.
|
||||
* No exchange failed through devel, or through nginx relaying with
|
||||
`proxy_half_close on`. 1.0.1 still loses 1-5 a run in the tunnels and
|
||||
relays, exchanges half-closed by the client with a large answer cut short,
|
||||
the bug fixed in devel (and in master and 0.9) since the morning session.
|
||||
* envoy at the fixed rate on Intel relays more than 3000 a second: it fell
|
||||
behind at times (p99 0.7-1.2 s) and the client made up the delayed
|
||||
exchanges.
|
||||
|
||||
## Earlier sessions
|
||||
|
||||
The morning session of 13 October ([data/](data/) `*-16k-*.csv`, `*-mix.csv`)
|
||||
ran devel before the half-close fix, the idle buffer release and the TLS
|
||||
changes, 1.0.1 without TCP_NODELAY and nginx relays without
|
||||
`proxy_half_close`, and had no envoy. Its ARM figures are 5-15% lower for
|
||||
every proxy alike, probably because two stray 3proxy processes were spinning
|
||||
on the host's CPUs at the time.
|
||||
|
||||
The session of 11-12 October, devel with 64 KB buffers, splice() by direction
|
||||
(`-s1`, `-s3`) and the automatic mode as an option, against 0.9.4, 1.0.1 and
|
||||
the same proxies, is in [data/](data/) (`*-auto-http.csv`, `*-tunnels.csv`).
|
||||
|
||||
Reference in new issue
Block a user