bench: TLS streams, envoy, a table for each machine

tls-term-stream* and tls-init-stream* run iperf3 streams through the TLS
relays. iperf3 has no TLS of its own, so bench/mix gets a TLS wrapper
mode, run on the client to begin TLS towards a proxy ending it and at the
origin to end the TLS a proxy begins.

--proxy envoy measures envoy where nginx and haproxy are measured: as an
HTTP proxy for the origin, as a TCP relay and in the TLS scenarios, its
circuit breakers lifted for the scenarios opening more than 1024
connections.

report.py --machine with a single machine gives its medians alone, with
the client- or origin-limited ones marked; --overall marks only the best
of each row. The mix client's -errlog prints every failed exchange.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bench/README.md

@@ -89,6 +89,7 @@ host) falls on all of them alike; the summary gives medians.
 | `mix-*-3k` | the same, exchanges started at 3000 a second whether the proxy keeps up or not: latencies compared at equal load |
 | `tls-term-ka-1k`, `tls-term-ka-1m`, `tls-term-close-1k` | TLS ended by the proxy: wrk speaks HTTPS to a relay of the proxy, which passes plain TCP to the origin; kept-alive connections with 1 KB and 1 MB answers, and a new connection, so a full handshake, for every request |
 | `tls-init-*` | TLS begun by the proxy: wrk speaks plain HTTP to a relay, which connects to the origin's HTTPS listener |
+| `tls-term-stream1-down`, `tls-term-stream16`, `tls-init-stream1-down`, `tls-init-stream16` | the same with iperf3 streams, one downloading or sixteen uploading |
 
 ### The mix scenarios
 
@@ -146,10 +147,11 @@ answer, and `peak_mb` divided by the pool what each open connection costs the
 proxy. The binaries are built with `bench/mix/build.sh`, on any system with
 Go, for both Linux architectures.
 
-nginx is not a forward proxy: it answers absolute URIs by passing them to its
+nginx and envoy are not forward proxies: they answer absolute URIs by passing
-upstream, which is the origin here. Neither nginx nor haproxy has CONNECT or
+them to their upstream, which is the origin here. nginx and haproxy have
-SOCKS, so they take part in the HTTP and `relay-*` scenarios but not in the
+neither CONNECT nor SOCKS, and envoy is measured as they are, so they take
-`connect-*` and `socks-*` ones. tinyproxy
+part in the HTTP and `relay-*` scenarios but not in the `connect-*` and
+`socks-*` ones. tinyproxy
 and squid are HTTP proxies, measured in the HTTP and `connect-*` scenarios;
 dante is a SOCKS server, measured in the `socks-*` ones.
 
@@ -160,10 +162,18 @@ The `tls-*` scenarios compare TLS done in the proxy: 3proxy (`tcppm` under
 `proxy_ssl on`), haproxy (`mode tcp` with `bind ... ssl crt` and `server ...
 ssl`) and stunnel, which takes part in these scenarios only. Every one relays
 TCP: no HTTP is read, so what differs is the TLS and how data is moved around
-it. The certificate is made for every run: an ECDSA P-256 key signed by a CA
+it. envoy takes part too, with BoringSSL where the others use OpenSSL. The certificate is made for every run: an ECDSA P-256 key signed by a CA
 of its own, with the proxy's and the origin's addresses in it; nobody
 verifies it. `none` has wrk speak HTTPS to the origin itself.
 
+iperf3 has no TLS of its own, so for the TLS streams `bench/mix tls` wraps
+its connections: on the client it begins TLS towards the proxy ending it (or
+towards the origin with no proxy), at the origin it ends the TLS a proxy
+begins and passes the stream to iperf3. The wrapper is in Go, a goroutine
+for each direction of each connection: on one stream it, rather than the
+proxy, can be the limit (about 10 Gbit/s a core), which the run's `bound`
+shows.
+
 wrk does not resume TLS sessions, so in `tls-term-close-1k` every connection
 is a full handshake. Towards the origin a proxy may resume them: nginx
 (`proxy_ssl_session_reuse`) and haproxy do by default, so `tls-init-close-1k`
@@ -194,7 +204,9 @@ sudo bench/bench.py \
 `bench/report.py <csv>` turns a run into Markdown tables: throughput, the
 proxy's CPU per gigabyte or per request, and its peak memory. With
 `--machine NAME=a.csv,b.csv` once per machine it makes a single throughput
-table of all of them, each cell their medians and their average.
+table of all of them, each cell their medians and their average; `--overall`
+then marks only the best of each row, for tables where one group has a
+single proxy.
 
 After the binary, a 3proxy entry can carry options for both its services:
 `-s1`, `-s2` and `-s3` splice downloads, uploads or both (`-s0` keeps 0.9.0 to
@@ -212,7 +224,12 @@ variant; `nginx::halfclose` turns on `proxy_half_close` in its relays, which
 nginx leaves off: then a client shutting down its sending side still gets the
 answer, as through 3proxy and haproxy.
 
-`--proxy stunnel` measures stunnel in the `tls-*` scenarios.
+`--proxy stunnel` measures stunnel in the `tls-*` scenarios. `--proxy envoy`
+measures envoy where nginx and haproxy are measured: as an HTTP proxy for the
+origin (its connections there pooled), as a TCP relay (`tcp_proxy`) and in
+the TLS scenarios; its circuit breakers, which allow 1024 connections to a
+cluster by default, are lifted for the scenarios that open more. The
+official release binaries run as they are (`envoy` on the PATH).
 
 `--scenario` picks scenarios, `--reps` and `--duration` set the repetitions
 and their length, `--netem "delay 1ms rate 10gbit"` puts a slower, longer link

bench/bench.py

@@ -51,6 +51,10 @@ MIX_PORTS = 16
 # (TLS from the client, plain to ORIGIN_HTTP) and initiate it (plain from the
 # client, TLS to ORIGIN_TLS). TLS holds the certificate files once made.
 ORIGIN_TLS, TLS_TERM, TLS_INIT = 8443, 5443, 5444
+# the same for TLS streams: iperf3 has no TLS, so bench/mix wraps its
+# connections, on the client (CLI_TLS, for TLS ended by the proxy) and at the
+# origin (ORIGIN_TLS_TCP, taking the TLS begun by the proxy)
+ORIGIN_TLS_TCP, TLS_TERM_TCP, TLS_INIT_TCP, CLI_TLS = 8444, 5445, 5446, 5202
 TLS = None
 # Each measurement takes the next of SLOTS sets of addresses: the proxy's to
 # connect from, and the mix client's to connect from. What a run leaves in
@@ -107,6 +111,10 @@ SCENARIOS = {
     "tls-init-ka-1k":    ("http", {"file": "1k", "conns": 64, "via": "tls-init"}),
     "tls-init-ka-1m":    ("http", {"file": "1m", "conns": 16, "via": "tls-init"}),
     "tls-init-close-1k": ("http", {"file": "1k", "conns": 64, "close": True, "via": "tls-init"}),
+    "tls-term-stream1-down": ("stream", {"streams": 1, "reverse": True, "via": "tls-term"}),
+    "tls-term-stream16": ("stream", {"streams": 16, "via": "tls-term"}),
+    "tls-init-stream1-down": ("stream", {"streams": 1, "reverse": True, "via": "tls-init"}),
+    "tls-init-stream16": ("stream", {"streams": 16, "via": "tls-init"}),
     # a pool of open connections and exchanges on random ones of them, sizes
     # and pieces from a web-like profile (bench/mix): -active of them going
     # at once all the time, or started at a fixed rate whatever the proxy
@@ -445,6 +453,11 @@ http {{
     procs = [spawn("bsrv", "srv", [tool("nginx"), "-c", conf, "-p", work], log),
              spawn("bsrv", "srv", [tool("iperf3"), "-s", "-B", SRV_ADDR, "-p", str(ORIGIN_TCP)], log)]
     ports = [ORIGIN_HTTP, ORIGIN_TCP] + ([ORIGIN_TLS] if TLS else [])
+    if args.wrap:
+        procs.append(spawn("bsrv", "srv", [args.wrap, "tls", "-listen", f"{SRV_ADDR}:{ORIGIN_TLS_TCP}",
+                                           "-connect", f"{SRV_ADDR}:{ORIGIN_TCP}",
+                                           "-cert", TLS["cert"], "-key", TLS["key"]], log))
+        ports.append(ORIGIN_TLS_TCP)
     if args.mix:
         procs.append(spawn("bsrv", "srv", [args.mix, "origin", "-listen", f"{SRV_ADDR}:{ORIGIN_MIX}",
                                            "-ports", str(MIX_PORTS)], log))
@@ -458,6 +471,64 @@ http {{
 
 # --- proxies -----------------------------------------------------------------
 
+def envoy_conf(work, label, out):
+    """envoy as nginx and haproxy are measured: an HTTP proxy for the origin
+    (absolute URIs taken, the origin's connections pooled), TCP relays, and
+    the TLS ones. Its default circuit breakers allow 1024 connections to a
+    cluster, far fewer than the scenarios open: they are lifted."""
+    def addr(host, port):
+        return {"socket_address": {"address": host, "port_value": port}}
+
+    def cluster(name, port, tls=False):
+        c = {"name": name, "type": "STATIC", "connect_timeout": "5s",
+             "upstream_bind_config": {"source_address": {"address": out, "port_value": 0}},
+             "circuit_breakers": {"thresholds": [{"max_connections": 1000000,
+                                                  "max_pending_requests": 1000000,
+                                                  "max_requests": 1000000}]},
+             "load_assignment": {"cluster_name": name, "endpoints": [{"lb_endpoints": [
+                 {"endpoint": {"address": addr(SRV_ADDR, port)}}]}]}}
+        if tls:
+            c["transport_socket"] = {"name": "envoy.transport_sockets.tls", "typed_config": {
+                "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext"}}
+        return c
+
+    def http(name, port, dest):
+        return {"name": name, "address": addr(PRX_IN, port), "filter_chains": [{"filters": [{
+            "name": "envoy.filters.network.http_connection_manager", "typed_config": {
+                "@type": "type.googleapis.com/envoy.extensions.filters.network."
+                         "http_connection_manager.v3.HttpConnectionManager",
+                "stat_prefix": name, "codec_type": "HTTP1",
+                "route_config": {"virtual_hosts": [{"name": "all", "domains": ["*"], "routes": [
+                    {"match": {"prefix": "/"}, "route": {"cluster": dest, "timeout": "0s"}}]}]},
+                "http_filters": [{"name": "envoy.filters.http.router", "typed_config": {
+                    "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"}}]}}]}]}
+
+    def tcp(name, port, dest, tls=False):
+        chain = {"filters": [{"name": "envoy.filters.network.tcp_proxy", "typed_config": {
+            "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
+            "stat_prefix": name, "cluster": dest}}]}
+        if tls:
+            chain["transport_socket"] = {"name": "envoy.transport_sockets.tls", "typed_config": {
+                "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
+                "common_tls_context": {"tls_certificates": [{
+                    "certificate_chain": {"filename": TLS["cert"]},
+                    "private_key": {"filename": TLS["key"]}}]}}}
+        return {"name": name, "address": addr(PRX_IN, port), "filter_chains": [chain]}
+
+    listeners = [http("http", HTTP_PORT, "origin"), http("httpmix", HTTP_MIX, "mix"),
+                 tcp("tcp", TCP_PORT, "tcp"), tcp("tcphttp", RELAY_HTTP, "origin"),
+                 tcp("tcpmix", RELAY_MIX, "mix")]
+    clusters = [cluster("origin", ORIGIN_HTTP), cluster("mix", ORIGIN_MIX), cluster("tcp", ORIGIN_TCP)]
+    if TLS:
+        listeners += [tcp("tlsterm", TLS_TERM, "origin", True), tcp("tlstermtcp", TLS_TERM_TCP, "tcp", True),
+                      tcp("tlsinit", TLS_INIT, "origintls"), tcp("tlsinittcp", TLS_INIT_TCP, "tcptls")]
+        clusters += [cluster("origintls", ORIGIN_TLS, True), cluster("tcptls", ORIGIN_TLS_TCP, True)]
+    conf = os.path.join(work, f"{label}.json")
+    with open(conf, "w") as f:
+        json.dump({"static_resources": {"listeners": listeners, "clusters": clusters}}, f, indent=1)
+    return conf
+
+
 def proxy_conf(kind, work, ncpu, label, opts="", out=PRX_OUT):
     """Configuration for one proxy: an HTTP proxy on HTTP_PORT and a TCP relay
     on TCP_PORT, both towards the origin. Returns the argv. opts are added to
@@ -479,12 +550,17 @@ socks {opts} -i{PRX_IN} -e{out} -p{SOCKS_PORT}
 ssl_server_key {TLS["key"]}
 ssl_serv
 tcppm {opts} -i{PRX_IN} -e{out} {TLS_TERM} {SRV_ADDR} {ORIGIN_HTTP}
+tcppm {opts} -i{PRX_IN} -e{out} {TLS_TERM_TCP} {SRV_ADDR} {ORIGIN_TCP}
 ssl_noserv
 ssl_cli
 tcppm {opts} -i{PRX_IN} -e{out} {TLS_INIT} {SRV_ADDR} {ORIGIN_TLS}
+tcppm {opts} -i{PRX_IN} -e{out} {TLS_INIT_TCP} {SRV_ADDR} {ORIGIN_TLS_TCP}
 ssl_nocli
 """)
         return [conf]
+    if kind == "envoy":
+        return ["-c", envoy_conf(work, label, out), "--concurrency", str(ncpu),
+                "--log-level", "error", "--disable-hot-restart"]
     if kind == "stunnel":
         conf = os.path.join(work, f"{label}.conf")
         with open(conf, "w") as f:
@@ -503,6 +579,17 @@ client = yes
 accept = {PRX_IN}:{TLS_INIT}
 connect = {SRV_ADDR}:{ORIGIN_TLS}
 local = {out}
+[termtcp]
+accept = {PRX_IN}:{TLS_TERM_TCP}
+connect = {SRV_ADDR}:{ORIGIN_TCP}
+local = {out}
+cert = {TLS["cert"]}
+key = {TLS["key"]}
+[inittcp]
+client = yes
+accept = {PRX_IN}:{TLS_INIT_TCP}
+connect = {SRV_ADDR}:{ORIGIN_TLS_TCP}
+local = {out}
 """)
         return [conf]
     if kind == "tinyproxy":
@@ -565,12 +652,15 @@ socks pass {{ from: 0.0.0.0/0 to: 0.0.0.0/0 }}
             halfclose = "    proxy_half_close on;\n" if opts == "halfclose" else ""
             nginx_tls = ""
             if TLS:
-                nginx_tls = (f"    server {{\n        listen {PRX_IN}:{TLS_TERM} ssl backlog=65535 reuseport;\n"
+                nginx_tls = ""
-                             f"        ssl_certificate {TLS['cert']};\n        ssl_certificate_key {TLS['key']};\n"
+                for port, dest in ((TLS_TERM, ORIGIN_HTTP), (TLS_TERM_TCP, ORIGIN_TCP)):
-                             f"        proxy_pass {SRV_ADDR}:{ORIGIN_HTTP};\n        proxy_bind {out};\n    }}\n"
+                    nginx_tls += (f"    server {{\n        listen {PRX_IN}:{port} ssl backlog=65535 reuseport;\n"
-                             f"    server {{\n        listen {PRX_IN}:{TLS_INIT} backlog=65535 reuseport;\n"
+                                  f"        ssl_certificate {TLS['cert']};\n        ssl_certificate_key {TLS['key']};\n"
-                             f"        proxy_pass {SRV_ADDR}:{ORIGIN_TLS};\n        proxy_ssl on;\n"
+                                  f"        proxy_pass {SRV_ADDR}:{dest};\n        proxy_bind {out};\n    }}\n")
-                             f"        proxy_bind {out};\n    }}\n")
+                for port, dest in ((TLS_INIT, ORIGIN_TLS), (TLS_INIT_TCP, ORIGIN_TLS_TCP)):
+                    nginx_tls += (f"    server {{\n        listen {PRX_IN}:{port} backlog=65535 reuseport;\n"
+                                  f"        proxy_pass {SRV_ADDR}:{dest};\n        proxy_ssl on;\n"
+                                  f"        proxy_bind {out};\n    }}\n")
             f.write(f"""{load}daemon off;
 worker_processes {ncpu};
 pid {work}/{label}.pid;
@@ -691,6 +781,16 @@ listen tlsinit
     bind {PRX_IN}:{TLS_INIT}
     source {out}
     server origin {SRV_ADDR}:{ORIGIN_TLS} ssl verify none
+listen tlstermtcp
+    mode tcp
+    bind {PRX_IN}:{TLS_TERM_TCP} ssl crt {TLS["pem"]}
+    source {out}
+    server origin {SRV_ADDR}:{ORIGIN_TCP}
+listen tlsinittcp
+    mode tcp
+    bind {PRX_IN}:{TLS_INIT_TCP}
+    source {out}
+    server origin {SRV_ADDR}:{ORIGIN_TLS_TCP} ssl verify none
 """)
         return ["-f", conf]
     raise ValueError(kind)
@@ -715,6 +815,9 @@ class Proxy:
                 sys.exit("nginx options: halfclose")
             self.binary = path or tool(kind)
             self.label = kind + ("-" + self.opts if self.opts else "")
+        elif kind == "envoy":
+            self.binary = path or tool("envoy")
+            self.label = kind
         elif kind in ("tinyproxy", "squid", "dante", "stunnel"):
             # dante's server is sockd, danted on Debian; stunnel stunnel4 there
             self.binary = path or (tool("sockd", "danted") if kind == "dante" else
@@ -731,12 +834,14 @@ class Proxy:
     PORTS = {"3proxy": (HTTP_PORT, TCP_PORT, SOCKS_PORT, RELAY_HTTP, RELAY_MIX),
              "nginx": (HTTP_PORT, TCP_PORT, RELAY_HTTP, HTTP_MIX, RELAY_MIX),
              "haproxy": (HTTP_PORT, TCP_PORT, RELAY_HTTP, HTTP_MIX, RELAY_MIX),
+             "envoy": (HTTP_PORT, TCP_PORT, RELAY_HTTP, HTTP_MIX, RELAY_MIX),
              "tinyproxy": (HTTP_PORT,), "squid": (HTTP_PORT,), "dante": (SOCKS_PORT,),
-             "stunnel": (TLS_TERM, TLS_INIT)}
+             "stunnel": (TLS_TERM, TLS_INIT, TLS_TERM_TCP, TLS_INIT_TCP)}
 
     def ports(self):
         """What to wait for: with TLS scenarios the relays for them too."""
-        extra = (TLS_TERM, TLS_INIT) if TLS and self.kind in ("3proxy", "nginx", "haproxy") else ()
+        extra = (TLS_TERM, TLS_INIT, TLS_TERM_TCP, TLS_INIT_TCP) \
+            if TLS and self.kind in ("3proxy", "nginx", "haproxy", "envoy") else ()
         return self.PORTS[self.kind] + extra
 
     def serves(self, scenario):
@@ -745,14 +850,14 @@ class Proxy:
         # TLS ended or begun on a relay: what 3proxy, nginx, haproxy and
         # stunnel do, and only stunnel
         if via in ("tls-term", "tls-init"):
-            return self.kind in ("3proxy", "nginx", "haproxy", "stunnel", "none")
+            return self.kind in ("3proxy", "nginx", "haproxy", "envoy", "stunnel", "none")
         if self.kind == "stunnel":
             return False
         if self.kind == "dante":
             return via == "socks"
         if self.kind in ("tinyproxy", "squid"):
             return via == "connect" or (kind in ("http", "mix") and not via)
-        if self.kind in ("nginx", "haproxy"):
+        if self.kind in ("nginx", "haproxy", "envoy"):
             # no CONNECT, no SOCKS
             return via in (None, "relay")
         return True
@@ -876,12 +981,38 @@ def run_http(proxy, params, args, work, mark):
 
 
 def run_stream(proxy, params, args, work, mark):
-    host, port, prefix = target(proxy, params, work, ORIGIN_TCP, TCP_PORT, TCP_PORT)
+    via = params.get("via")
+    wrapper = None
+    if via in ("tls-term", "tls-init"):
+        # iperf3 talks to a wrapper on the client; with TLS ended by the
+        # proxy, or with no proxy, it begins TLS there, and the origin's
+        # wrapper ends it; with TLS begun by the proxy it passes plain TCP
+        if proxy.kind == "none":
+            dest, tls = f"{SRV_ADDR}:{ORIGIN_TLS_TCP}", True
+        elif via == "tls-term":
+            dest, tls = f"{PRX_IN}:{TLS_TERM_TCP}", True
+        else:
+            dest, tls = f"{PRX_IN}:{TLS_INIT_TCP}", False
+        if tls:
+            log = open(os.path.join(work, "wrap.out"), "a")
+            wrapper = spawn("bcli", "cli", [args.wrap, "tls", "-listen", f"{CLI_ADDR}:{CLI_TLS}",
+                                            "-connect", dest], log)
+            wait_port("bcli", CLI_ADDR, CLI_TLS, 10)
+            host, port = CLI_ADDR, CLI_TLS
+        else:
+            host, port = PRX_IN, TLS_INIT_TCP
+        prefix = []
+    else:
+        host, port, prefix = target(proxy, params, work, ORIGIN_TCP, TCP_PORT, TCP_PORT)
     argv = prefix + [tool("iperf3"), "-c", host, "-p", str(port), "-t", str(args.duration),
             "-P", str(params["streams"]), "-J"]
     if params.get("reverse"):
         argv.append("-R")
-    out = run_in("bcli", "cli", argv, args.duration + 30)
+    try:
+        out = run_in("bcli", "cli", argv, args.duration + 30)
+    finally:
+        if wrapper:
+            stop(wrapper)
     res = {"raw": out, "rate": float("nan"), "p50_ms": float("nan"),
            "p99_ms": float("nan"), "requests": 0, "errors": 0, "retrans": 0, "timeouts": 0}
     try:
@@ -908,7 +1039,7 @@ def run_mix(proxy, params, args, work, mark):
         addr = f"{PRX_IN}:{RELAY_MIX}"
     elif via == "socks":
         addr = f"{PRX_IN}:{SOCKS_PORT}"
-    elif via == "http" and proxy.kind in ("nginx", "haproxy"):
+    elif via == "http" and proxy.kind in ("nginx", "haproxy", "envoy"):
         addr = f"{PRX_IN}:{HTTP_MIX}"
     else:
         addr = f"{PRX_IN}:{HTTP_PORT}"
@@ -1088,6 +1219,14 @@ def main():
             sys.exit(f"{args.mix} is missing: build it with bench/mix/build.sh")
     else:
         args.mix = None
+    # the TLS streams need its TLS wrapper
+    args.wrap = None
+    if any(SCENARIOS[s][0] == "stream" and SCENARIOS[s][1].get("via") in ("tls-term", "tls-init")
+           for s in scenarios):
+        arch = {"x86_64": "amd64", "aarch64": "arm64"}.get(os.uname().machine, os.uname().machine)
+        args.wrap = os.path.abspath(os.path.join(os.path.dirname(__file__), "mix", f"mix-linux-{arch}"))
+        if not os.access(args.wrap, os.X_OK):
+            sys.exit(f"{args.wrap} is missing: build it with bench/mix/build.sh")
 
     stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
     global TLS

bench/mix/main.go

@@ -26,6 +26,7 @@ package main
 import (
 	"bufio"
 	"bytes"
+	"crypto/tls"
 	"encoding/binary"
 	"encoding/json"
 	"errors"
@@ -45,6 +46,62 @@ import (
 	"time"
 )
 
+// --- a TLS wrapper ---------------------------------------------------------------
+
+// wrap relays connections to listen to target, with TLS on one side: given a
+// certificate it takes TLS and passes plain TCP on (the end of a TLS stream
+// at the origin), without one it takes plain TCP and begins TLS (for a client
+// with no TLS of its own, iperf3). A side shut down for writing is shut down
+// on the other side too, so half-closed connections work.
+func wrap(listen, target, cert, key string) error {
+	var server *tls.Config
+	if cert != "" {
+		pair, err := tls.LoadX509KeyPair(cert, key)
+		if err != nil {
+			return err
+		}
+		server = &tls.Config{Certificates: []tls.Certificate{pair}}
+	}
+	ln, err := net.Listen("tcp", listen)
+	if err != nil {
+		return err
+	}
+	for {
+		c, err := ln.Accept()
+		if err != nil {
+			return err
+		}
+		go func() {
+			var in, out net.Conn = c, nil
+			var err error
+			if server != nil {
+				in = tls.Server(c, server)
+				out, err = net.Dial("tcp", target)
+			} else {
+				out, err = tls.Dial("tcp", target, &tls.Config{InsecureSkipVerify: true})
+			}
+			if err != nil {
+				c.Close()
+				return
+			}
+			done := make(chan struct{}, 2)
+			pass := func(dst, src net.Conn) {
+				io.Copy(dst, src)
+				if cw, ok := dst.(interface{ CloseWrite() error }); ok {
+					cw.CloseWrite()
+				}
+				done <- struct{}{}
+			}
+			go pass(out, in)
+			go pass(in, out)
+			<-done
+			<-done
+			in.Close()
+			out.Close()
+		}()
+	}
+}
+
 // --- the profile -------------------------------------------------------------
 
 // lognormal draws from a lognormal distribution by its median and sigma,
@@ -332,6 +389,7 @@ type client struct {
 	aborts, resets, halfcloses              atomic.Uint64
 	complete, first                         histogram
 	dialErr, exchErr                        atomic.Value // the first of each, as text
+	errlog                                  bool         // every failed exchange to stderr
 }
 
 // shard is a share of the pool with a dispatcher of its own: one dispatcher
@@ -470,8 +528,10 @@ func (cl *client) release(c int) {
 func (cl *client) worker(id int, c net.Conn) {
 	scratch := make([]byte, 64<<10)
 	head := make([]byte, 0, 4096)
+	nth := 0 // exchanges on this connection so far
 	for j := range cl.jobs[id] {
 		if c == nil {
+			nth = 0
 			var err error
 			if c, err = cl.dial(id); err != nil {
 				cl.dialErr.CompareAndSwap(nil, err.Error())
@@ -488,6 +548,7 @@ func (cl *client) worker(id int, c net.Conn) {
 			}
 		}
 		n, err := cl.exchange(c, id, j, head, scratch)
+		nth++
 		j.slot <- struct{}{}
 		if err == errGone {
 			// closed as drawn, with a reset or not
@@ -509,6 +570,9 @@ func (cl *client) worker(id int, c net.Conn) {
 		if err != nil || j.e.close || halfclosed {
 			c.Close()
 			c = nil
+			if err != nil && cl.errlog {
+				fmt.Fprintf(os.Stderr, "error: connection %d, exchange %d on it: %v\n", id, nth, err)
+			}
 			if err != nil && cl.measuring.Load() {
 				cl.exchErr.CompareAndSwap(nil, err.Error())
 				var ne net.Error
@@ -567,7 +631,7 @@ func (cl *client) exchange(c net.Conn, id int, j job, head, scratch []byte) (int
 	}
 	head = append(head, "\r\n"...)
 	if err := send(c, r, clientWrites, head, e.upload); err != nil {
-		return 0, err
+		return 0, fmt.Errorf("sending (end %d, upload %d): %w", e.end, e.upload, err)
 	}
 	if halfclose {
 		if cw, ok := c.(interface{ CloseWrite() error }); ok {
@@ -624,7 +688,8 @@ func (cl *client) exchange(c net.Conn, id int, j job, head, scratch []byte) (int
 			if want >= 0 && body >= want {
 				break
 			}
-			return 0, err
+			return 0, fmt.Errorf("reading (end %d, close %v, upload %d, answer %d, got %d, body %d of %d): %w",
+				e.end, e.close, e.upload, e.answer, got, body, want, err)
 		}
 		if d := readPause(r); d > 0 {
 			time.Sleep(d)
@@ -794,7 +859,7 @@ func runClient(cl *client, active int, rate float64, warmup, duration time.Durat
 
 func main() {
 	if len(os.Args) < 2 {
-		fmt.Fprintln(os.Stderr, "usage: mix origin|client [flags]")
+		fmt.Fprintln(os.Stderr, "usage: mix origin|client|tls [flags]")
 		os.Exit(2)
 	}
 	fs := flag.NewFlagSet(os.Args[1], flag.ExitOnError)
@@ -807,6 +872,16 @@ func main() {
 			fmt.Fprintln(os.Stderr, err)
 			os.Exit(1)
 		}
+	case "tls":
+		listen := fs.String("listen", "127.0.0.1:5443", "address to take connections on")
+		target := fs.String("connect", "127.0.0.1:5201", "address to pass them to")
+		cert := fs.String("cert", "", "certificate: take TLS and pass plain TCP on; none: the other way round")
+		key := fs.String("key", "", "the certificate's key")
+		fs.Parse(os.Args[2:])
+		if err := wrap(*listen, *target, *cert, *key); err != nil {
+			fmt.Fprintln(os.Stderr, err)
+			os.Exit(1)
+		}
 	case "client":
 		cl := &client{}
 		fs.StringVar(&cl.via, "via", "http", "http (an HTTP proxy), connect, socks, relay or direct")
@@ -815,6 +890,7 @@ func main() {
 		fs.IntVar(&cl.origins, "origins", 1, "the origin's ports to spread over (origin -ports)")
 		sources := fs.String("sources", "", "client addresses to open connections from, comma separated")
 		fs.IntVar(&cl.pool, "pool", 20000, "connections kept open")
+		fs.BoolVar(&cl.errlog, "errlog", false, "print every failed exchange to stderr")
 		active := fs.Int("active", 2000, "exchanges in progress at most")
 		rate := fs.Float64("rate", 0, "exchanges started per second, 0 to keep -active going all the time")
 		duration := fs.Duration("duration", 10*time.Second, "how long to measure")
@@ -848,7 +924,7 @@ func main() {
 		}
 		runClient(cl, *active, *rate, *warmup, *duration, *dialers, *shards)
 	default:
-		fmt.Fprintln(os.Stderr, "usage: mix origin|client [flags]")
+		fmt.Fprintln(os.Stderr, "usage: mix origin|client|tls [flags]")
 		os.Exit(2)
 	}
 }

bench/report.py

@@ -27,7 +27,8 @@ of the group whose range (the average of its slowest runs to that of its
 fastest) reaches the best one's; * marks the best of the row, both groups
 together, in the same way. --split puts the splice group in a table of its own
 (* still compares across both), --prefix connect- takes only the scenarios so
-named.
+named. --overall marks the best of the row only, in bold and with *: for
+tables where one group has a single proxy, its group's best says nothing.
 """
 
 import argparse
@@ -119,7 +120,7 @@ def load(paths):
 BULK = STREAM + ("http-ka-1m",)
 
 
-def combined(machines, labels, splice, prefix=None, split=False):
+def combined(machines, labels, splice, prefix=None, split=False, overall_only=False):
     """One throughput table for several machines: each cell their medians
     and their average. Bold: the best of its group; *: the best of the row.
     split puts the proxies using splice() in a table of their own, * still
@@ -175,7 +176,17 @@ def combined(machines, labels, splice, prefix=None, split=False):
     plain = [p for p in proxies if p != "direct" and p not in splice]
     spliced = [p for p in proxies if p in splice]
     direct = [p for p in proxies if p == "direct"]
-    names = " / ".join(n for n, _ in machines) + " / average"
+    # one machine: its medians alone, with ‡ where most runs were limited by
+    # the client or the origin; several: their medians and their average
+    single = len(machines) == 1
+    names = machines[0][0] if single else " / ".join(n for n, _ in machines) + " / average"
+
+    def limited(s, p):
+        if not single:
+            return ""
+        bounds = [r.get("bound", "") for r in data[machines[0][0]].get((s, p), [])]
+        other = sum(b in ("client", "origin") for b in bounds)
+        return " ‡" if bounds and other * 2 > len(bounds) else ""
 
     def render(tables):
         """tables: lists of groups, each table printed on its own; bold for
@@ -198,13 +209,15 @@ def combined(machines, labels, splice, prefix=None, split=False):
                 for i, g in enumerate(t):
                     if i:
                         cells.append("‖")
-                    best = leaders(s, g)
+                    best = overall if overall_only else leaders(s, g)
                     for p in g:
-                        vals = [value(n, s, p) for n, _ in machines] + [average(s, p)]
+                        vals = [value(n, s, p) for n, _ in machines]
+                        if not single:
+                            vals.append(average(s, p))
                         if all(v is None for v in vals):
                             cells.append("–")
                             continue
-                        text = " / ".join(fmt(v, 1) for v in vals)
+                        text = " / ".join(fmt(v, 1) for v in vals) + limited(s, p)
                         if p in best:
                             text = f"**{text}**"
                         if p in overall:
@@ -231,6 +244,8 @@ def main():
     ap.add_argument("--machine", action="append", default=[],
                     help="NAME=a.csv[,b.csv]: one throughput table for all the "
                          "machines given, instead of the tables for one")
+    ap.add_argument("--overall", action="store_true",
+                    help="with --machine: only the best of the row, bold and *")
     ap.add_argument("--label", action="append", default=[], help="old=new")
     ap.add_argument("--title", default="")
     ap.add_argument("--splice", action="append", default=[],
@@ -240,7 +255,8 @@ def main():
 
     if args.machine:
         combined([(m.split("=", 1)[0], m.split("=", 1)[1].split(","))
-                  for m in args.machine], labels, args.splice, args.prefix, args.split)
+                  for m in args.machine], labels, args.splice, args.prefix, args.split,
+                 args.overall)
         return
     rows = load(args.csv)
     proxies, scenarios = [], []