- readconfig() kept its line buffer and argument array on every error
return, so each refused reload leaked them.
- A filter refusing a connection in filter_client (TLS: a failed
handshake) was not kept, so its filter_clear never ran and the state
it made was lost. makefilters() now clears it, and empties the slot
first: a pooled worker's slot may hold the previous connection's.
- ssl_filter_close() freed the client CA paths but not the server ones,
for every TLS service set up.
- Compiled patterns were never freed: an ACL's (pcre: and regex: host
names), an httpsrv rule's. copyacl() shared them between the
configuration's ACL and each service's copy; each copy now compiles
its own, and freeacl() and freehttprules() free them.
What LeakSanitizer still reports is OpenSSL's per-thread random
generator state of pooled threads, released only by OpenSSL at exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A section on ssl_ktls: what it needs, how 3proxy reports it, and that
it pays with splice() for TLS connections carrying large transfers.
The splice() section gets the :MIN,STREAK,SMALL tuning and says TLS
services splice where the kernel does TLS. Grace delay is recommended
by the clients' speed for steady streams only: for web traffic it
saved nothing on any of the links measured, nor with the second CPU
the HTTP proxy needed on the 5 Mbit/s one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py takes ktls among 3proxy's options (ssl_ktls before the TLS
services) and for nginx (nginx::ktls, nginx::halfclose+ktls: Options
KTLS on both sides).
On Intel, whose OpenSSL and nginx have kernel TLS: 3proxy splicing over
it moves bulk data 6-17% faster at 5-34% less CPU per gigabyte, with 1K
requests 0-11% slower; before splice() over kernel TLS, and for nginx,
which relays through user space either way, kernel TLS was slower.
CONNECT at 10 Mbit/s, 10 ms, 0.5% loss: -s takes about 9% off the CPU
per exchange, grace delay matched to the rate changes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s[N]:MIN,STREAK,SMALL sets where the automatic mode splices, as -g
does for grace delay: a direction is spliced after STREAK reads in a
row of MIN bytes or more (SMALL from a TLS socket), copied again after
more than STREAK splices in a row of less than SMALL, and a body of
known length is spliced if MIN or more of it is left. The defaults,
16384,2,4096, are what was built in; a value left out or 0 keeps its
default.
The man page describes it, and its HTML copy, which still said splice()
was not built by default, now has the current text. A test case relays
answers of every size, chunked ones and a 3 MB upload through every
mode, two of them with heuristics set, and compares them byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service with ssl_ktls keeps its splice() mode. A connection whose TLS
sockets all got kernel TLS both ways may splice, as a plain one does when
no data filters apply; one where the kernel did not take TLS (nosplice)
copies as before.
TLS may hold data for a socket that a splice() from it would not see:
OpenSSL reads more from a kernel TLS socket than it is asked for. sockmap
asks the TLS code (splicetls) and reads that through TLS into the buffer
first, once the pipe has drained, so the order is kept, then splices on.
A read from a TLS socket counts as large at 4K, TLS handing out a record
at a time, and the short read collecting what TLS left from the last one
does not end a run of large ones. A splice() refused because a record is
not data (an alert, a key update, a ticket) falls back to reading it
through TLS instead of ending the connection.
The kernel TLS test compares the bodies with the origin's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open Watcom has no struct _stat64 or _wstat64: its 64-bit stat is
struct _stati64 and _wstati64, so httpsrv.c did not compile there and
the 1.0.1 lite build had no binaries.
widepath() compared and wrote '\\\\', a two-character constant, for a
backslash: a UNC path (\\server\share) never took the \\?\UNC\ form,
and forward slashes became backslashes only because the constant was
truncated to its last byte.
The Watcom workflow passed although nmake failed; it now fails when
the build does or leaves no 3proxy.exe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9651848bf1)
ssl_ktls asks OpenSSL to hand TLS to the kernel (SSL_OP_ENABLE_KTLS)
on the connections of the services which follow, both to clients and
to servers; ssl_noktls, the default, keeps it in 3proxy. Where the
library, the kernel and the cipher allow it, the kernel then encrypts
and decrypts, and SSL_read/SSL_write go on working over it. What a
service got is logged on its first TLS connection.
Reading ahead is left off where the kernel took receiving, which reads
a record at a time. A build whose library has no kernel TLS (wolfSSL,
OpenSSL before 3.0 or without it) warns and ignores the command.
splice() is not used for TLS services yet.
Tested on AlmaLinux 10 (OpenSSL 3.5.8, tls module): both directions
of both legs of a TLS chain in the kernel, the suite passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 5 Mbit/s, 25 ms, 1% loss run again with a second proxy CPU, so
the HTTP proxy is not held by its CPU: it then keeps 3000 exchanges a
second in every configuration, and the grace delay settings rank as
with one CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grace delay is described as it works: NUM small reads in a row in one
direction within a second start the wait, larger reads end it. It
saves CPU on connections carrying data in small pieces and caps a
connection at one read per DELAY, 16K copying and 64K (the pipe) with
splice(); a table gives the caps for 1-10 ms, another the measured
gains per client link, and DELAY is to be matched to the clients'
speed. It does not help once the CPU is exhausted. The example becomes
-g8000,2,2: the old -g8000,3,10 capped copying at 13 Mbit/s.
splice() gets measured figures: 3 times the speed at 60-70% less CPU
for large transfers on a fast network, within 0-12% for web traffic
in the mix over emulated links. -s, the automatic mode, is the
default; the paragraph still said -s0 was. splice() is not used where
3proxy has to see the data: while data filters apply, and by services
ending or beginning TLS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results on both machines over per-connection emulated links: seven
grace delay settings with a fifth client CPU; the promising ones over
three link qualities (100 Mbit/s 2 ms 0.1%, 20 Mbit/s 5 ms 0.5%,
5 Mbit/s 25 ms 1%); copying against splice() with and without grace
delay, the CONNECT mix included; and the slow link with waits matched
to its rate (10 and 20 ms).
Grace delay saves 35-46% of 3proxy's CPU per gigabyte on streams at
20 Mbit/s and 21-29% at 5 Mbit/s with a 10 ms wait, does nothing on
fast links, and costs CPU with waits too short for the rate or once
the CPU is exhausted. Over these links splice() and copying cost about
the same in the mix scenarios. A caveat says that every scenario but
the mix ones is synthetic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grace delay section, measured on plain veth links, is replaced by
one over an emulated client link: 20 Mbit/s per connection, 5 ms and
0.5% loss each way, devel with four -g settings and nginx, streams
through a relay and the fixed-rate mix scenarios. The README says where
the mix profile's shape comes from and that its parameters are chosen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wrk's and mix's 90th percentile is recorded (p90_ms) and shown in the
report.py --latency tables when present.
--flow-rate limits each connection between the client and the proxy
(fq maxrate), under --netem when both are given, now with a large netem
queue. Both directions are shaped in the client's namespace, the
incoming one through an ifb, so the emulation runs on the client's CPUs;
segmentation offload and GRO are off on that link so that netem drops
single packets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
report.py --latency makes a table of the runs' 50th and 99th percentile
latency for one CSV, ranked by the 99th. RESULTS gets one for each
machine in the HTTP, CONNECT, SOCKS, relay and mix sections, with how to
read them: wrk runs a closed loop, so medians follow throughput and the
99th percentile against the median shows how evenly a proxy serves its
connections; the mix -3k runs are the open-loop ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP, CONNECT, SOCKS, TCP relay and mix tables rebuilt from the full run
on both machines with devel at 98fd0f3 (half-close fix, idle buffer
release, TLS changes), 1.0.1 with TCP_NODELAY, nginx relays with
proxy_half_close on, and envoy added. The morning session moves to the
earlier sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service keeps the configuration it was started with (srvparam.config)
and its threads and its clients' read it there: limits and counters
(alwaysauth, the limiter), RADIUS, dnspr's name servers, and the checks
for a reload in the accept loop and in sockmap(). The configuration is
still reloaded in place, so it is the current one for now; this is what
lets a reload build a new one while the old one still serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf is now a pointer to the current configuration rather than the
structure itself: at first confboot, which exists before main() runs, as
plugins and the standalone services use it too; later configurations
will start from confdefaults. Both are made from confinit.h, the one list
of initial values. The timeouts are an array of the configuration
instead of a pointer to a global one. Nothing behaves differently yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a numbered counter counted now belongs to the process: a counter
rule points to the value of its number, created the first time the
number is met (from the counter file, if one is open) and shared by the
rules of that number in every configuration loaded. A reload no longer
resets it when there is no counter file, and once configurations can
outlive one another, clients of the old and the new one count into the
same value. An unnumbered counter, and those LdapPlugin makes, keep a
value of their own as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf.haveerror was only initialised and srvparam.srvfds only declared;
the extern declarations of stdlog and demon named globals that do not
exist (both are fields of conf); a local in make_ace() and one in the
webadmin XML page were never used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They are the state of the process, not configuration: conf.time (now
curtime), timetoexit and needreload are globals of their own. Plugins
reach needreload through a pointer added at the end of struct
pluginlink; LdapPlugin uses it. The extern for a paused global, never
defined, goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nservers, numservers and authnserver (resolve.c, common.c) and
radiuslist, nradservers and radiussecret (authradius.c) were globals of
their own; they are configuration, set by nserver, authnserver and
radius, and now live in struct extparam with the rest of it.
MAXNSERVERS, MAXRADIUS and struct radserver move to structures.h, which
plugins include without proxy.h.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dialer ran a program when a name could not be resolved, for dial-up
links; with nscache it did not even run. The command is still accepted
and ignored, so configurations using it keep loading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the timeouts, each service now keeps the values given before it
instead of reading the global ones while its connections run. setopts()
takes the service's maxseg. The TLS handshakes wait the service's
CONNECT_TO, kept in its SSL configuration, rather than the global one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssl_filter_open() copies ssl_client_mode into the service's
configuration, but the filters deciding when to begin TLS towards the
server read the global value instead: the last ssl_client_mode in the
configuration applied to every service, and a reload changed it for the
services still running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A socket which hung up and of which nothing is asked was given to poll()
as INVALID_SOCKET, so that poll() would not return at once for the
hang-up every time. poll() ignores a negative descriptor, but SOCKET is
unsigned on Windows, and WSAPoll fails the whole call for it: sockmap()
ended the session as soon as a client half-closed. Such a socket is now
not in the array at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every table is now for one machine, ARM or Intel, with no averages of
the two, the copying proxies and those using splice() in one table:
bold the best of each group and * the best of the row for the HTTP proxy
and the relay, only the best of the row for CONNECT and SOCKS. The
tables for 13 October replace those of 11-12 October, whose columns for
splice by direction and 16 KB buffers are gone with them.
The HTTPS section is measured again with today's devel, with the TLS
streams and envoy. report.py takes the TLS streams and 1 MB answers for
bulk data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit lets a half-closed client be written to on Windows
too, so the case left out there is back.
This reverts commit 5d1cdf8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() took a hang-up on a socket it had data to write to for a peer
it could no longer write to. With poll() a hang-up comes once both sides
are shut down, so that held. WSAPoll reports one as soon as the peer has
half-closed, still reading, and a half-closed client got nothing on
Windows: the test added with the previous fix failed there.
The write is now tried, and a peer really gone makes it fail, which
ends the connection as before. WSAPoll keeps reporting the hang-up
whether the socket has room or not, so on Windows room is waited for
with select() first unless poll() reported it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WSAPoll reports a hang-up as soon as the peer has half-closed, rather
than once both sides are shut down as poll() does elsewhere, and
sockmap() takes a hang-up on a socket it is writing to for a peer it
can no longer write to: a half-closed client gets nothing there. That
is a separate problem, Windows only; the case added with the fix for
the other platforms is skipped there until it is dealt with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-stream* and tls-init-stream* run iperf3 streams through the TLS
relays. iperf3 has no TLS of its own, so bench/mix gets a TLS wrapper
mode, run on the client to begin TLS towards a proxy ending it and at the
origin to end the TLS a proxy begins.
--proxy envoy measures envoy where nginx and haproxy are measured: as an
HTTP proxy for the origin, as a TCP relay and in the TLS scenarios, its
circuit breakers lifted for the scenarios opening more than 1024
connections.
report.py --machine with a single machine gives its medians alone, with
the client- or origin-limited ones marked; --overall marks only the best
of each row. The mix client's -errlog prints every failed exchange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxychild() allocated its 64 KB buffer for headers once and kept it
for the life of the connection: for a CONNECT tunnel all the time data
was relayed, and for a kept-alive connection between its requests.
With 2000 tunnels that was 128 MB, with 20000 mostly idle connections
over a gigabyte.
The buffer is now freed before a tunnel starts relaying, and when a
kept-alive connection has waited a second for its next request; it is
allocated again once the next request starts coming. logurl() takes a
copy of its own for a request it logs after its buffer was freed.
2000 tunnels: 339 to 213 MB; 20000 connections, 3000 exchanges a
second: 2.65 to 1.63 GB through the HTTP proxy, 2.66 to 1.42 GB through
CONNECT; busy kept-alive connections within 1.5% of before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-ahead was off, so OpenSSL read every record in two calls, its
header and then its body. It was off because ssl_poll() asks
ssl_pending(), and SSL_pending() does not count data read ahead and not
yet decrypted: poll() would wait while data waits in OpenSSL.
SSL_has_pending() does count it. With OpenSSL 1.1.0 and later, which
have it, ssl_pending() uses it and records are read ahead; with older
ones and wolfSSL nothing changes.
Decrypting TLS streams takes 9-14% less time for the same data (ARM).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed buffers hold 16384 + 64 bytes, a whole TLS record and its
header, which tlspr needs. sockmap() read all of that from one side and
wrote it to the other, so a side using TLS was written a full record and
another of 64 bytes for every full buffer: twice the records, twice the
writes. Data relayed into the fixed buffers is now read 16384 bytes at
most; a buffer already holding more, as tlspr can leave it, or a larger
one taken for a filter, is used whole.
Ending TLS for 1 MB answers this takes 25% more throughput from the
same CPU (Intel, 4.4 to 5.5 Gbit/s); plain relaying is not affected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the client shuts down its sending side, sockmap() shuts down the
server's in turn, and once the server has answered and closed, poll()
reports a hang-up on its socket. A socket sockmap() had nothing to ask
of at that moment, its buffer being full for a slow client, was still
polled, and a hang-up there ended both directions: whatever of the
answer was still unread in the kernel was dropped and the client got an
end of stream in the middle of it, megabytes short with large answers.
A hang-up on such a socket now only marks it: what is left is read
once the buffer has room, up to the end of the stream as usual. Errors
still end the connection. Since poll() reports a hang-up whatever is
asked, the socket is left out of poll() while nothing is asked of it.
The same holds for the client's side, for an upload cut short after the
server half-closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-* and tls-init-* measure TLS ended or begun by the proxy on a
TCP relay: 3proxy tcppm under ssl_serv/ssl_cli, nginx stream, haproxy
mode tcp and stunnel, with a certificate made for each run and an HTTPS
listener on the origin. nginx::halfclose turns on proxy_half_close.
bench/mix is the Go load generator and origin for the mix-* scenarios.
A run stops when the origin exits (e.g. killed by the OOM killer)
instead of recording failures for everything after it.
RESULTS.md gets the HTTPS and mix tables; data/ the full suite with 16
KB buffers and -s by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside Linux nssocket() was a macro, and two of its callers have #ifdef
WITH_UN among its arguments. Preprocessor directives in macro arguments
are undefined behaviour: gcc and clang take them, MSVC does not. It is a
function on every system now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept-alive client connection reuses the connection to the server while
requests go to the same host. The check compared the start of the request
line up to the end of the host, method included, so a POST after a GET to
the same host, or a GET after a POST, closed the server connection and
opened another one. Only the host and port are compared now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C library reads /etc/localtime on the first use of local time and
connects to /dev/log on the first syslog() call. With a chroot before
either, every local time became UTC and every syslog message was lost.
The time zone is now loaded before chroot() and openlog() connects at
once. The chroot entry of 3proxy.cfg(5) lists what PAM, LDAP and ODBC
still need inside the chroot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without -Ne, the reply to CONNECT and BIND named the local address of the
client's connection to the proxy instead of the socket towards the
destination: the branch meant for UDP ASSOCIATE also took these. A BIND
client was told the proxy's own port and the peer could never reach the
listener. Introduced with -Ne / -Ni (0c8be90).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmarks live on the devel branch only, so link to their results
there; master has no bench/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RESULTS.md now leads with devel as of the automatic splice mode (lazy
pipes kept per connection): the HTTP proxy, CONNECT tunnels (against
squid and tinyproxy), SOCKSv5 tunnels (against dante) and TCP relays
(tcppm against nginx stream and haproxy mode tcp), each a copying and a
splice table, on the M4 and Skylake hosts; 16 KB buffers against 64 KB.
The earlier sessions follow.
data/: *-auto-http.csv (HTTP proxy) and *-tunnels.csv (CONNECT, SOCKS,
relay) for both machines.
README.md: which proxies take part in which scenarios, reworded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- 2000 clients posting 2 KB and getting 50 KB back, next to the GET one
- haproxy with splice-response and splice-request, like -s1 and -s2
- CONNECT and SOCKSv5 scenarios: wrk and iperf3 run under proxychains-ng,
so every connection asks the proxy for a tunnel; and the same loads
through a TCP relay to a fixed destination
- tinyproxy and squid (HTTP proxy, CONNECT), dante (SOCKS); each proxy
takes part only in the scenarios it supports
- 3proxy also gets its socks service and a tcppm to the web server
report.py: one throughput table for several machines (--machine), each
cell their medians and their average; bold for the best of a group, * for
the best of the row; --split puts the splice proxies in a table of their
own, --prefix picks scenarios.
RESULTS.md: the comparison as one table for both machines, splice by
direction (devel -s0/-s1/-s3 against haproxy's settings) and grace delay
with splice, with their data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s takes the directions to splice: -s1 what the server sends (downloads),
-s2 what the client sends (uploads), -s3 both. -s alone (-s7) splices both
only where it pays, as -s5 and -s6 do for one direction:
- a body of known length is spliced if at least 16K of it is left to read
once what is already buffered is counted, so an answer read whole with
its headers is just copied;
- a stream of unknown length starts copied, is spliced after 2 reads in a
row of at least 16K, and goes back to copying after 3 splices in a row
of less than 4K, once its pipe is empty.
Pipes are made when a direction first splices and kept with the
connection for its next mappings, so a kept-alive connection no longer
makes and closes two pipes per request; one left holding data by an
error or a timeout is closed. The direction not spliced keeps its buffer.
Before, -s spliced both directions and every mapping made both pipes,
which made small answers slower than without splice: with kept-alive
50K answers 2 pipe2() and 4 close() per request, now none.
Also corrects the default buffer size in the -b help (65536 for TCP).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- the proxy gets one CPU, the client four, the origin three (8 CPUs), so
the proxy is what runs out first
- CPU time is the busy time of a role's CPUs from /proc/stat, softirqs
included: with IRQ time accounting the cgroup's own figure leaves out
much of the network work
- every run records which side set its pace (proxy, client, origin), by
the busy share of the role's CPUs and of its busiest thread
- a 50 KB keep-alive scenario with 2000 clients
- haproxy with splice: haproxy::splice-auto and haproxy::splice
- wrk timeouts counted apart from errors; tools found by absolute path;
leftovers of an interrupted run are cleaned up
report.py turns runs into Markdown tables: throughput, the proxy's CPU
per gigabyte or per request, its peak memory, splice and copying proxies
in groups of their own, the best of each group marked.
RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy,
with and without splice, on an Apple M4 and an Intel Skylake virtual
machine, and how it was measured; data/ has the measurements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
