The sessions of a keyed UDP service were found by a hash of the
client's address with a fixed seed, so an address crafted to collide
with another client's would have its datagrams taken as that client's.
The hash is now seeded with the table's random entropy, and a session
found is used only if its client's address, port and family are the
datagram's; otherwise the datagram is a new client's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxymain.c stays the one unit built for 3proxy and included by each
standalone service, and now includes its parts:
mainconf.c services from their configuration lines: options,
srvprepare(), mainprepare(), srvinit(), srvfree(), and
copying and freeing access lists and authentication
mainlisten.c listening sockets, shared and taken over on a reload,
and reattach()
mainclient.c threads for the clients of TCP services, their pool,
srvspawn(), releaseparam(), and the filters' hooks
maintcp.c the main loop of a TCP service, tcploop(), taken out of
srvrun(), with tcpnext() and cblisten()
mainudp.c the main loop of a UDP service
proxymain.c keeps srvstart, srvrun() and main(), and declares the
functions of file scope of the parts so that they can be in any order.
The code is moved, not changed, but for tcploop(). The parts are
dependencies of the objects that include proxymain.c in Makefile.inc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
udppm: 20 clients at once each answered with their own datagrams, -s
ending a session with its first answer, STRING_LONG of silence ending
it (92), maxconn leaving a client without a slot unanswered, and a
session adopted on a reload and ended when the new rules deny its
client. dnspr: a query sent on to the name server and its answer
relayed, and a query answered while another waits for a silent name
server, which ends at DNS_TO (821).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service now runs in its own thread alone (udploop()): it reads the
service's socket and the sockets of its sessions towards their
destinations, keeps their deadlines, and answers every client from the
service's socket. A session is a struct clientparam as a client thread's
would be, with a slot of maxchild, among the service's children and
counted on its listener, but without a thread. The services supply
handlers (struct udpservice, childdef.udp): udppm keeps a session per
client address, forwarding its datagrams, behind the headers of a
parent chain, from a socket of its own and answering what comes back,
until STRING_L of silence or, with -s, the first answer; dnspr makes a
session of each query, answered at once from the fake address, the
name cache or the resolver, or sent on to the first name server over
UDP, whose answer is waited for in the loop up to DNS_TO (a name server
over TCP is still waited for in place). Authentication stays in the
loop for now; datagrams over a bandwidth limit, or of a client without
a free slot, are dropped.
On a reload the old service goes on serving the socket until it sees
its configuration replaced, then hands its sessions to the listener;
the newest service of its kind there, which until then does not read
the socket so that no client gets two sessions, adopts them: a slot of
its own, the old one given back, its filters and limits, and
re-authentication. A session is ended with 26 if the newest service is
of another kind, 27 without a slot, or with what the authentication
says; sessions nobody is left to adopt are ended by the last service to
stop on the listener.
On Windows a send on a socket waited while another thread polled it,
which delayed udppm's answers by up to a second, and old UDP services
did not exit after a reload; with one thread per socket neither
happens, and the second socket bound to the service's address
(NO_SHARE_UDP_SOCKET) is gone from the code and the build files.
udpinit, the global udp_table, udpnext(), udppmchild() and
dnsprchild() are gone; udpsockmap() serves SOCKS5 UDP ASSOCIATE only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Linux the port windows lie within net.ipv4.ip_local_port_range, as
the kernel honours no other, so the ephemeral port of a request the
range does not apply to falls into the 50-port window now and then
(CI saw 34001 in 33968-34017). The case now makes five such requests
and wants one of them outside the window: bound to it, all would be in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A build which does not share the UDP listening socket (NO_SHARE_UDP_SOCKET,
which Makefile.win, Makefile.watcom and CMake set for Windows, as Windows
does not take a recv and a send on one socket from two threads) has
dnspr answer from a socket of its own, bound to the service's address,
but udppm still answered through the service's socket. Since sockets
are taken over on a reload, that socket is now also read by the new
services while old sessions write to it. A udppm session in such a
build now makes its own socket too, as dnspr does and as udppm did
before; it is only written to, and closed when the session ends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a reload cannot take a listening Unix socket over (a bind-time
-ol option changed), the new service binds the path anew, and the old
listener, closing after it, removed the file, the new socket's by then,
leaving nothing to connect to. The socket's file (device and inode) is
now noted on binding and the path removed on closing only if it is
still that file.
structures.h includes <stddef.h> for offsetof() in SASIZE(), which
needs it wherever the header is used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Linux a proxy listening on a Unix socket relayed only the first
bytes of an answer (result 95): sockmap() read with recvfrom() into the
client's address, which for a Unix stream socket Linux fills with an
empty AF_UNIX one, and then sent with sendto() to that address, which a
connected Unix stream socket refuses. A TCP connection has no use for
addresses: sockmap() now sends and receives with NULL ones, and
connect-back sends its command with socksend() and reads the answer
with sockrecvfrom() without an address, which a NULL sin now allows.
An abstract socket name (-iunix:@NAME) was bound with the whole size of
struct sockaddr_un, padded with zeros, so a client naming it by its
bytes, as most do, reached another name. SASIZE() now gives an abstract
address its exact size; getsockname() calls that took SASIZE() of the
old contents as the buffer size take the buffer's size instead.
tcppm refused port 0 for a unix: target, though the manual gives
"tcppm 8080 unix:/var/run/app.sock 0" as the example: the port of a
Unix socket target is ignored, so 0 is allowed.
A test case covers a proxy on a Unix socket and on an abstract one on
Linux, tcppm to a Unix socket, a parent reached through one, and a
listening Unix socket across reloads: taken over each time, its file
kept, and removed once no service listens on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The downloads spanning a reload asked httpsrv for 64 KB blocks 60 ms
apart, meant as two seconds; but httpsrv sends at most 8 KB a block, so
they took 245 waits, some 15 seconds, and on the CI runners, whose
timers stretch, more than the 20 the cases waited. They now ask for
400 KB in 8 KB blocks 40 ms apart, two seconds indeed, and are waited
for up to a minute.
The UDP part of the takeover case slept 1.5 s after the reload adding
its udppm and then counted every datagram unanswered as lost; it now
asks until the service answers first, and wants more than 20
datagrams through the reloads rather than 100, what a slow runner can
send. That none is lost is the check.
On a failure, run.py prints every log line with a result code other
than 0, not only the last lines, so the reason a connection ended is
shown wherever it was logged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listener counts the traffic of its clients as it is logged: bytes from
servers (%I) and from clients (%O). clearstat(), which every log writer
calls once a line is logged, as well as lognone and logradius, adds the
counts to the client's listener before clearing them, through
listenertrafficfunc, set where services have listeners and NULL in the
standalone binaries.
The admin services page shows them as listenertrafin and
listenertrafout. The takeover case checks that they go on through three
reloads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S_HTTPSRV, the type of admin and httpsrv services, was added before
S_ZOMBIE without a name in the string table, so its index held
"ZOMBIE": the admin services page, %N in logs and RADIUS named admin
and httpsrv services and their clients ZOMBIE, and a zombie service had
no name at all (NULL). Index 21 is now "HTTPSRV", 22 "ZOMBIE"; 22 was
unused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listener counts its clients, whichever of its services they came to,
old or new: those connected now, raised when a client is spawned, and
those done with, raised as one leaves its service. As the listener
outlives a reload, so do the counts.
The services page of the admin interface shows them for each service as
listenerclients and listenerprocessed; services sharing a socket show
the same counts. The takeover case checks that the count goes on
through three reloads and that no client is left counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Services of one configuration on the same address, port, network
namespace and device now share one listener and its socket, all
accepting on it, instead of each binding its own: on Linux the second
bind failed unless SO_REUSEPORT was set. A listener keeps a list of the
services accepting on it; the first to start opens the socket under the
listener's open_mutex, as binding may take long, and the others use it
("Accepting connections, shared"). The socket is closed once the last of
them stops.
Such services must be of one type with the same -ol options: otherwise
the configuration is refused ("a service of another type listens
there", "a service with other socket options listens there").
mainclaim() now runs for every service, at startup too: it joins the
listener of its configuration for the address, takes over one of an
older configuration (one of its type first), or makes a new one, under
one hold of listener_mutex so that a listener found is not freed before
it is joined.
Services connecting back (-r) to one place share a pseudo-listener
without a socket, of any type. After a reload a client moves to one of
the newest services of its type on its listener with a slot free: 26 if
there is none of its type, 27 if none has a slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the -ol options which act when a socket is bound (SO_REUSEADDR,
SO_REUSEPORT, SO_EXCLUSIVEADDRUSE, SO_PORT_SCALABILITY,
SO_REUSE_UNICASTPORT, IP_TRANSPARENT, TCP_FASTOPEN) must be the same
for a socket to be taken over. The others are set on the socket taken
over, those no longer given turned off; TCP_MAXSEG, whose default is
not 0, still needs a new socket when it is dropped.
Several services may still share an address, in one configuration or
across reloads: each new one takes over a listener of its own, one of
an old service of its type first, or opens another.
A service connecting back (-r) has no socket to share, but now a
listener without one for the place it connects to: on a reload the new
service connecting to the same place takes it over, and the old
service's clients are re-authenticated as its clients, as a listening
service's are.
Nothing moves from srvparam to the listener: its socket is what accept
and UDP sessions use, and the rest is the service's configuration,
which the listener only copies to compare.
Tests: a live option changed keeps the socket, a bind-time one makes
another, two services on one address are both taken over, and a
download through a connect-back service goes on or is cut as its user
is kept or removed. 3proxy(8) describes the takeover and
re-authentication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An FTP control connection authorizes each OPEN or USER and each data
connection while it lasts, which may be across a reload: these go
through clientauth() too, as a client of the newest service on its
listener after a reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UDP sessions check for a reload in udpsockmap() as TCP connections do
in sockmap(): unless noforce, a session goes by the newest service on
its listener, re-authenticated as its client, or is dropped.
A request on a connection that outlives a reload, an HTTP one kept
alive, is authorized by clientauth(): it moves the connection to the
newest service on its listener first if a reload came since, then
authorizes as the service whose rules apply (PSRV). The proxy uses it
for every request, so a request of a user the new configuration
removed is answered 407 rather than authorized by the old rules and
cut while relayed. reattach() takes whether to authenticate: sockmap()
and udpsockmap() move and authenticate at once, clientauth() moves and
then authorizes the request.
A test case covers the drops: with one slot free in the new service one
of two downloads moves on and the other is cut (27; a service keeps one
of its maxconn slots for the next connection it accepts), and a proxy
download is cut when a socks service takes its port (26). The reload
case checks the kept-alive request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service's socket is kept as a listener like a TCP one, the
protocol part of what must match: on a reload a new UDP service on the
same address takes it over. It is the way the old service's sessions
answer as well, and they go on answering through it; the new service
reads what comes, a datagram of an old session forwarded to it as
before, sessions being found by the service's and the client's
addresses whichever service made them. The socket stays open while the
old sessions do.
The takeover case keeps a udppm session going through three reloads
and wants every datagram back; with the build before this datagrams
were lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reload, unless noforce, a client of an old service goes by the
rules of the newest service on its listener, the one accepting there
now. It takes one of that service's maxconn slots, which counts it
there and keeps that service and its configuration alive until the
client is done, and is re-authenticated as its client: access lists,
users, authentication, bandwidth limits and traffic counters come from
the new configuration. A later reload moves it on, giving the slot
back. The client stays its own service's child, which keeps its
filters and TLS state.
It is dropped if its service was removed or is of another type now
(26), or if the newest service has no free slot (27); what the
re-authentication refuses ends it as before. A service without a
listener (UDP, connecting back) keeps its clients as they are.
This restores what force did before the configuration was kept per
service: since then a client compared its version with its own
service's configuration, which never changes, and was never checked
again. Services now take their configuration's version as it is: a
reload numbers the new configuration before reading it, and so does
startup now, where services were numbered one ahead.
A listener now stays with its old service until that service's last
client is gone, those clients looking the newest service up through
it; the socket itself is still closed as soon as nobody accepts on it.
Connection limits are given back in the configuration they were taken
in. PSRV() is the service whose rules apply to a client.
The reload case checks that a download of a user the new configuration
keeps finishes under it, and that one of a user it removes is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service's TCP listening socket is kept as a listener: when a reload
commits, before the old services are told to stop, each new service
takes over the listener of an older configuration's service on the same
address, port, network namespace and device with the same -ol options.
It skips socket() and bind(), calling only listen() again for its own
backlog, and logs "Accepting connections, taken over". The old service
stops accepting at its next poll and lets go of the socket; whoever
lets go last closes it, unlinking a Unix socket's path.
So a service staying where it was no longer leaves a gap: before, the
old one closed its socket and the new one opened another, and a
connection queued on the old socket was reset. UDP services, whose
clients read from the service's socket, and two services of the same
configuration on one address keep their own sockets as before.
A test case makes requests nonstop through three reloads and wants none
refused or reset; against the build before this, connections were
reset. The reload case no longer waits out the switch. SIGUSR1 in
3proxy(8) describes how a reload behaves now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New TLS scenarios with 2000 clients, a 2K POST and a 50K answer each.
Over 20, 10 and 5 Mbit/s links the default 4K is as good as 8K and 16K,
a little better at 5 Mbit/s: it stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Splicing after two 8K reads instead of 16K is never better in the mix
scenarios and costs 20% more CPU as an HTTP proxy on the 20 Mbit/s link:
the default stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reload reads the file into a new configuration, allocated from the
defaults, and prepares its services without starting them. Only if the
whole file is read without an error does it come into force: the old
configuration's services stop accepting, the new ones start. On an
error everything prepared is dropped and the old configuration, which
never stopped, stays in force.
Each service holds its configuration until it is freed, after its last
client; a configuration no longer in force is released when the last
holder lets go (confhold/confput). freeconf() now only releases what a
configuration holds, touching nothing global.
What runs without a service of its own reads the configuration in
force, runconf, rather than conf, which a reload points at the one
being read: the resolver, which holds it while it waits (a client
resolves with its own service's), the log, the thread pool, the admin
pages, which walk the services under config_mutex, and the signal
handlers.
The users table, havelog and the resolver choice are per configuration
now, the resolver applied when the configuration comes into force, so
a refused file leaves the users as they were. The log file, the
configuration file name, the string table, daemon mode and the filter
list belong to the process and are handed from one configuration to
the next. Settings left out of a new file go back to their defaults,
where before they kept the old values.
The gap between the old and the new listening sockets remains: a
connection queued on an old service's socket when it closes is reset.
Handing the socket over is the next phase.
A test case reloads a broken file (nothing changes, alice still gets
in, bob from the refused file does not) and a good one (bob only, the
service on its new port, the old one closed, a download through the
old service finishing whole). 40 reloads, a third refused, under load
show no memory errors under AddressSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- readconfig() kept its line buffer and argument array on every error
return, so each refused reload leaked them.
- A filter refusing a connection in filter_client (TLS: a failed
handshake) was not kept, so its filter_clear never ran and the state
it made was lost. makefilters() now clears it, and empties the slot
first: a pooled worker's slot may hold the previous connection's.
- ssl_filter_close() freed the client CA paths but not the server ones,
for every TLS service set up.
- Compiled patterns were never freed: an ACL's (pcre: and regex: host
names), an httpsrv rule's. copyacl() shared them between the
configuration's ACL and each service's copy; each copy now compiles
its own, and freeacl() and freehttprules() free them.
What LeakSanitizer still reports is OpenSSL's per-thread random
generator state of pooled threads, released only by OpenSSL at exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A section on ssl_ktls: what it needs, how 3proxy reports it, and that
it pays with splice() for TLS connections carrying large transfers.
The splice() section gets the :MIN,STREAK,SMALL tuning and says TLS
services splice where the kernel does TLS. Grace delay is recommended
by the clients' speed for steady streams only: for web traffic it
saved nothing on any of the links measured, nor with the second CPU
the HTTP proxy needed on the 5 Mbit/s one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py takes ktls among 3proxy's options (ssl_ktls before the TLS
services) and for nginx (nginx::ktls, nginx::halfclose+ktls: Options
KTLS on both sides).
On Intel, whose OpenSSL and nginx have kernel TLS: 3proxy splicing over
it moves bulk data 6-17% faster at 5-34% less CPU per gigabyte, with 1K
requests 0-11% slower; before splice() over kernel TLS, and for nginx,
which relays through user space either way, kernel TLS was slower.
CONNECT at 10 Mbit/s, 10 ms, 0.5% loss: -s takes about 9% off the CPU
per exchange, grace delay matched to the rate changes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s[N]:MIN,STREAK,SMALL sets where the automatic mode splices, as -g
does for grace delay: a direction is spliced after STREAK reads in a
row of MIN bytes or more (SMALL from a TLS socket), copied again after
more than STREAK splices in a row of less than SMALL, and a body of
known length is spliced if MIN or more of it is left. The defaults,
16384,2,4096, are what was built in; a value left out or 0 keeps its
default.
The man page describes it, and its HTML copy, which still said splice()
was not built by default, now has the current text. A test case relays
answers of every size, chunked ones and a 3 MB upload through every
mode, two of them with heuristics set, and compares them byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service with ssl_ktls keeps its splice() mode. A connection whose TLS
sockets all got kernel TLS both ways may splice, as a plain one does when
no data filters apply; one where the kernel did not take TLS (nosplice)
copies as before.
TLS may hold data for a socket that a splice() from it would not see:
OpenSSL reads more from a kernel TLS socket than it is asked for. sockmap
asks the TLS code (splicetls) and reads that through TLS into the buffer
first, once the pipe has drained, so the order is kept, then splices on.
A read from a TLS socket counts as large at 4K, TLS handing out a record
at a time, and the short read collecting what TLS left from the last one
does not end a run of large ones. A splice() refused because a record is
not data (an alert, a key update, a ticket) falls back to reading it
through TLS instead of ending the connection.
The kernel TLS test compares the bodies with the origin's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open Watcom has no struct _stat64 or _wstat64: its 64-bit stat is
struct _stati64 and _wstati64, so httpsrv.c did not compile there and
the 1.0.1 lite build had no binaries.
widepath() compared and wrote '\\\\', a two-character constant, for a
backslash: a UNC path (\\server\share) never took the \\?\UNC\ form,
and forward slashes became backslashes only because the constant was
truncated to its last byte.
The Watcom workflow passed although nmake failed; it now fails when
the build does or leaves no 3proxy.exe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9651848bf1)
ssl_ktls asks OpenSSL to hand TLS to the kernel (SSL_OP_ENABLE_KTLS)
on the connections of the services which follow, both to clients and
to servers; ssl_noktls, the default, keeps it in 3proxy. Where the
library, the kernel and the cipher allow it, the kernel then encrypts
and decrypts, and SSL_read/SSL_write go on working over it. What a
service got is logged on its first TLS connection.
Reading ahead is left off where the kernel took receiving, which reads
a record at a time. A build whose library has no kernel TLS (wolfSSL,
OpenSSL before 3.0 or without it) warns and ignores the command.
splice() is not used for TLS services yet.
Tested on AlmaLinux 10 (OpenSSL 3.5.8, tls module): both directions
of both legs of a TLS chain in the kernel, the suite passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 5 Mbit/s, 25 ms, 1% loss run again with a second proxy CPU, so
the HTTP proxy is not held by its CPU: it then keeps 3000 exchanges a
second in every configuration, and the grace delay settings rank as
with one CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grace delay is described as it works: NUM small reads in a row in one
direction within a second start the wait, larger reads end it. It
saves CPU on connections carrying data in small pieces and caps a
connection at one read per DELAY, 16K copying and 64K (the pipe) with
splice(); a table gives the caps for 1-10 ms, another the measured
gains per client link, and DELAY is to be matched to the clients'
speed. It does not help once the CPU is exhausted. The example becomes
-g8000,2,2: the old -g8000,3,10 capped copying at 13 Mbit/s.
splice() gets measured figures: 3 times the speed at 60-70% less CPU
for large transfers on a fast network, within 0-12% for web traffic
in the mix over emulated links. -s, the automatic mode, is the
default; the paragraph still said -s0 was. splice() is not used where
3proxy has to see the data: while data filters apply, and by services
ending or beginning TLS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results on both machines over per-connection emulated links: seven
grace delay settings with a fifth client CPU; the promising ones over
three link qualities (100 Mbit/s 2 ms 0.1%, 20 Mbit/s 5 ms 0.5%,
5 Mbit/s 25 ms 1%); copying against splice() with and without grace
delay, the CONNECT mix included; and the slow link with waits matched
to its rate (10 and 20 ms).
Grace delay saves 35-46% of 3proxy's CPU per gigabyte on streams at
20 Mbit/s and 21-29% at 5 Mbit/s with a 10 ms wait, does nothing on
fast links, and costs CPU with waits too short for the rate or once
the CPU is exhausted. Over these links splice() and copying cost about
the same in the mix scenarios. A caveat says that every scenario but
the mix ones is synthetic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grace delay section, measured on plain veth links, is replaced by
one over an emulated client link: 20 Mbit/s per connection, 5 ms and
0.5% loss each way, devel with four -g settings and nginx, streams
through a relay and the fixed-rate mix scenarios. The README says where
the mix profile's shape comes from and that its parameters are chosen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wrk's and mix's 90th percentile is recorded (p90_ms) and shown in the
report.py --latency tables when present.
--flow-rate limits each connection between the client and the proxy
(fq maxrate), under --netem when both are given, now with a large netem
queue. Both directions are shaped in the client's namespace, the
incoming one through an ifb, so the emulation runs on the client's CPUs;
segmentation offload and GRO are off on that link so that netem drops
single packets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
report.py --latency makes a table of the runs' 50th and 99th percentile
latency for one CSV, ranked by the 99th. RESULTS gets one for each
machine in the HTTP, CONNECT, SOCKS, relay and mix sections, with how to
read them: wrk runs a closed loop, so medians follow throughput and the
99th percentile against the median shows how evenly a proxy serves its
connections; the mix -3k runs are the open-loop ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP, CONNECT, SOCKS, TCP relay and mix tables rebuilt from the full run
on both machines with devel at 98fd0f3 (half-close fix, idle buffer
release, TLS changes), 1.0.1 with TCP_NODELAY, nginx relays with
proxy_half_close on, and envoy added. The morning session moves to the
earlier sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service keeps the configuration it was started with (srvparam.config)
and its threads and its clients' read it there: limits and counters
(alwaysauth, the limiter), RADIUS, dnspr's name servers, and the checks
for a reload in the accept loop and in sockmap(). The configuration is
still reloaded in place, so it is the current one for now; this is what
lets a reload build a new one while the old one still serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf is now a pointer to the current configuration rather than the
structure itself: at first confboot, which exists before main() runs, as
plugins and the standalone services use it too; later configurations
will start from confdefaults. Both are made from confinit.h, the one list
of initial values. The timeouts are an array of the configuration
instead of a pointer to a global one. Nothing behaves differently yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a numbered counter counted now belongs to the process: a counter
rule points to the value of its number, created the first time the
number is met (from the counter file, if one is open) and shared by the
rules of that number in every configuration loaded. A reload no longer
resets it when there is no counter file, and once configurations can
outlive one another, clients of the old and the new one count into the
same value. An unnumbered counter, and those LdapPlugin makes, keep a
value of their own as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf.haveerror was only initialised and srvparam.srvfds only declared;
the extern declarations of stdlog and demon named globals that do not
exist (both are fields of conf); a local in make_ace() and one in the
webadmin XML page were never used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They are the state of the process, not configuration: conf.time (now
curtime), timetoexit and needreload are globals of their own. Plugins
reach needreload through a pointer added at the end of struct
pluginlink; LdapPlugin uses it. The extern for a paused global, never
defined, goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nservers, numservers and authnserver (resolve.c, common.c) and
radiuslist, nradservers and radiussecret (authradius.c) were globals of
their own; they are configuration, set by nserver, authnserver and
radius, and now live in struct extparam with the rest of it.
MAXNSERVERS, MAXRADIUS and struct radserver move to structures.h, which
plugins include without proxy.h.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dialer ran a program when a name could not be resolved, for dial-up
links; with nscache it did not even run. The command is still accepted
and ignored, so configurations using it keep loading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the timeouts, each service now keeps the values given before it
instead of reading the global ones while its connections run. setopts()
takes the service's maxseg. The TLS handshakes wait the service's
CONNECT_TO, kept in its SSL configuration, rather than the global one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssl_filter_open() copies ssl_client_mode into the service's
configuration, but the filters deciding when to begin TLS towards the
server read the global value instead: the last ssl_client_mode in the
configuration applied to every service, and a reload changed it for the
services still running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A socket which hung up and of which nothing is asked was given to poll()
as INVALID_SOCKET, so that poll() would not return at once for the
hang-up every time. poll() ignores a negative descriptor, but SOCKET is
unsigned on Windows, and WSAPoll fails the whole call for it: sockmap()
ended the session as soon as a client half-closed. Such a socket is now
not in the array at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every table is now for one machine, ARM or Intel, with no averages of
the two, the copying proxies and those using splice() in one table:
bold the best of each group and * the best of the row for the HTTP proxy
and the relay, only the best of the row for CONNECT and SOCKS. The
tables for 13 October replace those of 11-12 October, whose columns for
splice by direction and 16 KB buffers are gone with them.
The HTTPS section is measured again with today's devel, with the TLS
streams and envoy. report.py takes the TLS streams and 1 MB answers for
bulk data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit lets a half-closed client be written to on Windows
too, so the case left out there is back.
This reverts commit 5d1cdf8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() took a hang-up on a socket it had data to write to for a peer
it could no longer write to. With poll() a hang-up comes once both sides
are shut down, so that held. WSAPoll reports one as soon as the peer has
half-closed, still reading, and a half-closed client got nothing on
Windows: the test added with the previous fix failed there.
The write is now tried, and a peer really gone makes it fail, which
ends the connection as before. WSAPoll keeps reporting the hang-up
whether the socket has room or not, so on Windows room is waited for
with select() first unless poll() reported it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WSAPoll reports a hang-up as soon as the peer has half-closed, rather
than once both sides are shut down as poll() does elsewhere, and
sockmap() takes a hang-up on a socket it is writing to for a peer it
can no longer write to: a half-closed client gets nothing there. That
is a separate problem, Windows only; the case added with the fix for
the other platforms is skipped there until it is dealt with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-stream* and tls-init-stream* run iperf3 streams through the TLS
relays. iperf3 has no TLS of its own, so bench/mix gets a TLS wrapper
mode, run on the client to begin TLS towards a proxy ending it and at the
origin to end the TLS a proxy begins.
--proxy envoy measures envoy where nginx and haproxy are measured: as an
HTTP proxy for the origin, as a TCP relay and in the TLS scenarios, its
circuit breakers lifted for the scenarios opening more than 1024
connections.
report.py --machine with a single machine gives its medians alone, with
the client- or origin-limited ones marked; --overall marks only the best
of each row. The mix client's -errlog prints every failed exchange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxychild() allocated its 64 KB buffer for headers once and kept it
for the life of the connection: for a CONNECT tunnel all the time data
was relayed, and for a kept-alive connection between its requests.
With 2000 tunnels that was 128 MB, with 20000 mostly idle connections
over a gigabyte.
The buffer is now freed before a tunnel starts relaying, and when a
kept-alive connection has waited a second for its next request; it is
allocated again once the next request starts coming. logurl() takes a
copy of its own for a request it logs after its buffer was freed.
2000 tunnels: 339 to 213 MB; 20000 connections, 3000 exchanges a
second: 2.65 to 1.63 GB through the HTTP proxy, 2.66 to 1.42 GB through
CONNECT; busy kept-alive connections within 1.5% of before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-ahead was off, so OpenSSL read every record in two calls, its
header and then its body. It was off because ssl_poll() asks
ssl_pending(), and SSL_pending() does not count data read ahead and not
yet decrypted: poll() would wait while data waits in OpenSSL.
SSL_has_pending() does count it. With OpenSSL 1.1.0 and later, which
have it, ssl_pending() uses it and records are read ahead; with older
ones and wolfSSL nothing changes.
Decrypting TLS streams takes 9-14% less time for the same data (ARM).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed buffers hold 16384 + 64 bytes, a whole TLS record and its
header, which tlspr needs. sockmap() read all of that from one side and
wrote it to the other, so a side using TLS was written a full record and
another of 64 bytes for every full buffer: twice the records, twice the
writes. Data relayed into the fixed buffers is now read 16384 bytes at
most; a buffer already holding more, as tlspr can leave it, or a larger
one taken for a filter, is used whole.
Ending TLS for 1 MB answers this takes 25% more throughput from the
same CPU (Intel, 4.4 to 5.5 Gbit/s); plain relaying is not affected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the client shuts down its sending side, sockmap() shuts down the
server's in turn, and once the server has answered and closed, poll()
reports a hang-up on its socket. A socket sockmap() had nothing to ask
of at that moment, its buffer being full for a slow client, was still
polled, and a hang-up there ended both directions: whatever of the
answer was still unread in the kernel was dropped and the client got an
end of stream in the middle of it, megabytes short with large answers.
A hang-up on such a socket now only marks it: what is left is read
once the buffer has room, up to the end of the stream as usual. Errors
still end the connection. Since poll() reports a hang-up whatever is
asked, the socket is left out of poll() while nothing is asked of it.
The same holds for the client's side, for an upload cut short after the
server half-closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-* and tls-init-* measure TLS ended or begun by the proxy on a
TCP relay: 3proxy tcppm under ssl_serv/ssl_cli, nginx stream, haproxy
mode tcp and stunnel, with a certificate made for each run and an HTTPS
listener on the origin. nginx::halfclose turns on proxy_half_close.
bench/mix is the Go load generator and origin for the mix-* scenarios.
A run stops when the origin exits (e.g. killed by the OOM killer)
instead of recording failures for everything after it.
RESULTS.md gets the HTTPS and mix tables; data/ the full suite with 16
KB buffers and -s by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside Linux nssocket() was a macro, and two of its callers have #ifdef
WITH_UN among its arguments. Preprocessor directives in macro arguments
are undefined behaviour: gcc and clang take them, MSVC does not. It is a
function on every system now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept-alive client connection reuses the connection to the server while
requests go to the same host. The check compared the start of the request
line up to the end of the host, method included, so a POST after a GET to
the same host, or a GET after a POST, closed the server connection and
opened another one. Only the host and port are compared now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C library reads /etc/localtime on the first use of local time and
connects to /dev/log on the first syslog() call. With a chroot before
either, every local time became UTC and every syslog message was lost.
The time zone is now loaded before chroot() and openlog() connects at
once. The chroot entry of 3proxy.cfg(5) lists what PAM, LDAP and ODBC
still need inside the chroot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without -Ne, the reply to CONNECT and BIND named the local address of the
client's connection to the proxy instead of the socket towards the
destination: the branch meant for UDP ASSOCIATE also took these. A BIND
client was told the proxy's own port and the peer could never reach the
listener. Introduced with -Ne / -Ni (0c8be90).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmarks live on the devel branch only, so link to their results
there; master has no bench/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RESULTS.md now leads with devel as of the automatic splice mode (lazy
pipes kept per connection): the HTTP proxy, CONNECT tunnels (against
squid and tinyproxy), SOCKSv5 tunnels (against dante) and TCP relays
(tcppm against nginx stream and haproxy mode tcp), each a copying and a
splice table, on the M4 and Skylake hosts; 16 KB buffers against 64 KB.
The earlier sessions follow.
data/: *-auto-http.csv (HTTP proxy) and *-tunnels.csv (CONNECT, SOCKS,
relay) for both machines.
README.md: which proxies take part in which scenarios, reworded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- 2000 clients posting 2 KB and getting 50 KB back, next to the GET one
- haproxy with splice-response and splice-request, like -s1 and -s2
- CONNECT and SOCKSv5 scenarios: wrk and iperf3 run under proxychains-ng,
so every connection asks the proxy for a tunnel; and the same loads
through a TCP relay to a fixed destination
- tinyproxy and squid (HTTP proxy, CONNECT), dante (SOCKS); each proxy
takes part only in the scenarios it supports
- 3proxy also gets its socks service and a tcppm to the web server
report.py: one throughput table for several machines (--machine), each
cell their medians and their average; bold for the best of a group, * for
the best of the row; --split puts the splice proxies in a table of their
own, --prefix picks scenarios.
RESULTS.md: the comparison as one table for both machines, splice by
direction (devel -s0/-s1/-s3 against haproxy's settings) and grace delay
with splice, with their data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s takes the directions to splice: -s1 what the server sends (downloads),
-s2 what the client sends (uploads), -s3 both. -s alone (-s7) splices both
only where it pays, as -s5 and -s6 do for one direction:
- a body of known length is spliced if at least 16K of it is left to read
once what is already buffered is counted, so an answer read whole with
its headers is just copied;
- a stream of unknown length starts copied, is spliced after 2 reads in a
row of at least 16K, and goes back to copying after 3 splices in a row
of less than 4K, once its pipe is empty.
Pipes are made when a direction first splices and kept with the
connection for its next mappings, so a kept-alive connection no longer
makes and closes two pipes per request; one left holding data by an
error or a timeout is closed. The direction not spliced keeps its buffer.
Before, -s spliced both directions and every mapping made both pipes,
which made small answers slower than without splice: with kept-alive
50K answers 2 pipe2() and 4 close() per request, now none.
Also corrects the default buffer size in the -b help (65536 for TCP).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py:
- the proxy gets one CPU, the client four, the origin three (8 CPUs), so
the proxy is what runs out first
- CPU time is the busy time of a role's CPUs from /proc/stat, softirqs
included: with IRQ time accounting the cgroup's own figure leaves out
much of the network work
- every run records which side set its pace (proxy, client, origin), by
the busy share of the role's CPUs and of its busiest thread
- a 50 KB keep-alive scenario with 2000 clients
- haproxy with splice: haproxy::splice-auto and haproxy::splice
- wrk timeouts counted apart from errors; tools found by absolute path;
leftovers of an interrupted run are cleaned up
report.py turns runs into Markdown tables: throughput, the proxy's CPU
per gigabyte or per request, its peak memory, splice and copying proxies
in groups of their own, the best of each group marked.
RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy,
with and without splice, on an Apple M4 and an Intel Skylake virtual
machine, and how it was measured; data/ has the measurements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench/build.sh builds git refs or the working tree, each with the
Makefile it shipped with. bench/bench.py runs the client, the proxy and
the origin in network namespaces of their own, linked by veth pairs,
and in cgroups pinned to separate CPUs, then measures TCP relaying
(iperf3) and HTTP proxying (wrk) for 3proxy builds, nginx, haproxy and
the kernel forwarding alone, reporting throughput, latency and the
proxy's CPU time and peak memory per unit of work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reload asked for through /R happens on the next tick of the main
loop, and between the old services stopping and the new ones starting a
request is refused: on Windows the new listener cannot bind the port
until the old one has closed it, and retries with a growing delay. When
the tick fell inside the case, every request after it failed with
ConnectionRefusedError (MSVC run of 98e8dfb). The case now waits for
both services to have restarted after each /R.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extport binds a local port out of its range and connects from it. A port
which is free to bind may still be half of a pair which is closing from an
earlier connection to the same destination, and Windows fails the connect
with WSAEADDRINUSE where other systems fail the bind and fall back. The
request then failed with error 13 for a reason another port would have
avoided, which is what the range is searched for in the first place.
Retry the connect with another port from the range. Only for that error, so
a destination which is simply unreachable still costs one attempt, and out of
the same ten tries the bind already spends, so the attempts parentretries
makes are not multiplied. EADDRINUSE joins the errno values proxy.h maps to
their Winsock numbers, without which the test would never match on Windows.
parent_ports gave its three services one window between them, which is how
the case met the collision on the Windows runner. Each has its own now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nserver and authnserver parse their argument with parsehost(), which reads
the first colon as the address/port separator, so a bare IPv6 address picks
up a bogus port number. Document the bracket form in 3proxy.cfg(5) and in
both HOWTOs.
Refs #1275, #460
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Package maintainers, Gentoo among them, build from source and had nothing
signed to verify against: the source archive GitHub generates for a tag is
neither signed nor guaranteed to stay byte-identical.
Add release-tarball.yml. It builds 3proxy-<version>.tar.gz with git archive
from the release tag, signs the tarball and SHA256SUMS-src with the release
key, attests build provenance and uploads all four files to the release.
Also drop the SECURITY.md line about Authenticode signing, left over from the
removal of self-signed Windows binaries.
Closes#1270
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signing the release binaries with a self-signed certificate gives no
trust benefit: Windows does not recognize the issuer, so SmartScreen
warns anyway, and the signature is a frequent trigger for antivirus
false positives.
Drop the Decode Certificate / Extract public certificate / Sign steps
from the MSVC and Watcom workflows, stop shipping the extracted
3proxy.crt in the distribution, and remove the certificate URL from the
build version string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pop3p, imapp and smtpp are built with MAILPROXY=true, and ftppr and the
ftp:// scheme of the HTTP proxy with FTP=true; with CMake the switches are
3PROXY_USE_MAILPROXY and 3PROXY_USE_FTP. Neither is in a default build, and
the standalone binaries follow what was built.
A configuration naming one of them is still read either way. The three mail
protocols amount to a STARTTLS negotiation now that mail is carried over
TLS, so without a proxy of their own those names are tlspr speaking the
protocol: the file holds a stand-in which sets the protocol for the
connection and returns tlspr for the caller to run, which serves the
service name and a parent chain alike and leaves conf.c and the redirect
table untouched. FTP has no such fallback, so the service is known, answers
nothing and logs the refusal.
The Linux workflow builds both, so all of it is still compiled and run.
bindwithrange() honours a range by setting IP_LOCAL_PORT_RANGE where it
exists, which leaves the kernel to pick a port inside it and skip the ones
still closing. Everywhere else it binds a port out of the range at random,
ten times, and lets the system choose when all ten are taken.
A port which carried a connection cannot be bound again until that close
completes - four minutes of it on Windows - so a window of fifty ports
shared by three services in one configuration can have too few left, and
the connection then binds outside the range and the case fails. The window
is five hundred ports there instead. The Linux one is unchanged: the
kernel picks from it and needs no room to spare.
op_rewrite checked what it built with targetunsafe(), which describes a
path on this machine: on Windows it must name a drive or a share, so a
rewritten request path was refused and every rewrite rule failed there.
A rewrite produces a request path and is checked as one.
The header loop stopped at HTTPSRV_MAXHDR and answered anyway, leaving
the rest of the request in the stream for the next one to be read out
of. It refuses the request instead. A header longer than the buffer
arrives as several lines, so the count bounds what is read rather than
what a client may send in one header.
The test for the buffer growth after a PCRE rewrite sent its request
through to an httpsrv origin, which stops reading at that same cap; it
uses an origin which reads whatever it is sent, since what is under test
is the proxy in the middle.