A listener counts the traffic of its clients as it is logged: bytes from
servers (%I) and from clients (%O). clearstat(), which every log writer
calls once a line is logged, as well as lognone and logradius, adds the
counts to the client's listener before clearing them, through
listenertrafficfunc, set where services have listeners and NULL in the
standalone binaries.
The admin services page shows them as listenertrafin and
listenertrafout. The takeover case checks that they go on through three
reloads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S_HTTPSRV, the type of admin and httpsrv services, was added before
S_ZOMBIE without a name in the string table, so its index held
"ZOMBIE": the admin services page, %N in logs and RADIUS named admin
and httpsrv services and their clients ZOMBIE, and a zombie service had
no name at all (NULL). Index 21 is now "HTTPSRV", 22 "ZOMBIE"; 22 was
unused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listener counts its clients, whichever of its services they came to,
old or new: those connected now, raised when a client is spawned, and
those done with, raised as one leaves its service. As the listener
outlives a reload, so do the counts.
The services page of the admin interface shows them for each service as
listenerclients and listenerprocessed; services sharing a socket show
the same counts. The takeover case checks that the count goes on
through three reloads and that no client is left counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Services of one configuration on the same address, port, network
namespace and device now share one listener and its socket, all
accepting on it, instead of each binding its own: on Linux the second
bind failed unless SO_REUSEPORT was set. A listener keeps a list of the
services accepting on it; the first to start opens the socket under the
listener's open_mutex, as binding may take long, and the others use it
("Accepting connections, shared"). The socket is closed once the last of
them stops.
Such services must be of one type with the same -ol options: otherwise
the configuration is refused ("a service of another type listens
there", "a service with other socket options listens there").
mainclaim() now runs for every service, at startup too: it joins the
listener of its configuration for the address, takes over one of an
older configuration (one of its type first), or makes a new one, under
one hold of listener_mutex so that a listener found is not freed before
it is joined.
Services connecting back (-r) to one place share a pseudo-listener
without a socket, of any type. After a reload a client moves to one of
the newest services of its type on its listener with a slot free: 26 if
there is none of its type, 27 if none has a slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the -ol options which act when a socket is bound (SO_REUSEADDR,
SO_REUSEPORT, SO_EXCLUSIVEADDRUSE, SO_PORT_SCALABILITY,
SO_REUSE_UNICASTPORT, IP_TRANSPARENT, TCP_FASTOPEN) must be the same
for a socket to be taken over. The others are set on the socket taken
over, those no longer given turned off; TCP_MAXSEG, whose default is
not 0, still needs a new socket when it is dropped.
Several services may still share an address, in one configuration or
across reloads: each new one takes over a listener of its own, one of
an old service of its type first, or opens another.
A service connecting back (-r) has no socket to share, but now a
listener without one for the place it connects to: on a reload the new
service connecting to the same place takes it over, and the old
service's clients are re-authenticated as its clients, as a listening
service's are.
Nothing moves from srvparam to the listener: its socket is what accept
and UDP sessions use, and the rest is the service's configuration,
which the listener only copies to compare.
Tests: a live option changed keeps the socket, a bind-time one makes
another, two services on one address are both taken over, and a
download through a connect-back service goes on or is cut as its user
is kept or removed. 3proxy(8) describes the takeover and
re-authentication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UDP sessions check for a reload in udpsockmap() as TCP connections do
in sockmap(): unless noforce, a session goes by the newest service on
its listener, re-authenticated as its client, or is dropped.
A request on a connection that outlives a reload, an HTTP one kept
alive, is authorized by clientauth(): it moves the connection to the
newest service on its listener first if a reload came since, then
authorizes as the service whose rules apply (PSRV). The proxy uses it
for every request, so a request of a user the new configuration
removed is answered 407 rather than authorized by the old rules and
cut while relayed. reattach() takes whether to authenticate: sockmap()
and udpsockmap() move and authenticate at once, clientauth() moves and
then authorizes the request.
A test case covers the drops: with one slot free in the new service one
of two downloads moves on and the other is cut (27; a service keeps one
of its maxconn slots for the next connection it accepts), and a proxy
download is cut when a socks service takes its port (26). The reload
case checks the kept-alive request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service's socket is kept as a listener like a TCP one, the
protocol part of what must match: on a reload a new UDP service on the
same address takes it over. It is the way the old service's sessions
answer as well, and they go on answering through it; the new service
reads what comes, a datagram of an old session forwarded to it as
before, sessions being found by the service's and the client's
addresses whichever service made them. The socket stays open while the
old sessions do.
The takeover case keeps a udppm session going through three reloads
and wants every datagram back; with the build before this datagrams
were lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reload, unless noforce, a client of an old service goes by the
rules of the newest service on its listener, the one accepting there
now. It takes one of that service's maxconn slots, which counts it
there and keeps that service and its configuration alive until the
client is done, and is re-authenticated as its client: access lists,
users, authentication, bandwidth limits and traffic counters come from
the new configuration. A later reload moves it on, giving the slot
back. The client stays its own service's child, which keeps its
filters and TLS state.
It is dropped if its service was removed or is of another type now
(26), or if the newest service has no free slot (27); what the
re-authentication refuses ends it as before. A service without a
listener (UDP, connecting back) keeps its clients as they are.
This restores what force did before the configuration was kept per
service: since then a client compared its version with its own
service's configuration, which never changes, and was never checked
again. Services now take their configuration's version as it is: a
reload numbers the new configuration before reading it, and so does
startup now, where services were numbered one ahead.
A listener now stays with its old service until that service's last
client is gone, those clients looking the newest service up through
it; the socket itself is still closed as soon as nobody accepts on it.
Connection limits are given back in the configuration they were taken
in. PSRV() is the service whose rules apply to a client.
The reload case checks that a download of a user the new configuration
keeps finishes under it, and that one of a user it removes is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service's TCP listening socket is kept as a listener: when a reload
commits, before the old services are told to stop, each new service
takes over the listener of an older configuration's service on the same
address, port, network namespace and device with the same -ol options.
It skips socket() and bind(), calling only listen() again for its own
backlog, and logs "Accepting connections, taken over". The old service
stops accepting at its next poll and lets go of the socket; whoever
lets go last closes it, unlinking a Unix socket's path.
So a service staying where it was no longer leaves a gap: before, the
old one closed its socket and the new one opened another, and a
connection queued on the old socket was reset. UDP services, whose
clients read from the service's socket, and two services of the same
configuration on one address keep their own sockets as before.
A test case makes requests nonstop through three reloads and wants none
refused or reset; against the build before this, connections were
reset. The reload case no longer waits out the switch. SIGUSR1 in
3proxy(8) describes how a reload behaves now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reload reads the file into a new configuration, allocated from the
defaults, and prepares its services without starting them. Only if the
whole file is read without an error does it come into force: the old
configuration's services stop accepting, the new ones start. On an
error everything prepared is dropped and the old configuration, which
never stopped, stays in force.
Each service holds its configuration until it is freed, after its last
client; a configuration no longer in force is released when the last
holder lets go (confhold/confput). freeconf() now only releases what a
configuration holds, touching nothing global.
What runs without a service of its own reads the configuration in
force, runconf, rather than conf, which a reload points at the one
being read: the resolver, which holds it while it waits (a client
resolves with its own service's), the log, the thread pool, the admin
pages, which walk the services under config_mutex, and the signal
handlers.
The users table, havelog and the resolver choice are per configuration
now, the resolver applied when the configuration comes into force, so
a refused file leaves the users as they were. The log file, the
configuration file name, the string table, daemon mode and the filter
list belong to the process and are handed from one configuration to
the next. Settings left out of a new file go back to their defaults,
where before they kept the old values.
The gap between the old and the new listening sockets remains: a
connection queued on an old service's socket when it closes is reset.
Handing the socket over is the next phase.
A test case reloads a broken file (nothing changes, alice still gets
in, bob from the refused file does not) and a good one (bob only, the
service on its new port, the old one closed, a download through the
old service finishing whole). 40 reloads, a third refused, under load
show no memory errors under AddressSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s[N]:MIN,STREAK,SMALL sets where the automatic mode splices, as -g
does for grace delay: a direction is spliced after STREAK reads in a
row of MIN bytes or more (SMALL from a TLS socket), copied again after
more than STREAK splices in a row of less than SMALL, and a body of
known length is spliced if MIN or more of it is left. The defaults,
16384,2,4096, are what was built in; a value left out or 0 keeps its
default.
The man page describes it, and its HTML copy, which still said splice()
was not built by default, now has the current text. A test case relays
answers of every size, chunked ones and a 3 MB upload through every
mode, two of them with heuristics set, and compares them byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service with ssl_ktls keeps its splice() mode. A connection whose TLS
sockets all got kernel TLS both ways may splice, as a plain one does when
no data filters apply; one where the kernel did not take TLS (nosplice)
copies as before.
TLS may hold data for a socket that a splice() from it would not see:
OpenSSL reads more from a kernel TLS socket than it is asked for. sockmap
asks the TLS code (splicetls) and reads that through TLS into the buffer
first, once the pipe has drained, so the order is kept, then splices on.
A read from a TLS socket counts as large at 4K, TLS handing out a record
at a time, and the short read collecting what TLS left from the last one
does not end a run of large ones. A splice() refused because a record is
not data (an alert, a key update, a ticket) falls back to reading it
through TLS instead of ending the connection.
The kernel TLS test compares the bodies with the origin's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssl_ktls asks OpenSSL to hand TLS to the kernel (SSL_OP_ENABLE_KTLS)
on the connections of the services which follow, both to clients and
to servers; ssl_noktls, the default, keeps it in 3proxy. Where the
library, the kernel and the cipher allow it, the kernel then encrypts
and decrypts, and SSL_read/SSL_write go on working over it. What a
service got is logged on its first TLS connection.
Reading ahead is left off where the kernel took receiving, which reads
a record at a time. A build whose library has no kernel TLS (wolfSSL,
OpenSSL before 3.0 or without it) warns and ignores the command.
splice() is not used for TLS services yet.
Tested on AlmaLinux 10 (OpenSSL 3.5.8, tls module): both directions
of both legs of a TLS chain in the kernel, the suite passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a numbered counter counted now belongs to the process: a counter
rule points to the value of its number, created the first time the
number is met (from the counter file, if one is open) and shared by the
rules of that number in every configuration loaded. A reload no longer
resets it when there is no counter file, and once configurations can
outlive one another, clients of the old and the new one count into the
same value. An unnumbered counter, and those LdapPlugin makes, keep a
value of their own as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit lets a half-closed client be written to on Windows
too, so the case left out there is back.
This reverts commit 5d1cdf8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WSAPoll reports a hang-up as soon as the peer has half-closed, rather
than once both sides are shut down as poll() does elsewhere, and
sockmap() takes a hang-up on a socket it is writing to for a peer it
can no longer write to: a half-closed client gets nothing there. That
is a separate problem, Windows only; the case added with the fix for
the other platforms is skipped there until it is dealt with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the client shuts down its sending side, sockmap() shuts down the
server's in turn, and once the server has answered and closed, poll()
reports a hang-up on its socket. A socket sockmap() had nothing to ask
of at that moment, its buffer being full for a slow client, was still
polled, and a hang-up there ended both directions: whatever of the
answer was still unread in the kernel was dropped and the client got an
end of stream in the middle of it, megabytes short with large answers.
A hang-up on such a socket now only marks it: what is left is read
once the buffer has room, up to the end of the stream as usual. Errors
still end the connection. Since poll() reports a hang-up whatever is
asked, the socket is left out of poll() while nothing is asked of it.
The same holds for the client's side, for an upload cut short after the
server half-closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reload asked for through /R happens on the next tick of the main
loop, and between the old services stopping and the new ones starting a
request is refused: on Windows the new listener cannot bind the port
until the old one has closed it, and retries with a growing delay. When
the tick fell inside the case, every request after it failed with
ConnectionRefusedError (MSVC run of 98e8dfb). The case now waits for
both services to have restarted after each /R.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extport binds a local port out of its range and connects from it. A port
which is free to bind may still be half of a pair which is closing from an
earlier connection to the same destination, and Windows fails the connect
with WSAEADDRINUSE where other systems fail the bind and fall back. The
request then failed with error 13 for a reason another port would have
avoided, which is what the range is searched for in the first place.
Retry the connect with another port from the range. Only for that error, so
a destination which is simply unreachable still costs one attempt, and out of
the same ten tries the bind already spends, so the attempts parentretries
makes are not multiplied. EADDRINUSE joins the errno values proxy.h maps to
their Winsock numbers, without which the test would never match on Windows.
parent_ports gave its three services one window between them, which is how
the case met the collision on the Windows runner. Each has its own now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bindwithrange() honours a range by setting IP_LOCAL_PORT_RANGE where it
exists, which leaves the kernel to pick a port inside it and skip the ones
still closing. Everywhere else it binds a port out of the range at random,
ten times, and lets the system choose when all ten are taken.
A port which carried a connection cannot be bound again until that close
completes - four minutes of it on Windows - so a window of fifty ports
shared by three services in one configuration can have too few left, and
the connection then binds outside the range and the case fails. The window
is five hundred ports there instead. The Linux one is unchanged: the
kernel picks from it and needs no room to spare.
op_rewrite checked what it built with targetunsafe(), which describes a
path on this machine: on Windows it must name a drive or a share, so a
rewritten request path was refused and every rewrite rule failed there.
A rewrite produces a request path and is checked as one.
The header loop stopped at HTTPSRV_MAXHDR and answered anyway, leaving
the rest of the request in the stream for the next one to be read out
of. It refuses the request instead. A header longer than the buffer
arrives as several lines, so the count bounds what is read rather than
what a client may send in one header.
The test for the buffer growth after a PCRE rewrite sent its request
through to an httpsrv origin, which stops reading at that same cap; it
uses an origin which reads whatever it is sent, since what is under test
is the proxy in the middle.
tcppm and udppm forward a port each, so test both directions of each: a
request through the TCP mapper reaching its target, a datagram through the
UDP one coming back answered, and a mapper whose rules deny the client
answering nothing.
tlspr takes its destination from the name in the handshake, so point that
name at 127.0.0.1 with nsrecord and give the certificate the same name: the
name then both chooses where the request goes and is what the client checks.
dnspr answers from its cache, including the documented 0.0.0.0 record, which
is handed out as an address rather than withheld.
auto is asked to serve an HTTP proxy request, SOCKS4, SOCKS5 and a TLS
handshake on one port, and to make nothing of a request that is none of
them. Its protocols reach different places, so there are two origins.
The SOCKS UDP association was only checked for the port it binds. Send
datagrams through it as well, large and small, and check a second
association gets its own port - and that one bound inside an intport range
still relays.
A UDP service has no socket to connect to, so readiness is found by asking
until it answers rather than racing it.
A rewrite only reached the copy of the request kept for logging. On a
direct connection the server is sent the request line held in the header
buffer, which was parsed and reduced to its path before the filters ran, so
the rewrite showed up in the log and nowhere else. Through an HTTP parent
the rewritten copy is what goes out, which is why it appeared to work
there.
Splice the new path back into that buffer. The destination was chosen, and
the access rules applied to it, before the rewrite happened, so a rewrite
that changes the method or the authority is left alone: acting on it would
send the request somewhere the rules never saw. Say so in the manual.
The copy needed to notice a rewrite is only taken when a request filter
exists, so a proxy without one does no extra work.
The filter walked its list of access rules but tested the first entry each
time round, so anything pcre_extend appended was never consulted and the
command did nothing at all. Test the entry the loop is on.
pcre_extend takes an ACE and no FILTER_ACTION - the rule keeps the action it
was given - so correct the manual, which documented an argument the command
does not read.
The tests cover matching and denial, rule order, client headers, matching
options, the access rule a pcre rule carries, pcre_extend, and rewriting a
reply header and reply data. Request rewriting is covered through an HTTP
parent, which is the path where it reaches the wire.
httpsrv parsed Content-Length and never read what followed. The reply is
followed by a close, and closing a socket that still holds unread data
resets the connection instead of ending it, so a POST could cost the client
the reply it was about to read. Windows does that reliably; the same test
passes on Linux and macOS, which is why it looked flaky. Drain the body,
bounded at a megabyte.
The test CA was built with -addext, which LibreSSL - the openssl on a stock
macOS - does not apply the same way, leaving a certificate that is not
usable as a CA and a client that cannot build a chain to it. Put the
extensions in a file both accept, and verify the generated chain before any
of it is handed to a proxy, so a failure there is not read as a fault in
the proxy.
The MITM case reached its origin by address, so it depended on the
certificate 3proxy spoofs carrying an IP alternative name. It does when the
upstream certificate is copied, which is what happens on Linux and macOS
but not on Windows, where the client then refused the connection. Point a
name at 127.0.0.1 with nsrecord instead, and check the chain rather than
the name: an intercepted certificate names the upstream host, not the one
that was asked for. The log assertions gain from it too, since the name is
better evidence than a port that the request was seen.
nsrecord needs nserver as well as nscache, and has to follow nscache, so
say that in the manual: the record goes into the table nscache allocates,
and the table is only consulted when nserver is set.
The port-range fallback was exercised with a range the same case had
already used, so on a busy machine it could fail to bind for the ordinary
reason rather than the one under test. Use privileged ports, which nothing
can take.
When a case fails, print what its servers wrote: the reason usually goes to
the server's stderr, which was captured and then thrown away.
Three arrangements, with key material generated for the run rather than
kept in the tree: a proxy wrapped in TLS, a proxy that reaches a TLS parent
and verifies it against the CA, and MITM.
The MITM case checks what interception is for: the decrypted request line,
URI and all, reaches the log, where the same request through a plain
CONNECT tunnel leaves only the host and port.
The origin runs in its own process there so the proxy log holds only what
the proxy saw, and log assertions wait, since a record is written when the
connection finishes rather than when the reply arrives.
Verification of the spoofed certificate is deliberately not strict: 3proxy
issues those without an Authority Key Identifier, which Python rejects
under its 3.13 defaults.
