An FTP control connection authorizes each OPEN or USER and each data
connection while it lasts, which may be across a reload: these go
through clientauth() too, as a client of the newest service on its
listener after a reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UDP sessions check for a reload in udpsockmap() as TCP connections do
in sockmap(): unless noforce, a session goes by the newest service on
its listener, re-authenticated as its client, or is dropped.
A request on a connection that outlives a reload, an HTTP one kept
alive, is authorized by clientauth(): it moves the connection to the
newest service on its listener first if a reload came since, then
authorizes as the service whose rules apply (PSRV). The proxy uses it
for every request, so a request of a user the new configuration
removed is answered 407 rather than authorized by the old rules and
cut while relayed. reattach() takes whether to authenticate: sockmap()
and udpsockmap() move and authenticate at once, clientauth() moves and
then authorizes the request.
A test case covers the drops: with one slot free in the new service one
of two downloads moves on and the other is cut (27; a service keeps one
of its maxconn slots for the next connection it accepts), and a proxy
download is cut when a socks service takes its port (26). The reload
case checks the kept-alive request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service's socket is kept as a listener like a TCP one, the
protocol part of what must match: on a reload a new UDP service on the
same address takes it over. It is the way the old service's sessions
answer as well, and they go on answering through it; the new service
reads what comes, a datagram of an old session forwarded to it as
before, sessions being found by the service's and the client's
addresses whichever service made them. The socket stays open while the
old sessions do.
The takeover case keeps a udppm session going through three reloads
and wants every datagram back; with the build before this datagrams
were lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reload, unless noforce, a client of an old service goes by the
rules of the newest service on its listener, the one accepting there
now. It takes one of that service's maxconn slots, which counts it
there and keeps that service and its configuration alive until the
client is done, and is re-authenticated as its client: access lists,
users, authentication, bandwidth limits and traffic counters come from
the new configuration. A later reload moves it on, giving the slot
back. The client stays its own service's child, which keeps its
filters and TLS state.
It is dropped if its service was removed or is of another type now
(26), or if the newest service has no free slot (27); what the
re-authentication refuses ends it as before. A service without a
listener (UDP, connecting back) keeps its clients as they are.
This restores what force did before the configuration was kept per
service: since then a client compared its version with its own
service's configuration, which never changes, and was never checked
again. Services now take their configuration's version as it is: a
reload numbers the new configuration before reading it, and so does
startup now, where services were numbered one ahead.
A listener now stays with its old service until that service's last
client is gone, those clients looking the newest service up through
it; the socket itself is still closed as soon as nobody accepts on it.
Connection limits are given back in the configuration they were taken
in. PSRV() is the service whose rules apply to a client.
The reload case checks that a download of a user the new configuration
keeps finishes under it, and that one of a user it removes is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service's TCP listening socket is kept as a listener: when a reload
commits, before the old services are told to stop, each new service
takes over the listener of an older configuration's service on the same
address, port, network namespace and device with the same -ol options.
It skips socket() and bind(), calling only listen() again for its own
backlog, and logs "Accepting connections, taken over". The old service
stops accepting at its next poll and lets go of the socket; whoever
lets go last closes it, unlinking a Unix socket's path.
So a service staying where it was no longer leaves a gap: before, the
old one closed its socket and the new one opened another, and a
connection queued on the old socket was reset. UDP services, whose
clients read from the service's socket, and two services of the same
configuration on one address keep their own sockets as before.
A test case makes requests nonstop through three reloads and wants none
refused or reset; against the build before this, connections were
reset. The reload case no longer waits out the switch. SIGUSR1 in
3proxy(8) describes how a reload behaves now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reload reads the file into a new configuration, allocated from the
defaults, and prepares its services without starting them. Only if the
whole file is read without an error does it come into force: the old
configuration's services stop accepting, the new ones start. On an
error everything prepared is dropped and the old configuration, which
never stopped, stays in force.
Each service holds its configuration until it is freed, after its last
client; a configuration no longer in force is released when the last
holder lets go (confhold/confput). freeconf() now only releases what a
configuration holds, touching nothing global.
What runs without a service of its own reads the configuration in
force, runconf, rather than conf, which a reload points at the one
being read: the resolver, which holds it while it waits (a client
resolves with its own service's), the log, the thread pool, the admin
pages, which walk the services under config_mutex, and the signal
handlers.
The users table, havelog and the resolver choice are per configuration
now, the resolver applied when the configuration comes into force, so
a refused file leaves the users as they were. The log file, the
configuration file name, the string table, daemon mode and the filter
list belong to the process and are handed from one configuration to
the next. Settings left out of a new file go back to their defaults,
where before they kept the old values.
The gap between the old and the new listening sockets remains: a
connection queued on an old service's socket when it closes is reset.
Handing the socket over is the next phase.
A test case reloads a broken file (nothing changes, alice still gets
in, bob from the refused file does not) and a good one (bob only, the
service on its new port, the old one closed, a download through the
old service finishing whole). 40 reloads, a third refused, under load
show no memory errors under AddressSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- readconfig() kept its line buffer and argument array on every error
return, so each refused reload leaked them.
- A filter refusing a connection in filter_client (TLS: a failed
handshake) was not kept, so its filter_clear never ran and the state
it made was lost. makefilters() now clears it, and empties the slot
first: a pooled worker's slot may hold the previous connection's.
- ssl_filter_close() freed the client CA paths but not the server ones,
for every TLS service set up.
- Compiled patterns were never freed: an ACL's (pcre: and regex: host
names), an httpsrv rule's. copyacl() shared them between the
configuration's ACL and each service's copy; each copy now compiles
its own, and freeacl() and freehttprules() free them.
What LeakSanitizer still reports is OpenSSL's per-thread random
generator state of pooled threads, released only by OpenSSL at exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s[N]:MIN,STREAK,SMALL sets where the automatic mode splices, as -g
does for grace delay: a direction is spliced after STREAK reads in a
row of MIN bytes or more (SMALL from a TLS socket), copied again after
more than STREAK splices in a row of less than SMALL, and a body of
known length is spliced if MIN or more of it is left. The defaults,
16384,2,4096, are what was built in; a value left out or 0 keeps its
default.
The man page describes it, and its HTML copy, which still said splice()
was not built by default, now has the current text. A test case relays
answers of every size, chunked ones and a 3 MB upload through every
mode, two of them with heuristics set, and compares them byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service with ssl_ktls keeps its splice() mode. A connection whose TLS
sockets all got kernel TLS both ways may splice, as a plain one does when
no data filters apply; one where the kernel did not take TLS (nosplice)
copies as before.
TLS may hold data for a socket that a splice() from it would not see:
OpenSSL reads more from a kernel TLS socket than it is asked for. sockmap
asks the TLS code (splicetls) and reads that through TLS into the buffer
first, once the pipe has drained, so the order is kept, then splices on.
A read from a TLS socket counts as large at 4K, TLS handing out a record
at a time, and the short read collecting what TLS left from the last one
does not end a run of large ones. A splice() refused because a record is
not data (an alert, a key update, a ticket) falls back to reading it
through TLS instead of ending the connection.
The kernel TLS test compares the bodies with the origin's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open Watcom has no struct _stat64 or _wstat64: its 64-bit stat is
struct _stati64 and _wstati64, so httpsrv.c did not compile there and
the 1.0.1 lite build had no binaries.
widepath() compared and wrote '\\\\', a two-character constant, for a
backslash: a UNC path (\\server\share) never took the \\?\UNC\ form,
and forward slashes became backslashes only because the constant was
truncated to its last byte.
The Watcom workflow passed although nmake failed; it now fails when
the build does or leaves no 3proxy.exe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9651848bf1)
ssl_ktls asks OpenSSL to hand TLS to the kernel (SSL_OP_ENABLE_KTLS)
on the connections of the services which follow, both to clients and
to servers; ssl_noktls, the default, keeps it in 3proxy. Where the
library, the kernel and the cipher allow it, the kernel then encrypts
and decrypts, and SSL_read/SSL_write go on working over it. What a
service got is logged on its first TLS connection.
Reading ahead is left off where the kernel took receiving, which reads
a record at a time. A build whose library has no kernel TLS (wolfSSL,
OpenSSL before 3.0 or without it) warns and ignores the command.
splice() is not used for TLS services yet.
Tested on AlmaLinux 10 (OpenSSL 3.5.8, tls module): both directions
of both legs of a TLS chain in the kernel, the suite passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service keeps the configuration it was started with (srvparam.config)
and its threads and its clients' read it there: limits and counters
(alwaysauth, the limiter), RADIUS, dnspr's name servers, and the checks
for a reload in the accept loop and in sockmap(). The configuration is
still reloaded in place, so it is the current one for now; this is what
lets a reload build a new one while the old one still serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf is now a pointer to the current configuration rather than the
structure itself: at first confboot, which exists before main() runs, as
plugins and the standalone services use it too; later configurations
will start from confdefaults. Both are made from confinit.h, the one list
of initial values. The timeouts are an array of the configuration
instead of a pointer to a global one. Nothing behaves differently yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a numbered counter counted now belongs to the process: a counter
rule points to the value of its number, created the first time the
number is met (from the counter file, if one is open) and shared by the
rules of that number in every configuration loaded. A reload no longer
resets it when there is no counter file, and once configurations can
outlive one another, clients of the old and the new one count into the
same value. An unnumbered counter, and those LdapPlugin makes, keep a
value of their own as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf.haveerror was only initialised and srvparam.srvfds only declared;
the extern declarations of stdlog and demon named globals that do not
exist (both are fields of conf); a local in make_ace() and one in the
webadmin XML page were never used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They are the state of the process, not configuration: conf.time (now
curtime), timetoexit and needreload are globals of their own. Plugins
reach needreload through a pointer added at the end of struct
pluginlink; LdapPlugin uses it. The extern for a paused global, never
defined, goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nservers, numservers and authnserver (resolve.c, common.c) and
radiuslist, nradservers and radiussecret (authradius.c) were globals of
their own; they are configuration, set by nserver, authnserver and
radius, and now live in struct extparam with the rest of it.
MAXNSERVERS, MAXRADIUS and struct radserver move to structures.h, which
plugins include without proxy.h.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dialer ran a program when a name could not be resolved, for dial-up
links; with nscache it did not even run. The command is still accepted
and ignored, so configurations using it keep loading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the timeouts, each service now keeps the values given before it
instead of reading the global ones while its connections run. setopts()
takes the service's maxseg. The TLS handshakes wait the service's
CONNECT_TO, kept in its SSL configuration, rather than the global one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssl_filter_open() copies ssl_client_mode into the service's
configuration, but the filters deciding when to begin TLS towards the
server read the global value instead: the last ssl_client_mode in the
configuration applied to every service, and a reload changed it for the
services still running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A socket which hung up and of which nothing is asked was given to poll()
as INVALID_SOCKET, so that poll() would not return at once for the
hang-up every time. poll() ignores a negative descriptor, but SOCKET is
unsigned on Windows, and WSAPoll fails the whole call for it: sockmap()
ended the session as soon as a client half-closed. Such a socket is now
not in the array at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() took a hang-up on a socket it had data to write to for a peer
it could no longer write to. With poll() a hang-up comes once both sides
are shut down, so that held. WSAPoll reports one as soon as the peer has
half-closed, still reading, and a half-closed client got nothing on
Windows: the test added with the previous fix failed there.
The write is now tried, and a peer really gone makes it fail, which
ends the connection as before. WSAPoll keeps reporting the hang-up
whether the socket has room or not, so on Windows room is waited for
with select() first unless poll() reported it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxychild() allocated its 64 KB buffer for headers once and kept it
for the life of the connection: for a CONNECT tunnel all the time data
was relayed, and for a kept-alive connection between its requests.
With 2000 tunnels that was 128 MB, with 20000 mostly idle connections
over a gigabyte.
The buffer is now freed before a tunnel starts relaying, and when a
kept-alive connection has waited a second for its next request; it is
allocated again once the next request starts coming. logurl() takes a
copy of its own for a request it logs after its buffer was freed.
2000 tunnels: 339 to 213 MB; 20000 connections, 3000 exchanges a
second: 2.65 to 1.63 GB through the HTTP proxy, 2.66 to 1.42 GB through
CONNECT; busy kept-alive connections within 1.5% of before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-ahead was off, so OpenSSL read every record in two calls, its
header and then its body. It was off because ssl_poll() asks
ssl_pending(), and SSL_pending() does not count data read ahead and not
yet decrypted: poll() would wait while data waits in OpenSSL.
SSL_has_pending() does count it. With OpenSSL 1.1.0 and later, which
have it, ssl_pending() uses it and records are read ahead; with older
ones and wolfSSL nothing changes.
Decrypting TLS streams takes 9-14% less time for the same data (ARM).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed buffers hold 16384 + 64 bytes, a whole TLS record and its
header, which tlspr needs. sockmap() read all of that from one side and
wrote it to the other, so a side using TLS was written a full record and
another of 64 bytes for every full buffer: twice the records, twice the
writes. Data relayed into the fixed buffers is now read 16384 bytes at
most; a buffer already holding more, as tlspr can leave it, or a larger
one taken for a filter, is used whole.
Ending TLS for 1 MB answers this takes 25% more throughput from the
same CPU (Intel, 4.4 to 5.5 Gbit/s); plain relaying is not affected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the client shuts down its sending side, sockmap() shuts down the
server's in turn, and once the server has answered and closed, poll()
reports a hang-up on its socket. A socket sockmap() had nothing to ask
of at that moment, its buffer being full for a slow client, was still
polled, and a hang-up there ended both directions: whatever of the
answer was still unread in the kernel was dropped and the client got an
end of stream in the middle of it, megabytes short with large answers.
A hang-up on such a socket now only marks it: what is left is read
once the buffer has room, up to the end of the stream as usual. Errors
still end the connection. Since poll() reports a hang-up whatever is
asked, the socket is left out of poll() while nothing is asked of it.
The same holds for the client's side, for an upload cut short after the
server half-closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside Linux nssocket() was a macro, and two of its callers have #ifdef
WITH_UN among its arguments. Preprocessor directives in macro arguments
are undefined behaviour: gcc and clang take them, MSVC does not. It is a
function on every system now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept-alive client connection reuses the connection to the server while
requests go to the same host. The check compared the start of the request
line up to the end of the host, method included, so a POST after a GET to
the same host, or a GET after a POST, closed the server connection and
opened another one. Only the host and port are compared now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C library reads /etc/localtime on the first use of local time and
connects to /dev/log on the first syslog() call. With a chroot before
either, every local time became UTC and every syslog message was lost.
The time zone is now loaded before chroot() and openlog() connects at
once. The chroot entry of 3proxy.cfg(5) lists what PAM, LDAP and ODBC
still need inside the chroot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without -Ne, the reply to CONNECT and BIND named the local address of the
client's connection to the proxy instead of the socket towards the
destination: the branch meant for UDP ASSOCIATE also took these. A BIND
client was told the proxy's own port and the peer could never reach the
listener. Introduced with -Ne / -Ni (0c8be90).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s takes the directions to splice: -s1 what the server sends (downloads),
-s2 what the client sends (uploads), -s3 both. -s alone (-s7) splices both
only where it pays, as -s5 and -s6 do for one direction:
- a body of known length is spliced if at least 16K of it is left to read
once what is already buffered is counted, so an answer read whole with
its headers is just copied;
- a stream of unknown length starts copied, is spliced after 2 reads in a
row of at least 16K, and goes back to copying after 3 splices in a row
of less than 4K, once its pipe is empty.
Pipes are made when a direction first splices and kept with the
connection for its next mappings, so a kept-alive connection no longer
makes and closes two pipes per request; one left holding data by an
error or a timeout is closed. The direction not spliced keeps its buffer.
Before, -s spliced both directions and every mapping made both pipes,
which made small answers slower than without splice: with kept-alive
50K answers 2 pipe2() and 4 close() per request, now none.
Also corrects the default buffer size in the -b help (65536 for TCP).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On receive, the struct sockaddr is zeroed and does not have an address
family until recvfrom fills it in. SASIZE assumes that if there is no
address family, it is INET. This means that for IPv6 UDP sockets, the
sockaddr is truncated and the reply is forwarded to the wrong address.
Instead, use sizeof(PROXYSOCKADDRTYPE).
extport binds a local port out of its range and connects from it. A port
which is free to bind may still be half of a pair which is closing from an
earlier connection to the same destination, and Windows fails the connect
with WSAEADDRINUSE where other systems fail the bind and fall back. The
request then failed with error 13 for a reason another port would have
avoided, which is what the range is searched for in the first place.
Retry the connect with another port from the range. Only for that error, so
a destination which is simply unreachable still costs one attempt, and out of
the same ten tries the bind already spends, so the attempts parentretries
makes are not multiplied. EADDRINUSE joins the errno values proxy.h maps to
their Winsock numbers, without which the test would never match on Windows.
parent_ports gave its three services one window between them, which is how
the case met the collision on the Windows runner. Each has its own now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
