3proxy ignored SIGPIPE only after readconfig(), which starts the
services: a client gone while written to in the meantime killed the
process (as auto's case saw on macOS). Standalone services never ignored
it. Sends pass no MSG_NOSIGNAL, so this is the only guard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A logger (struct logger) is identified by the string log or -l gives it:
"" stdout, "/dev/null" none, "@IDENT" syslog, "&DSN" ODBC, "radius",
anything else a file. Each type (struct logtype) has init, formatting of
a connection into a buffer, writing, close, and a periodic tick for
rotation and flushing; ODBC and RADIUS register their types from 3proxy,
which alone has them. Services and configurations hold the logger they
use; the last to let go closes it, except the standard ones (none,
stdout, stderr, syslog), which are opened at startup and kept.
-l takes every log type now, and a file named by several services or by
both log and -l is opened once. rotate and archiver are bound to the
logger of the last log command when its configuration comes into force,
and every file logger is rotated by the main loop.
Errors with no connection go to a new stderr logger; after daemon, or as
a Windows service, stdout logging writes nothing and errors go to the
last log configured. A log whose writes keep failing is closed and
reopened; while it cannot be, its lines go to stderr and it is retried
every minute. ODBC's own reconnection throttle gives way to this.
Formatting (logbuf) is separate from counting traffic and clearing the
statistics, done once by logdo(), the services' log function, so a later
step can format or copy a connection's line and write it from a thread
of its own. dologname() and timechanged() move to log.c, which the
standalone services link as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loop went through every session three times a turn: for the nearest
deadline, to register its sockets with epoll, and for the deadlines come
and the sessions ended. A datagram cost as much as the sessions there
were, idle ones too.
- The deadlines are in a binary heap, the nearest at its top: the wait
is until the top, and the sessions timed out are taken from the top
down to the first not due. A datagram only moves a session's deadline
later, which leaves the heap as it is: the session is looked at again
when its old deadline comes, and put back where its deadline is then,
once in its timeout however many datagrams it had. A deadline set
earlier by a handler goes up the heap.
- A session ended is put on a list and let go at the end of the turn,
and its sockets are registered with epoll after the loop or a handler
had it, not every session every turn.
- Deadlines are milliseconds of monotime(), CLOCK_MONOTONIC (the
performance counter on Windows), which the time of day being set does
not move: before, a clock set back kept sessions for as long, and one
set forward ended them all. The loop reads it once a turn, in u->now.
With epoll, a datagram costs the same with 20000 idle sessions as with
none (38-40 us of the proxy's CPU a round trip, ARM), where it took 59
before. poll() elsewhere still builds its set every turn.
Tests: a session heard from every half second lives through two seconds
of idle timeout, and ends two seconds after the last datagram.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxymain.c stays the one unit built for 3proxy and included by each
standalone service, and now includes its parts:
mainconf.c services from their configuration lines: options,
srvprepare(), mainprepare(), srvinit(), srvfree(), and
copying and freeing access lists and authentication
mainlisten.c listening sockets, shared and taken over on a reload,
and reattach()
mainclient.c threads for the clients of TCP services, their pool,
srvspawn(), releaseparam(), and the filters' hooks
maintcp.c the main loop of a TCP service, tcploop(), taken out of
srvrun(), with tcpnext() and cblisten()
mainudp.c the main loop of a UDP service
proxymain.c keeps srvstart, srvrun() and main(), and declares the
functions of file scope of the parts so that they can be in any order.
The code is moved, not changed, but for tcploop(). The parts are
dependencies of the objects that include proxymain.c in Makefile.inc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service now runs in its own thread alone (udploop()): it reads the
service's socket and the sockets of its sessions towards their
destinations, keeps their deadlines, and answers every client from the
service's socket. A session is a struct clientparam as a client thread's
would be, with a slot of maxchild, among the service's children and
counted on its listener, but without a thread. The services supply
handlers (struct udpservice, childdef.udp): udppm keeps a session per
client address, forwarding its datagrams, behind the headers of a
parent chain, from a socket of its own and answering what comes back,
until STRING_L of silence or, with -s, the first answer; dnspr makes a
session of each query, answered at once from the fake address, the
name cache or the resolver, or sent on to the first name server over
UDP, whose answer is waited for in the loop up to DNS_TO (a name server
over TCP is still waited for in place). Authentication stays in the
loop for now; datagrams over a bandwidth limit, or of a client without
a free slot, are dropped.
On a reload the old service goes on serving the socket until it sees
its configuration replaced, then hands its sessions to the listener;
the newest service of its kind there, which until then does not read
the socket so that no client gets two sessions, adopts them: a slot of
its own, the old one given back, its filters and limits, and
re-authentication. A session is ended with 26 if the newest service is
of another kind, 27 without a slot, or with what the authentication
says; sessions nobody is left to adopt are ended by the last service to
stop on the listener.
On Windows a send on a socket waited while another thread polled it,
which delayed udppm's answers by up to a second, and old UDP services
did not exit after a reload; with one thread per socket neither
happens, and the second socket bound to the service's address
(NO_SHARE_UDP_SOCKET) is gone from the code and the build files.
udpinit, the global udp_table, udpnext(), udppmchild() and
dnsprchild() are gone; udpsockmap() serves SOCKS5 UDP ASSOCIATE only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a reload cannot take a listening Unix socket over (a bind-time
-ol option changed), the new service binds the path anew, and the old
listener, closing after it, removed the file, the new socket's by then,
leaving nothing to connect to. The socket's file (device and inode) is
now noted on binding and the path removed on closing only if it is
still that file.
structures.h includes <stddef.h> for offsetof() in SASIZE(), which
needs it wherever the header is used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Linux a proxy listening on a Unix socket relayed only the first
bytes of an answer (result 95): sockmap() read with recvfrom() into the
client's address, which for a Unix stream socket Linux fills with an
empty AF_UNIX one, and then sent with sendto() to that address, which a
connected Unix stream socket refuses. A TCP connection has no use for
addresses: sockmap() now sends and receives with NULL ones, and
connect-back sends its command with socksend() and reads the answer
with sockrecvfrom() without an address, which a NULL sin now allows.
An abstract socket name (-iunix:@NAME) was bound with the whole size of
struct sockaddr_un, padded with zeros, so a client naming it by its
bytes, as most do, reached another name. SASIZE() now gives an abstract
address its exact size; getsockname() calls that took SASIZE() of the
old contents as the buffer size take the buffer's size instead.
tcppm refused port 0 for a unix: target, though the manual gives
"tcppm 8080 unix:/var/run/app.sock 0" as the example: the port of a
Unix socket target is ignored, so 0 is allowed.
A test case covers a proxy on a Unix socket and on an abstract one on
Linux, tcppm to a Unix socket, a parent reached through one, and a
listening Unix socket across reloads: taken over each time, its file
kept, and removed once no service listens on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listener counts the traffic of its clients as it is logged: bytes from
servers (%I) and from clients (%O). clearstat(), which every log writer
calls once a line is logged, as well as lognone and logradius, adds the
counts to the client's listener before clearing them, through
listenertrafficfunc, set where services have listeners and NULL in the
standalone binaries.
The admin services page shows them as listenertrafin and
listenertrafout. The takeover case checks that they go on through three
reloads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listener counts its clients, whichever of its services they came to,
old or new: those connected now, raised when a client is spawned, and
those done with, raised as one leaves its service. As the listener
outlives a reload, so do the counts.
The services page of the admin interface shows them for each service as
listenerclients and listenerprocessed; services sharing a socket show
the same counts. The takeover case checks that the count goes on
through three reloads and that no client is left counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Services of one configuration on the same address, port, network
namespace and device now share one listener and its socket, all
accepting on it, instead of each binding its own: on Linux the second
bind failed unless SO_REUSEPORT was set. A listener keeps a list of the
services accepting on it; the first to start opens the socket under the
listener's open_mutex, as binding may take long, and the others use it
("Accepting connections, shared"). The socket is closed once the last of
them stops.
Such services must be of one type with the same -ol options: otherwise
the configuration is refused ("a service of another type listens
there", "a service with other socket options listens there").
mainclaim() now runs for every service, at startup too: it joins the
listener of its configuration for the address, takes over one of an
older configuration (one of its type first), or makes a new one, under
one hold of listener_mutex so that a listener found is not freed before
it is joined.
Services connecting back (-r) to one place share a pseudo-listener
without a socket, of any type. After a reload a client moves to one of
the newest services of its type on its listener with a slot free: 26 if
there is none of its type, 27 if none has a slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the -ol options which act when a socket is bound (SO_REUSEADDR,
SO_REUSEPORT, SO_EXCLUSIVEADDRUSE, SO_PORT_SCALABILITY,
SO_REUSE_UNICASTPORT, IP_TRANSPARENT, TCP_FASTOPEN) must be the same
for a socket to be taken over. The others are set on the socket taken
over, those no longer given turned off; TCP_MAXSEG, whose default is
not 0, still needs a new socket when it is dropped.
Several services may still share an address, in one configuration or
across reloads: each new one takes over a listener of its own, one of
an old service of its type first, or opens another.
A service connecting back (-r) has no socket to share, but now a
listener without one for the place it connects to: on a reload the new
service connecting to the same place takes it over, and the old
service's clients are re-authenticated as its clients, as a listening
service's are.
Nothing moves from srvparam to the listener: its socket is what accept
and UDP sessions use, and the rest is the service's configuration,
which the listener only copies to compare.
Tests: a live option changed keeps the socket, a bind-time one makes
another, two services on one address are both taken over, and a
download through a connect-back service goes on or is cut as its user
is kept or removed. 3proxy(8) describes the takeover and
re-authentication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UDP sessions check for a reload in udpsockmap() as TCP connections do
in sockmap(): unless noforce, a session goes by the newest service on
its listener, re-authenticated as its client, or is dropped.
A request on a connection that outlives a reload, an HTTP one kept
alive, is authorized by clientauth(): it moves the connection to the
newest service on its listener first if a reload came since, then
authorizes as the service whose rules apply (PSRV). The proxy uses it
for every request, so a request of a user the new configuration
removed is answered 407 rather than authorized by the old rules and
cut while relayed. reattach() takes whether to authenticate: sockmap()
and udpsockmap() move and authenticate at once, clientauth() moves and
then authorizes the request.
A test case covers the drops: with one slot free in the new service one
of two downloads moves on and the other is cut (27; a service keeps one
of its maxconn slots for the next connection it accepts), and a proxy
download is cut when a socks service takes its port (26). The reload
case checks the kept-alive request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service's socket is kept as a listener like a TCP one, the
protocol part of what must match: on a reload a new UDP service on the
same address takes it over. It is the way the old service's sessions
answer as well, and they go on answering through it; the new service
reads what comes, a datagram of an old session forwarded to it as
before, sessions being found by the service's and the client's
addresses whichever service made them. The socket stays open while the
old sessions do.
The takeover case keeps a udppm session going through three reloads
and wants every datagram back; with the build before this datagrams
were lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reload, unless noforce, a client of an old service goes by the
rules of the newest service on its listener, the one accepting there
now. It takes one of that service's maxconn slots, which counts it
there and keeps that service and its configuration alive until the
client is done, and is re-authenticated as its client: access lists,
users, authentication, bandwidth limits and traffic counters come from
the new configuration. A later reload moves it on, giving the slot
back. The client stays its own service's child, which keeps its
filters and TLS state.
It is dropped if its service was removed or is of another type now
(26), or if the newest service has no free slot (27); what the
re-authentication refuses ends it as before. A service without a
listener (UDP, connecting back) keeps its clients as they are.
This restores what force did before the configuration was kept per
service: since then a client compared its version with its own
service's configuration, which never changes, and was never checked
again. Services now take their configuration's version as it is: a
reload numbers the new configuration before reading it, and so does
startup now, where services were numbered one ahead.
A listener now stays with its old service until that service's last
client is gone, those clients looking the newest service up through
it; the socket itself is still closed as soon as nobody accepts on it.
Connection limits are given back in the configuration they were taken
in. PSRV() is the service whose rules apply to a client.
The reload case checks that a download of a user the new configuration
keeps finishes under it, and that one of a user it removes is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service's TCP listening socket is kept as a listener: when a reload
commits, before the old services are told to stop, each new service
takes over the listener of an older configuration's service on the same
address, port, network namespace and device with the same -ol options.
It skips socket() and bind(), calling only listen() again for its own
backlog, and logs "Accepting connections, taken over". The old service
stops accepting at its next poll and lets go of the socket; whoever
lets go last closes it, unlinking a Unix socket's path.
So a service staying where it was no longer leaves a gap: before, the
old one closed its socket and the new one opened another, and a
connection queued on the old socket was reset. UDP services, whose
clients read from the service's socket, and two services of the same
configuration on one address keep their own sockets as before.
A test case makes requests nonstop through three reloads and wants none
refused or reset; against the build before this, connections were
reset. The reload case no longer waits out the switch. SIGUSR1 in
3proxy(8) describes how a reload behaves now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reload reads the file into a new configuration, allocated from the
defaults, and prepares its services without starting them. Only if the
whole file is read without an error does it come into force: the old
configuration's services stop accepting, the new ones start. On an
error everything prepared is dropped and the old configuration, which
never stopped, stays in force.
Each service holds its configuration until it is freed, after its last
client; a configuration no longer in force is released when the last
holder lets go (confhold/confput). freeconf() now only releases what a
configuration holds, touching nothing global.
What runs without a service of its own reads the configuration in
force, runconf, rather than conf, which a reload points at the one
being read: the resolver, which holds it while it waits (a client
resolves with its own service's), the log, the thread pool, the admin
pages, which walk the services under config_mutex, and the signal
handlers.
The users table, havelog and the resolver choice are per configuration
now, the resolver applied when the configuration comes into force, so
a refused file leaves the users as they were. The log file, the
configuration file name, the string table, daemon mode and the filter
list belong to the process and are handed from one configuration to
the next. Settings left out of a new file go back to their defaults,
where before they kept the old values.
The gap between the old and the new listening sockets remains: a
connection queued on an old service's socket when it closes is reset.
Handing the socket over is the next phase.
A test case reloads a broken file (nothing changes, alice still gets
in, bob from the refused file does not) and a good one (bob only, the
service on its new port, the old one closed, a download through the
old service finishing whole). 40 reloads, a third refused, under load
show no memory errors under AddressSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- readconfig() kept its line buffer and argument array on every error
return, so each refused reload leaked them.
- A filter refusing a connection in filter_client (TLS: a failed
handshake) was not kept, so its filter_clear never ran and the state
it made was lost. makefilters() now clears it, and empties the slot
first: a pooled worker's slot may hold the previous connection's.
- ssl_filter_close() freed the client CA paths but not the server ones,
for every TLS service set up.
- Compiled patterns were never freed: an ACL's (pcre: and regex: host
names), an httpsrv rule's. copyacl() shared them between the
configuration's ACL and each service's copy; each copy now compiles
its own, and freeacl() and freehttprules() free them.
What LeakSanitizer still reports is OpenSSL's per-thread random
generator state of pooled threads, released only by OpenSSL at exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s[N]:MIN,STREAK,SMALL sets where the automatic mode splices, as -g
does for grace delay: a direction is spliced after STREAK reads in a
row of MIN bytes or more (SMALL from a TLS socket), copied again after
more than STREAK splices in a row of less than SMALL, and a body of
known length is spliced if MIN or more of it is left. The defaults,
16384,2,4096, are what was built in; a value left out or 0 keeps its
default.
The man page describes it, and its HTML copy, which still said splice()
was not built by default, now has the current text. A test case relays
answers of every size, chunked ones and a 3 MB upload through every
mode, two of them with heuristics set, and compares them byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service keeps the configuration it was started with (srvparam.config)
and its threads and its clients' read it there: limits and counters
(alwaysauth, the limiter), RADIUS, dnspr's name servers, and the checks
for a reload in the accept loop and in sockmap(). The configuration is
still reloaded in place, so it is the current one for now; this is what
lets a reload build a new one while the old one still serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf is now a pointer to the current configuration rather than the
structure itself: at first confboot, which exists before main() runs, as
plugins and the standalone services use it too; later configurations
will start from confdefaults. Both are made from confinit.h, the one list
of initial values. The timeouts are an array of the configuration
instead of a pointer to a global one. Nothing behaves differently yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They are the state of the process, not configuration: conf.time (now
curtime), timetoexit and needreload are globals of their own. Plugins
reach needreload through a pointer added at the end of struct
pluginlink; LdapPlugin uses it. The extern for a paused global, never
defined, goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the timeouts, each service now keeps the values given before it
instead of reading the global ones while its connections run. setopts()
takes the service's maxseg. The TLS handshakes wait the service's
CONNECT_TO, kept in its SSL configuration, rather than the global one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s takes the directions to splice: -s1 what the server sends (downloads),
-s2 what the client sends (uploads), -s3 both. -s alone (-s7) splices both
only where it pays, as -s5 and -s6 do for one direction:
- a body of known length is spliced if at least 16K of it is left to read
once what is already buffered is counted, so an answer read whole with
its headers is just copied;
- a stream of unknown length starts copied, is spliced after 2 reads in a
row of at least 16K, and goes back to copying after 3 splices in a row
of less than 4K, once its pipe is empty.
Pipes are made when a direction first splices and kept with the
connection for its next mappings, so a kept-alive connection no longer
makes and closes two pipes per request; one left holding data by an
error or a timeout is closed. The direction not spliced keeps its buffer.
Before, -s spliced both directions and every mapping made both pipes,
which made small answers slower than without splice: with kept-alive
50K answers 2 pipe2() and 4 close() per request, now none.
Also corrects the default buffer size in the -b help (65536 for TCP).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
