Names are resolved with getaddrinfo(), thread-safe everywhere: what was
left of gethostbyname() (my_gethostbyname() over gethostbyname_r(), in
its glibc and Solaris forms, GETHOSTBYNAME_R, the mutex for the plain
one) served only the IPv4-only resolver already gone. Makefile.Solaris
(Sun's compiler, _SOLARIS) and Makefile.unix go: CMake builds for the
other systems; the documents say so, and their examples use
Makefile.FreeBSD.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows CE (_WINCE: WinMain() and its window, the command line made
into argv, the paths and functions it went without) and Windows 9x
(--install and --remove through the RunServices key of the registry,
which is what GetVersionEx() was asked for) are no longer built for.
--install and --remove use the Service Manager alone; Winsock 2.2 is
asked for, which WSAPoll() needs, rather than 1.1.
The service command of a configuration, deprecated, still does
nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Win32 "lite" build with Open Watcom (Makefile.watcom, its release
workflow, the CMake branch for it) was the only one built without IPv6
(NOIPV6), without fibers and event loops, with a select() of its own
for poll() (mypoll()), and with the calling conventions of Watcom's
pragmas in the plugins. They are gone with it: every build has IPv6,
and polls with poll() on Unix and WSAPoll() on Windows, which were
what every other build asked for (WITH_POLL, WITH_WSAPOLL, the
3PROXY_USE_POLL and 3PROXY_USE_WSAPOLL options of CMake, FD_SETSIZE
for select()). getip(), the IPv4-only resolver of NOIPV6, goes too.
Plugin binary compatibility: plugins built with Open Watcom, and their
calling convention, are no longer supported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
glibc's getaddrinfo() sizes what it takes with alloca(), IPv6 most, by
the thread's stack, not by the fiber's it may be called on: a client
moved to a new configuration on a reload goes on on its fiber, and the
new configuration's services may resolve with the system resolver,
which otherwise is never called on a loop (threadperclient). A fiber
yields to have it called on the loop thread's own stack
(onthreadstack()), and goes on with its answer; the loop waits for it
as the fiber would have.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A logger whose thread could not be started is written by its clients
themselves. A client of a loop, a fiber or a callback on the loop's
thread, would hold the loop while a database or a server takes its
line, so on a loop's thread the line is dropped instead, counted as the
log's lines dropped are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A loop waits for sockets with the polls of AFD, the driver under
Windows' sockets, as epoll and kqueue do elsewhere: each socket's poll is
given to \Device\Afd once and completes to the loop's I/O completion
port, which a message from another thread is posted to as well. WSAPoll()
stays the backend with LOOP_POLL and for Watcom and Windows CE.
A poll reports once and has to be made again. Edge-triggered sockets, the
registrations of fibers' sockets, are polled again for what an operation
has found them not to be ready for: loopagain(), called where readiness
known is found gone, a no-op with epoll and kqueue. A poll in flight for
less than wanted is cancelled and made again as it completes; one of a
socket deleted is freed as it completes.
With it the sockets of fibers stay registered on Windows too (SOCKREGS):
the Windows socket functions tell the loop of a socket found not ready
and of one closed, as the Unix ones do, and registrations are kept by the
handle divided by 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The socket of a connection to the destination is bound to the external
address before connect(), with port 0 unless a range is given. bind()
then takes an ephemeral port for the socket alone: not one in use
towards another destination, nor one of a connection still closing in
TIME_WAIT, which connect() would reuse. With many connections the range
runs out and bind() fails, result 12, or searches long for a port: 7 ms
a call was seen. The more connections are open at once, the sooner:
with event loops, serving more of them at once, the client of every
tenth connection or so of a saturated relay got its end at once.
Where the system has IP_BIND_ADDRESS_NO_PORT (Linux 4.2), the port is
left to connect() for a socket that is going to connect: chosen for
the pair with the destination, together with IP_LOCAL_PORT_RANGE when
a range is given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap waits on its splice() pipes as on its sockets, and on an event
loop a pipe waited on is registered there like a socket. They were
closed with close(), past the socket functions, so the loop kept the
registration of a closed descriptor, which epoll and kqueue drop on the
close: a socket given the same number later was taken for it, never
registered, and its fiber waited on it forever. fdclose() tells the
loop first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fiber waiting on a socket registered it with its loop and took it
off again on every wait: two epoll_ctl() calls a wait, some 39 for an
exchange of the mix benchmark, about an eighth of the proxy's time. A
socket is now registered once, edge-triggered for reading and writing,
until it is closed, and the loop keeps what it was reported ready for
until an operation finds it not to be: read, write, accept, connect,
splice() from a pipe and TLS tell it so through fiberblockfunc, the
close through fiberclosefunc. A wait for what a socket is known to be
ready for is answered at once, with no system call and no switch; only
with none of them ready does the fiber wait in its loop. Readiness is
trusted once: answered again with nothing reported or found since, what
was there may have been taken without its end being found, and the
system is asked with poll(). epoll and kqueue only; with poll() each
wait still registers its sockets.
What takes readability for an end rather than reading, the control
connection of a UDP association, the client waiting for a SOCKS BIND,
the server between requests of a kept-alive HTTP connection, asks
first with sockreadable(), a peek below any TLS layer.
kqueue: the end of either side, the peer's or our own shutdown, is
readable or writable, not a hang up, which poll() reports only with
both: kept, it made a wait for the answer after our shutdown return
at once, and sockmap give up with 93.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Client, outgoing and FTP data sockets, and the inetd socket of a
standalone service, were given SO_LINGER, the client's for the long
string timeout (60 seconds by default), the outgoing one's for the
LINGER timeout. It kept nothing from being lost: without it the system
sends what is queued after the close all the same, and the time running
out reset the connection, losing what was still queued. All it did was
make close() wait for the peer to acknowledge the data, holding the
connection's thread and maxconn slot meanwhile, and, for clients served
on an event loop, every client of the loop: with one loop it was idle a
quarter of the time in a benchmark of many short exchanges.
The LINGER timeout keeps its use: how long a half-closed connection
waits for the rest of its data, and how long httpsrv reads what is left
of a request before closing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -L, or loopmode on for the services which follow, the clients of a
TCP service are served on event loops: loops threads (by default one
for each CPU) each run a loop, and each client is given in turn to one
of them, which serves it on a fiber with the same code a thread runs.
Its waits in sockpoll() wait in the loop, its sockets and a timer
registered there, and the fiber goes on once one is ready or the time
is out; a sleep is a timer alone. A client costs a stack, as large as
its thread's would be, and no thread. The loops start with the first
client served on one and run as long as the process; each keeps up to
256 idle fibers for its next clients.
srvspawn() posts the client to its loop once it is counted, what is
posted made before so that posting cannot fail. A service whose clients
cannot share a thread (threadperclient), or listening for connections
back (-R), all of its clients waiting for one socket, which a loop
waits for once, keeps a thread for each client, as does every service
of a build or a processor without fibers, and the standalone services.
tests/run.py --loopmode runs every configuration with loopmode on, on
two loops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The clients of a -R service all wait for the connection back on the
same listening socket, which was blocking: one connection woke them
all, one took it, and the others were left in accept() until the next,
past their time out and through a reload. The socket is non-blocking,
and an accept() which finds nothing, the connection taken by another
client or gone, is not one of the client's tries. nonblock() makes a
socket non-blocking, acceptagain() tells such an accept().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A layer over the sockets, TLS or a plugin's, no longer waits itself.
Its _prepoll only tells what it knows without asking the system: the
fds it knows to be ready, data TLS has already decrypted, or, returning
0, that the sockets have to be polled, for events it may rewrite: a
TLS read waiting for its socket to be writable. The wait is
sockpoll()'s, which calls _prepoll first and then polls, so that a wait
can be made other than with poll() in one place: a client of an event
loop is to wait in the loop. Whatever a rewritten wait gets is given to
the caller as what it asked for, its chance to try again, and the
events it asked for are given back. With no fds sockpoll() sleeps,
which WSAPoll() would refuse.
TLS remembers what a read or a write it could not do waits for. A read
needing to write, or a write to read, was an error which ended the
connection, as when TLS 1.3 answers a key update while the socket
cannot take more; it now waits for what TLS needs, as one needing what
it was asked for always did.
Incompatible for plugins, which have to be built again with the same
version of 3proxy, the structures they share not kept compatible:
- struct sockfuncs has _prepoll(state, fds, nfds) in place of
_poll(state, fds, nfds, timeout). A plugin replacing _poll replaces
_prepoll: it must not wait, returns the number of fds it set revents
of (events then left as they are), 0 for the fds to be polled (with
events as it leaves them, revents left to the poll) or -1, and calls
the _prepoll it replaced when it has nothing to say itself.
- Waiting for sockets is sockpoll(sf, state, fds, nfds, timeout), not
so._poll() of the socket functions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A peer gone is then an error of the send rather than SIGPIPE. SIGPIPE
stays ignored for what cannot pass the flag: TLS writes and splice().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loop went through every session three times a turn: for the nearest
deadline, to register its sockets with epoll, and for the deadlines come
and the sessions ended. A datagram cost as much as the sessions there
were, idle ones too.
- The deadlines are in a binary heap, the nearest at its top: the wait
is until the top, and the sessions timed out are taken from the top
down to the first not due. A datagram only moves a session's deadline
later, which leaves the heap as it is: the session is looked at again
when its old deadline comes, and put back where its deadline is then,
once in its timeout however many datagrams it had. A deadline set
earlier by a handler goes up the heap.
- A session ended is put on a list and let go at the end of the turn,
and its sockets are registered with epoll after the loop or a handler
had it, not every session every turn.
- Deadlines are milliseconds of monotime(), CLOCK_MONOTONIC (the
performance counter on Windows), which the time of day being set does
not move: before, a clock set back kept sessions for as long, and one
set forward ended them all. The loop reads it once a turn, in u->now.
With epoll, a datagram costs the same with 20000 idle sessions as with
none (38-40 us of the proxy's CPU a round trip, ARM), where it took 59
before. poll() elsewhere still builds its set every turn.
Tests: a session heard from every half second lives through two seconds
of idle timeout, and ends two seconds after the last datagram.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each service resolves names the way the last of defaultresolve (the name
servers in force, configured or the system's with the hosts file),
fakeresolve (127.0.0.2) and sysresolve (the C library) read before it
says; the last one in the configuration also applies outside services.
defaultresolve is the new command, and the default.
The name servers and the hosts file no longer depend on the mode: nserver
lines, or the system's, are always in force, so dnspr keeps forwarding;
with fakeresolve it does not, as before. fakeresolver moves to common.c,
which the standalone services link without resolve.c.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A listener counts the traffic of its clients as it is logged: bytes from
servers (%I) and from clients (%O). clearstat(), which every log writer
calls once a line is logged, as well as lognone and logradius, adds the
counts to the client's listener before clearing them, through
listenertrafficfunc, set where services have listeners and NULL in the
standalone binaries.
The admin services page shows them as listenertrafin and
listenertrafout. The takeover case checks that they go on through three
reloads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UDP sessions check for a reload in udpsockmap() as TCP connections do
in sockmap(): unless noforce, a session goes by the newest service on
its listener, re-authenticated as its client, or is dropped.
A request on a connection that outlives a reload, an HTTP one kept
alive, is authorized by clientauth(): it moves the connection to the
newest service on its listener first if a reload came since, then
authorizes as the service whose rules apply (PSRV). The proxy uses it
for every request, so a request of a user the new configuration
removed is answered 407 rather than authorized by the old rules and
cut while relayed. reattach() takes whether to authenticate: sockmap()
and udpsockmap() move and authenticate at once, clientauth() moves and
then authorizes the request.
A test case covers the drops: with one slot free in the new service one
of two downloads moves on and the other is cut (27; a service keeps one
of its maxconn slots for the next connection it accepts), and a proxy
download is cut when a socks service takes its port (26). The reload
case checks the kept-alive request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reload, unless noforce, a client of an old service goes by the
rules of the newest service on its listener, the one accepting there
now. It takes one of that service's maxconn slots, which counts it
there and keeps that service and its configuration alive until the
client is done, and is re-authenticated as its client: access lists,
users, authentication, bandwidth limits and traffic counters come from
the new configuration. A later reload moves it on, giving the slot
back. The client stays its own service's child, which keeps its
filters and TLS state.
It is dropped if its service was removed or is of another type now
(26), or if the newest service has no free slot (27); what the
re-authentication refuses ends it as before. A service without a
listener (UDP, connecting back) keeps its clients as they are.
This restores what force did before the configuration was kept per
service: since then a client compared its version with its own
service's configuration, which never changes, and was never checked
again. Services now take their configuration's version as it is: a
reload numbers the new configuration before reading it, and so does
startup now, where services were numbered one ahead.
A listener now stays with its old service until that service's last
client is gone, those clients looking the newest service up through
it; the socket itself is still closed as soon as nobody accepts on it.
Connection limits are given back in the configuration they were taken
in. PSRV() is the service whose rules apply to a client.
The reload case checks that a download of a user the new configuration
keeps finishes under it, and that one of a user it removes is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reload reads the file into a new configuration, allocated from the
defaults, and prepares its services without starting them. Only if the
whole file is read without an error does it come into force: the old
configuration's services stop accepting, the new ones start. On an
error everything prepared is dropped and the old configuration, which
never stopped, stays in force.
Each service holds its configuration until it is freed, after its last
client; a configuration no longer in force is released when the last
holder lets go (confhold/confput). freeconf() now only releases what a
configuration holds, touching nothing global.
What runs without a service of its own reads the configuration in
force, runconf, rather than conf, which a reload points at the one
being read: the resolver, which holds it while it waits (a client
resolves with its own service's), the log, the thread pool, the admin
pages, which walk the services under config_mutex, and the signal
handlers.
The users table, havelog and the resolver choice are per configuration
now, the resolver applied when the configuration comes into force, so
a refused file leaves the users as they were. The log file, the
configuration file name, the string table, daemon mode and the filter
list belong to the process and are handed from one configuration to
the next. Settings left out of a new file go back to their defaults,
where before they kept the old values.
The gap between the old and the new listening sockets remains: a
connection queued on an old service's socket when it closes is reset.
Handing the socket over is the next phase.
A test case reloads a broken file (nothing changes, alice still gets
in, bob from the refused file does not) and a good one (bob only, the
service on its new port, the old one closed, a download through the
old service finishing whole). 40 reloads, a third refused, under load
show no memory errors under AddressSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf is now a pointer to the current configuration rather than the
structure itself: at first confboot, which exists before main() runs, as
plugins and the standalone services use it too; later configurations
will start from confdefaults. Both are made from confinit.h, the one list
of initial values. The timeouts are an array of the configuration
instead of a pointer to a global one. Nothing behaves differently yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf.haveerror was only initialised and srvparam.srvfds only declared;
the extern declarations of stdlog and demon named globals that do not
exist (both are fields of conf); a local in make_ace() and one in the
webadmin XML page were never used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They are the state of the process, not configuration: conf.time (now
curtime), timetoexit and needreload are globals of their own. Plugins
reach needreload through a pointer added at the end of struct
pluginlink; LdapPlugin uses it. The extern for a paused global, never
defined, goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nservers, numservers and authnserver (resolve.c, common.c) and
radiuslist, nradservers and radiussecret (authradius.c) were globals of
their own; they are configuration, set by nserver, authnserver and
radius, and now live in struct extparam with the rest of it.
MAXNSERVERS, MAXRADIUS and struct radserver move to structures.h, which
plugins include without proxy.h.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dialer ran a program when a name could not be resolved, for dial-up
links; with nscache it did not even run. The command is still accepted
and ignored, so configurations using it keep loading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the timeouts, each service now keeps the values given before it
instead of reading the global ones while its connections run. setopts()
takes the service's maxseg. The TLS handshakes wait the service's
CONNECT_TO, kept in its SSL configuration, rather than the global one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside Linux nssocket() was a macro, and two of its callers have #ifdef
WITH_UN among its arguments. Preprocessor directives in macro arguments
are undefined behaviour: gcc and clang take them, MSVC does not. It is a
function on every system now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extport binds a local port out of its range and connects from it. A port
which is free to bind may still be half of a pair which is closing from an
earlier connection to the same destination, and Windows fails the connect
with WSAEADDRINUSE where other systems fail the bind and fall back. The
request then failed with error 13 for a reason another port would have
avoided, which is what the range is searched for in the first place.
Retry the connect with another port from the range. Only for that error, so
a destination which is simply unreachable still costs one attempt, and out of
the same ten tries the bind already spends, so the attempts parentretries
makes are not multiplied. EADDRINUSE joins the errno values proxy.h maps to
their Winsock numbers, without which the test would never match on Windows.
parent_ports gave its three services one window between them, which is how
the case met the collision on the Windows runner. Each has its own now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
