loopmode is cb unless a configuration says otherwise, as -L2 had it
for a service: its clients on event loops, a connection's data relayed
by the loop itself where nothing but the sockets is between its sides.
A service whose clients cannot share a thread (threadperclient: PAM,
LDAP, FilePlugin, names resolved by the system, -ne) or listening for
connections back (-R) still gives each a thread, as do the standalone
services and builds without fibers. loopmode on and -L keep the loops
with the relay on the client's stack; loopmode off and -L0 a thread
for each client.
The man page tells of cb. tests/run.py --loopmode takes the mode
(cb, on, off) to give every configuration, with two loops; the Linux
workflow runs the suite once more with a thread for each client, which
is no longer what a configuration gets by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() and the callbacks' relay each had their own 64K for what a
pipe is given at most (MAXSPLICE, CBSPLICE). It is PIPESIZE now, one
for both, 64K as before: what a pipe holds unless asked otherwise.
Built with -DPIPESIZE=<bytes>, pipes are made that large with
F_SETPIPE_SZ (pipenew()); one the system will not make so large is not
used, and the data is copied, as when there is no pipe to be had.
Measured on ARM and Intel (mix-http, mix-relay), 16K and 32K were
slower, 128K and 256K no faster than 64K, which stays the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fiber's stack was a thread's, PTHREAD_STACK_MIN included: 128K on
arm64 glibc, which is there for a thread's signals, and a loop's thread
takes none. It is BASESTACKSIZE with the service's stacksize added, and
the room kept for wolfSSL's handshake (TLSSTACKSIZE), as a thread's but
for that. Only reserved either way: measured with OpenSSL, a fiber holds
8K to 12K of it, TLS handshakes included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
glibc's getaddrinfo() sizes what it takes with alloca(), IPv6 most, by
the thread's stack, not by the fiber's it may be called on: a client
moved to a new configuration on a reload goes on on its fiber, and the
new configuration's services may resolve with the system resolver,
which otherwise is never called on a loop (threadperclient). A fiber
yields to have it called on the loop thread's own stack
(onthreadstack()), and goes on with its answer; the loop waits for it
as the fiber would have.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() made both buffers again as it returned, for whatever came
after it, and the callbacks before the end: after the last mapping there
is only the end, the log line and a refusal sent from a buffer of its
own, with no use for them. They are made again only for a mapping that
is not the last, and for a fiber mapping on after a reload; the
functions reading a side's buffer make it if it is not there, and a
worker taken from the pool for its next connection gets its own again.
proxy.c's other last mappings, a CONNECT through a parent and an answer
ending as the server closes, are marked so too (maplast(), proxytail),
as the CONNECT of its own was: their end, and with callbacks the fiber,
is the loop's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fiber's wait asks the system again what each socket was reported ready
for and answered with once already: one poll() for each of them, two for
an HTTP connection kept alive waiting on both of its sides. They are
asked all at once now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() of fibers and threads kept both buffers for as long as it
mapped, even with a direction spliced. On a connection's last mapping
(maptail set), as the callbacks already do, a direction spliced lets its
empty buffer go: bufdrop(), shared with the callbacks now. It is made
again before a read into it, back to copying or draining what TLS held,
and both are there again once sockmap() returns, for whatever comes
after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A logger whose thread could not be started is written by its clients
themselves. A client of a loop, a fiber or a callback on the loop's
thread, would hold the loop while a database or a server takes its
line, so on a loop's thread the line is dropped instead, counted as the
log's lines dropped are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A loop waits for sockets with the polls of AFD, the driver under
Windows' sockets, as epoll and kqueue do elsewhere: each socket's poll is
given to \Device\Afd once and completes to the loop's I/O completion
port, which a message from another thread is posted to as well. WSAPoll()
stays the backend with LOOP_POLL and for Watcom and Windows CE.
A poll reports once and has to be made again. Edge-triggered sockets, the
registrations of fibers' sockets, are polled again for what an operation
has found them not to be ready for: loopagain(), called where readiness
known is found gone, a no-op with epoll and kqueue. A poll in flight for
less than wanted is cancelled and made again as it completes; one of a
socket deleted is freed as it completes.
With it the sockets of fibers stay registered on Windows too (SOCKREGS):
the Windows socket functions tell the loop of a socket found not ready
and of one closed, as the Unix ones do, and registrations are kept by the
handle divided by 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
loopsstart() runs on the thread of the service whose client is the first
one to be served on a loop, not on the configuration's thread: two
services given their first clients at the same time could each initialize
its mutex, take it, and start loops of their own. A loop of the second
array was then given to a thread of the first one as well: two threads
ran one loop, and its timers and fibers were taken by both (crashes in
the timers' heap, seen on Windows under load).
The mutex is a global one now, made at startup with the others.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ends of a client relayed as callbacks called releaseparam() and
then freeparam(), which releases it again: the connection limits' slot
was given back twice (see the commit before). freeparam() alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A connection's two data buffers are malloc()'d with it (bufalloc())
instead of being part of struct clientparam, which is CLIENTBUFSIZE*2
smaller. The callbacks' relay of a client whose fiber gave up its stack
frees a side's buffer while it is empty and makes it again to read, and
gives both back before sockmap() or the child's tail goes on: a pool of
idle relayed connections holds no buffers. Fibers and threads keep
theirs as before.
mix scenarios, callbacks (-L -L2), peak memory before/after, throughput
the same:
Intel: relay 972/632 MB, connect 981/581, socks 992/594
ARM: relay 1000/673 MB, connect 1000/615, socks 1020/629
struct clientparam's clibufdata and srvbufdata are pointers now:
plugins built against the old structure are not compatible.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A way with no length to keep to (CBALL) was counted down as data was
written, so the check for it failed after the first write: the run of
large reads that switches to splice() never got past one, and a stream
was copied all along. Not counted down any more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
give up their fiber while relayed
The children for which mapping is their last thing set their tail too:
- socks, for CONNECT: its log line; the request's buffer is freed before
mapping, nothing needing it after;
- proxy, for CONNECT: the status page its end sends for the result
where one is, now proxystatus() for both, and the log line of its
request, a copy of which is the tail's. Not in a request httpsrv hands
over, whose end is httpsrv's;
- pop3p, imapp (with a copy of its tag), smtpp (its command freed before
mapping): the log line and the error reply their ends send.
A tail may wait now, sending to the client: once mapping is over the end
of a client whose fiber gave up its stack runs on a fiber taken again
for it, which, after a reload, has sockmap() go on with the mapping
first. Only with no stack to be had is it ended there, logged with no
tail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A child for which mapping is the last thing it does (tcppm, tlspr) sets
maptail, what it does once mapping is over as its own end would: its
log line, prepared before for tlspr. Relayed by the loop, the client's
fiber then gives up its stack at once: left in the middle of the
child's code, never to be resumed, it is started over from the start of
its stack for the next client (fiberreset), or freed where it cannot be
(Windows). Once mapping is over the loop ends the client as the child
and its fiber would: the result, the tail, the connection released and
freed. After a reload, which sockmap() is to see to, the client is
given a fiber again for sockmap() to go on with the mapping, then its
end. Not while a reload is pending: sockmap() sees to it on the fiber.
The callbacks' state is no longer on the fiber's stack but allocated.
clientparam has maptail and maptaildata: plugins are to be built again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What was left to sockmap() is relayed by the callbacks too:
- bandwidth limits: the pause bandlimfunc asks for after data moved is
a timer, the sockets' events waiting for it;
- grace (-g): small reads many times this second, either way, pause
for the grace delay before more, as sockmap() waits before polling;
- traffic limits (maxtrafin, maxtrafout): the mapping ends with 10;
- dumps to the log (logdumpsrv, logdumpcli) as the counts pass them;
- control sockets (ctrlsock, ctrlsocksrv): registered with the loop,
anything from them or their end ends the mapping;
- data filters, before data is sent, which may take another buffer
for it, a buffer of a filter given back once empty (bufrelease);
- the filters before the data, run once on the fiber: TLS begins
there. TLS, or anything else between the socket functions and the
socket, is read whatever the socket says when _prepoll() says it
holds data already, and copied, not spliced;
- back to copying after more than splicestreak spliced reads in a
row smaller than splicesmall, once the pipe is through, and from
copying to splice only for data of no known length, as sockmap().
A pipe still holding data is closed on every end of the mapping.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Given back to sockmap() once its pipes were empty, the relay of the
callbacks went on filling them meanwhile: a steady stream kept them
from emptying, and the connection from going by the newest rules.
Once a reload is seen nothing more is read, the pipes are emptied
within LINGER_TO, or the connection ends as timed out, and sockmap()
takes over as soon as they are, not with the next event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a reload the relay of the callbacks gives the connection back to
sockmap(), which knows nothing of what its pipes hold, nor of how much
of a body of known length has already gone through: what was in a pipe
would have come out of order, and the whole length of a body been read
again, into what follows it. The pipes are emptied first, the relay
going on meanwhile, and the length left of a body is the one sockmap()
is given.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each way of the relay, the client's data and the server's, splices as
sockmap() does where it may: from the start when told to (-s1, -s2,
-s3) or for a body of known length of splicemin or more, and with -s
(automatic) once splicestreak reads in a row have been as large. Not
back to copying once spliced, as sockmap() is: a draft. What a way has
buffered goes first, its pipe is used once that is through; a pipe
left holding data is closed, as sockmap() does. The relay is reworked
around one way, the same code both ways.
The callback's hook is given what sockmap() was given to splice, less
what its filters forbid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A body of known length (waitclient64, waitserver64), the HTTP proxy's
requests and answers, was left to sockmap(). The relay of the callbacks
moves it as sockmap() does: that way only, as much as it has left, the
other way not at all, and ends with 98 or 99 once it is all through.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() has a connection go by the newest configuration's rules after
a reload (reattachfunc), which may end it. The relay of the callbacks
did not, and a connection the new rules refuse went on. Once the
configuration is newer than the connection, it is given back to
sockmap() as it is, which does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The data of a connection mapped by a fiber of a loop is relayed by the
loop itself, on its own stack: the sockets registered with it call back
a relay moving what they are ready for, a read and a write each way a
round, up to 16 rounds before the rest waits for the loop's next turn.
The fiber, parked meanwhile, is resumed once the mapping is over, as
sockmap() would end it, with its result: the idle timeout, shortened to
LINGER_TO once a side has ended, half-close or not as the service has
it, errors, and data left.
Only a plain relay: with filters, limits of bandwidth or traffic,
grace, dumps to the log, TLS or anything else between the socket
functions of the system, a control socket or a body of known length,
sockmap() maps it itself. No splice() yet: data is copied, so it
compares with -s0. The fiber keeps its stack.
For benchmarks: -L2 on a service (with -L for the count of loops), or
loopmode cb.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a socket was reported ready for is trusted once: answered again
with nothing reported since, the system is asked with poll(). Counted
per socket, the count was set back by any wait, even one woken by its
timer with nothing reported, and by any event of the socket, even for
the other direction. sockmap(), finding nothing to do after an answer,
sleeps in such a wait: the readiness found not to be there, a splice()
into a full pipe given the socket's, was trusted again after each
sleep, until it gave up after 11 turns with 93. Under load every third
session of a relay or a CONNECT ended so in the middle of its data.
What was answered and not reported since is now kept per bit: an event
makes its own bits fresh, a wait answers with what it found, and those
are asked again next time.
poll() of the BSDs and macOS tells a socket we shut down for sending
as hung up, which the loop does not (see loop.c): asking it, that is
dropped, or httpsrv lingering for the client's end found it hung up
and not readable, and turned until its time was out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fiber waiting on a socket registered it with its loop and took it
off again on every wait: two epoll_ctl() calls a wait, some 39 for an
exchange of the mix benchmark, about an eighth of the proxy's time. A
socket is now registered once, edge-triggered for reading and writing,
until it is closed, and the loop keeps what it was reported ready for
until an operation finds it not to be: read, write, accept, connect,
splice() from a pipe and TLS tell it so through fiberblockfunc, the
close through fiberclosefunc. A wait for what a socket is known to be
ready for is answered at once, with no system call and no switch; only
with none of them ready does the fiber wait in its loop. Readiness is
trusted once: answered again with nothing reported or found since, what
was there may have been taken without its end being found, and the
system is asked with poll(). epoll and kqueue only; with poll() each
wait still registers its sockets.
What takes readability for an end rather than reading, the control
connection of a UDP association, the client waiting for a SOCKS BIND,
the server between requests of a kept-alive HTTP connection, asks
first with sockreadable(), a peek below any TLS layer.
kqueue: the end of either side, the peer's or our own shutdown, is
readable or writable, not a hang up, which poll() reports only with
both: kept, it made a wait for the answer after our shutdown return
at once, and sockmap give up with 93.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With -L, or loopmode on for the services which follow, the clients of a
TCP service are served on event loops: loops threads (by default one
for each CPU) each run a loop, and each client is given in turn to one
of them, which serves it on a fiber with the same code a thread runs.
Its waits in sockpoll() wait in the loop, its sockets and a timer
registered there, and the fiber goes on once one is ready or the time
is out; a sleep is a timer alone. A client costs a stack, as large as
its thread's would be, and no thread. The loops start with the first
client served on one and run as long as the process; each keeps up to
256 idle fibers for its next clients.
srvspawn() posts the client to its loop once it is counted, what is
posted made before so that posting cannot fail. A service whose clients
cannot share a thread (threadperclient), or listening for connections
back (-R), all of its clients waiting for one socket, which a loop
waits for once, keeps a thread for each client, as does every service
of a build or a processor without fibers, and the standalone services.
tests/run.py --loopmode runs every configuration with loopmode on, on
two loops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
