A listener counts its clients, whichever of its services they came to,
old or new: those connected now, raised when a client is spawned, and
those done with, raised as one leaves its service. As the listener
outlives a reload, so do the counts.
The services page of the admin interface shows them for each service as
listenerclients and listenerprocessed; services sharing a socket show
the same counts. The takeover case checks that the count goes on
through three reloads and that no client is left counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Services of one configuration on the same address, port, network
namespace and device now share one listener and its socket, all
accepting on it, instead of each binding its own: on Linux the second
bind failed unless SO_REUSEPORT was set. A listener keeps a list of the
services accepting on it; the first to start opens the socket under the
listener's open_mutex, as binding may take long, and the others use it
("Accepting connections, shared"). The socket is closed once the last of
them stops.
Such services must be of one type with the same -ol options: otherwise
the configuration is refused ("a service of another type listens
there", "a service with other socket options listens there").
mainclaim() now runs for every service, at startup too: it joins the
listener of its configuration for the address, takes over one of an
older configuration (one of its type first), or makes a new one, under
one hold of listener_mutex so that a listener found is not freed before
it is joined.
Services connecting back (-r) to one place share a pseudo-listener
without a socket, of any type. After a reload a client moves to one of
the newest services of its type on its listener with a slot free: 26 if
there is none of its type, 27 if none has a slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the -ol options which act when a socket is bound (SO_REUSEADDR,
SO_REUSEPORT, SO_EXCLUSIVEADDRUSE, SO_PORT_SCALABILITY,
SO_REUSE_UNICASTPORT, IP_TRANSPARENT, TCP_FASTOPEN) must be the same
for a socket to be taken over. The others are set on the socket taken
over, those no longer given turned off; TCP_MAXSEG, whose default is
not 0, still needs a new socket when it is dropped.
Several services may still share an address, in one configuration or
across reloads: each new one takes over a listener of its own, one of
an old service of its type first, or opens another.
A service connecting back (-r) has no socket to share, but now a
listener without one for the place it connects to: on a reload the new
service connecting to the same place takes it over, and the old
service's clients are re-authenticated as its clients, as a listening
service's are.
Nothing moves from srvparam to the listener: its socket is what accept
and UDP sessions use, and the rest is the service's configuration,
which the listener only copies to compare.
Tests: a live option changed keeps the socket, a bind-time one makes
another, two services on one address are both taken over, and a
download through a connect-back service goes on or is cut as its user
is kept or removed. 3proxy(8) describes the takeover and
re-authentication.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An FTP control connection authorizes each OPEN or USER and each data
connection while it lasts, which may be across a reload: these go
through clientauth() too, as a client of the newest service on its
listener after a reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UDP sessions check for a reload in udpsockmap() as TCP connections do
in sockmap(): unless noforce, a session goes by the newest service on
its listener, re-authenticated as its client, or is dropped.
A request on a connection that outlives a reload, an HTTP one kept
alive, is authorized by clientauth(): it moves the connection to the
newest service on its listener first if a reload came since, then
authorizes as the service whose rules apply (PSRV). The proxy uses it
for every request, so a request of a user the new configuration
removed is answered 407 rather than authorized by the old rules and
cut while relayed. reattach() takes whether to authenticate: sockmap()
and udpsockmap() move and authenticate at once, clientauth() moves and
then authorizes the request.
A test case covers the drops: with one slot free in the new service one
of two downloads moves on and the other is cut (27; a service keeps one
of its maxconn slots for the next connection it accepts), and a proxy
download is cut when a socks service takes its port (26). The reload
case checks the kept-alive request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service's socket is kept as a listener like a TCP one, the
protocol part of what must match: on a reload a new UDP service on the
same address takes it over. It is the way the old service's sessions
answer as well, and they go on answering through it; the new service
reads what comes, a datagram of an old session forwarded to it as
before, sessions being found by the service's and the client's
addresses whichever service made them. The socket stays open while the
old sessions do.
The takeover case keeps a udppm session going through three reloads
and wants every datagram back; with the build before this datagrams
were lost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a reload, unless noforce, a client of an old service goes by the
rules of the newest service on its listener, the one accepting there
now. It takes one of that service's maxconn slots, which counts it
there and keeps that service and its configuration alive until the
client is done, and is re-authenticated as its client: access lists,
users, authentication, bandwidth limits and traffic counters come from
the new configuration. A later reload moves it on, giving the slot
back. The client stays its own service's child, which keeps its
filters and TLS state.
It is dropped if its service was removed or is of another type now
(26), or if the newest service has no free slot (27); what the
re-authentication refuses ends it as before. A service without a
listener (UDP, connecting back) keeps its clients as they are.
This restores what force did before the configuration was kept per
service: since then a client compared its version with its own
service's configuration, which never changes, and was never checked
again. Services now take their configuration's version as it is: a
reload numbers the new configuration before reading it, and so does
startup now, where services were numbered one ahead.
A listener now stays with its old service until that service's last
client is gone, those clients looking the newest service up through
it; the socket itself is still closed as soon as nobody accepts on it.
Connection limits are given back in the configuration they were taken
in. PSRV() is the service whose rules apply to a client.
The reload case checks that a download of a user the new configuration
keeps finishes under it, and that one of a user it removes is cut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service's TCP listening socket is kept as a listener: when a reload
commits, before the old services are told to stop, each new service
takes over the listener of an older configuration's service on the same
address, port, network namespace and device with the same -ol options.
It skips socket() and bind(), calling only listen() again for its own
backlog, and logs "Accepting connections, taken over". The old service
stops accepting at its next poll and lets go of the socket; whoever
lets go last closes it, unlinking a Unix socket's path.
So a service staying where it was no longer leaves a gap: before, the
old one closed its socket and the new one opened another, and a
connection queued on the old socket was reset. UDP services, whose
clients read from the service's socket, and two services of the same
configuration on one address keep their own sockets as before.
A test case makes requests nonstop through three reloads and wants none
refused or reset; against the build before this, connections were
reset. The reload case no longer waits out the switch. SIGUSR1 in
3proxy(8) describes how a reload behaves now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New TLS scenarios with 2000 clients, a 2K POST and a 50K answer each.
Over 20, 10 and 5 Mbit/s links the default 4K is as good as 8K and 16K,
a little better at 5 Mbit/s: it stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Splicing after two 8K reads instead of 16K is never better in the mix
scenarios and costs 20% more CPU as an HTTP proxy on the 20 Mbit/s link:
the default stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reload reads the file into a new configuration, allocated from the
defaults, and prepares its services without starting them. Only if the
whole file is read without an error does it come into force: the old
configuration's services stop accepting, the new ones start. On an
error everything prepared is dropped and the old configuration, which
never stopped, stays in force.
Each service holds its configuration until it is freed, after its last
client; a configuration no longer in force is released when the last
holder lets go (confhold/confput). freeconf() now only releases what a
configuration holds, touching nothing global.
What runs without a service of its own reads the configuration in
force, runconf, rather than conf, which a reload points at the one
being read: the resolver, which holds it while it waits (a client
resolves with its own service's), the log, the thread pool, the admin
pages, which walk the services under config_mutex, and the signal
handlers.
The users table, havelog and the resolver choice are per configuration
now, the resolver applied when the configuration comes into force, so
a refused file leaves the users as they were. The log file, the
configuration file name, the string table, daemon mode and the filter
list belong to the process and are handed from one configuration to
the next. Settings left out of a new file go back to their defaults,
where before they kept the old values.
The gap between the old and the new listening sockets remains: a
connection queued on an old service's socket when it closes is reset.
Handing the socket over is the next phase.
A test case reloads a broken file (nothing changes, alice still gets
in, bob from the refused file does not) and a good one (bob only, the
service on its new port, the old one closed, a download through the
old service finishing whole). 40 reloads, a third refused, under load
show no memory errors under AddressSanitizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- readconfig() kept its line buffer and argument array on every error
return, so each refused reload leaked them.
- A filter refusing a connection in filter_client (TLS: a failed
handshake) was not kept, so its filter_clear never ran and the state
it made was lost. makefilters() now clears it, and empties the slot
first: a pooled worker's slot may hold the previous connection's.
- ssl_filter_close() freed the client CA paths but not the server ones,
for every TLS service set up.
- Compiled patterns were never freed: an ACL's (pcre: and regex: host
names), an httpsrv rule's. copyacl() shared them between the
configuration's ACL and each service's copy; each copy now compiles
its own, and freeacl() and freehttprules() free them.
What LeakSanitizer still reports is OpenSSL's per-thread random
generator state of pooled threads, released only by OpenSSL at exit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A section on ssl_ktls: what it needs, how 3proxy reports it, and that
it pays with splice() for TLS connections carrying large transfers.
The splice() section gets the :MIN,STREAK,SMALL tuning and says TLS
services splice where the kernel does TLS. Grace delay is recommended
by the clients' speed for steady streams only: for web traffic it
saved nothing on any of the links measured, nor with the second CPU
the HTTP proxy needed on the 5 Mbit/s one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench.py takes ktls among 3proxy's options (ssl_ktls before the TLS
services) and for nginx (nginx::ktls, nginx::halfclose+ktls: Options
KTLS on both sides).
On Intel, whose OpenSSL and nginx have kernel TLS: 3proxy splicing over
it moves bulk data 6-17% faster at 5-34% less CPU per gigabyte, with 1K
requests 0-11% slower; before splice() over kernel TLS, and for nginx,
which relays through user space either way, kernel TLS was slower.
CONNECT at 10 Mbit/s, 10 ms, 0.5% loss: -s takes about 9% off the CPU
per exchange, grace delay matched to the rate changes nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-s[N]:MIN,STREAK,SMALL sets where the automatic mode splices, as -g
does for grace delay: a direction is spliced after STREAK reads in a
row of MIN bytes or more (SMALL from a TLS socket), copied again after
more than STREAK splices in a row of less than SMALL, and a body of
known length is spliced if MIN or more of it is left. The defaults,
16384,2,4096, are what was built in; a value left out or 0 keeps its
default.
The man page describes it, and its HTML copy, which still said splice()
was not built by default, now has the current text. A test case relays
answers of every size, chunked ones and a 3 MB upload through every
mode, two of them with heuristics set, and compares them byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service with ssl_ktls keeps its splice() mode. A connection whose TLS
sockets all got kernel TLS both ways may splice, as a plain one does when
no data filters apply; one where the kernel did not take TLS (nosplice)
copies as before.
TLS may hold data for a socket that a splice() from it would not see:
OpenSSL reads more from a kernel TLS socket than it is asked for. sockmap
asks the TLS code (splicetls) and reads that through TLS into the buffer
first, once the pipe has drained, so the order is kept, then splices on.
A read from a TLS socket counts as large at 4K, TLS handing out a record
at a time, and the short read collecting what TLS left from the last one
does not end a run of large ones. A splice() refused because a record is
not data (an alert, a key update, a ticket) falls back to reading it
through TLS instead of ending the connection.
The kernel TLS test compares the bodies with the origin's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open Watcom has no struct _stat64 or _wstat64: its 64-bit stat is
struct _stati64 and _wstati64, so httpsrv.c did not compile there and
the 1.0.1 lite build had no binaries.
widepath() compared and wrote '\\\\', a two-character constant, for a
backslash: a UNC path (\\server\share) never took the \\?\UNC\ form,
and forward slashes became backslashes only because the constant was
truncated to its last byte.
The Watcom workflow passed although nmake failed; it now fails when
the build does or leaves no 3proxy.exe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9651848bf1)
ssl_ktls asks OpenSSL to hand TLS to the kernel (SSL_OP_ENABLE_KTLS)
on the connections of the services which follow, both to clients and
to servers; ssl_noktls, the default, keeps it in 3proxy. Where the
library, the kernel and the cipher allow it, the kernel then encrypts
and decrypts, and SSL_read/SSL_write go on working over it. What a
service got is logged on its first TLS connection.
Reading ahead is left off where the kernel took receiving, which reads
a record at a time. A build whose library has no kernel TLS (wolfSSL,
OpenSSL before 3.0 or without it) warns and ignores the command.
splice() is not used for TLS services yet.
Tested on AlmaLinux 10 (OpenSSL 3.5.8, tls module): both directions
of both legs of a TLS chain in the kernel, the suite passing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 5 Mbit/s, 25 ms, 1% loss run again with a second proxy CPU, so
the HTTP proxy is not held by its CPU: it then keeps 3000 exchanges a
second in every configuration, and the grace delay settings rank as
with one CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grace delay is described as it works: NUM small reads in a row in one
direction within a second start the wait, larger reads end it. It
saves CPU on connections carrying data in small pieces and caps a
connection at one read per DELAY, 16K copying and 64K (the pipe) with
splice(); a table gives the caps for 1-10 ms, another the measured
gains per client link, and DELAY is to be matched to the clients'
speed. It does not help once the CPU is exhausted. The example becomes
-g8000,2,2: the old -g8000,3,10 capped copying at 13 Mbit/s.
splice() gets measured figures: 3 times the speed at 60-70% less CPU
for large transfers on a fast network, within 0-12% for web traffic
in the mix over emulated links. -s, the automatic mode, is the
default; the paragraph still said -s0 was. splice() is not used where
3proxy has to see the data: while data filters apply, and by services
ending or beginning TLS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Results on both machines over per-connection emulated links: seven
grace delay settings with a fifth client CPU; the promising ones over
three link qualities (100 Mbit/s 2 ms 0.1%, 20 Mbit/s 5 ms 0.5%,
5 Mbit/s 25 ms 1%); copying against splice() with and without grace
delay, the CONNECT mix included; and the slow link with waits matched
to its rate (10 and 20 ms).
Grace delay saves 35-46% of 3proxy's CPU per gigabyte on streams at
20 Mbit/s and 21-29% at 5 Mbit/s with a 10 ms wait, does nothing on
fast links, and costs CPU with waits too short for the rate or once
the CPU is exhausted. Over these links splice() and copying cost about
the same in the mix scenarios. A caveat says that every scenario but
the mix ones is synthetic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grace delay section, measured on plain veth links, is replaced by
one over an emulated client link: 20 Mbit/s per connection, 5 ms and
0.5% loss each way, devel with four -g settings and nginx, streams
through a relay and the fixed-rate mix scenarios. The README says where
the mix profile's shape comes from and that its parameters are chosen.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wrk's and mix's 90th percentile is recorded (p90_ms) and shown in the
report.py --latency tables when present.
--flow-rate limits each connection between the client and the proxy
(fq maxrate), under --netem when both are given, now with a large netem
queue. Both directions are shaped in the client's namespace, the
incoming one through an ifb, so the emulation runs on the client's CPUs;
segmentation offload and GRO are off on that link so that netem drops
single packets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
report.py --latency makes a table of the runs' 50th and 99th percentile
latency for one CSV, ranked by the 99th. RESULTS gets one for each
machine in the HTTP, CONNECT, SOCKS, relay and mix sections, with how to
read them: wrk runs a closed loop, so medians follow throughput and the
99th percentile against the median shows how evenly a proxy serves its
connections; the mix -3k runs are the open-loop ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP, CONNECT, SOCKS, TCP relay and mix tables rebuilt from the full run
on both machines with devel at 98fd0f3 (half-close fix, idle buffer
release, TLS changes), 1.0.1 with TCP_NODELAY, nginx relays with
proxy_half_close on, and envoy added. The morning session moves to the
earlier sessions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service keeps the configuration it was started with (srvparam.config)
and its threads and its clients' read it there: limits and counters
(alwaysauth, the limiter), RADIUS, dnspr's name servers, and the checks
for a reload in the accept loop and in sockmap(). The configuration is
still reloaded in place, so it is the current one for now; this is what
lets a reload build a new one while the old one still serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf is now a pointer to the current configuration rather than the
structure itself: at first confboot, which exists before main() runs, as
plugins and the standalone services use it too; later configurations
will start from confdefaults. Both are made from confinit.h, the one list
of initial values. The timeouts are an array of the configuration
instead of a pointer to a global one. Nothing behaves differently yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What a numbered counter counted now belongs to the process: a counter
rule points to the value of its number, created the first time the
number is met (from the counter file, if one is open) and shared by the
rules of that number in every configuration loaded. A reload no longer
resets it when there is no counter file, and once configurations can
outlive one another, clients of the old and the new one count into the
same value. An unnumbered counter, and those LdapPlugin makes, keep a
value of their own as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
conf.haveerror was only initialised and srvparam.srvfds only declared;
the extern declarations of stdlog and demon named globals that do not
exist (both are fields of conf); a local in make_ace() and one in the
webadmin XML page were never used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
They are the state of the process, not configuration: conf.time (now
curtime), timetoexit and needreload are globals of their own. Plugins
reach needreload through a pointer added at the end of struct
pluginlink; LdapPlugin uses it. The extern for a paused global, never
defined, goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nservers, numservers and authnserver (resolve.c, common.c) and
radiuslist, nradservers and radiussecret (authradius.c) were globals of
their own; they are configuration, set by nserver, authnserver and
radius, and now live in struct extparam with the rest of it.
MAXNSERVERS, MAXRADIUS and struct radserver move to structures.h, which
plugins include without proxy.h.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dialer ran a program when a name could not be resolved, for dial-up
links; with nscache it did not even run. The command is still accepted
and ignored, so configurations using it keep loading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like the timeouts, each service now keeps the values given before it
instead of reading the global ones while its connections run. setopts()
takes the service's maxseg. The TLS handshakes wait the service's
CONNECT_TO, kept in its SSL configuration, rather than the global one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssl_filter_open() copies ssl_client_mode into the service's
configuration, but the filters deciding when to begin TLS towards the
server read the global value instead: the last ssl_client_mode in the
configuration applied to every service, and a reload changed it for the
services still running.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A socket which hung up and of which nothing is asked was given to poll()
as INVALID_SOCKET, so that poll() would not return at once for the
hang-up every time. poll() ignores a negative descriptor, but SOCKET is
unsigned on Windows, and WSAPoll fails the whole call for it: sockmap()
ended the session as soon as a client half-closed. Such a socket is now
not in the array at all.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every table is now for one machine, ARM or Intel, with no averages of
the two, the copying proxies and those using splice() in one table:
bold the best of each group and * the best of the row for the HTTP proxy
and the relay, only the best of the row for CONNECT and SOCKS. The
tables for 13 October replace those of 11-12 October, whose columns for
splice by direction and 16 KB buffers are gone with them.
The HTTPS section is measured again with today's devel, with the TLS
streams and envoy. report.py takes the TLS streams and 1 MB answers for
bulk data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit lets a half-closed client be written to on Windows
too, so the case left out there is back.
This reverts commit 5d1cdf8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() took a hang-up on a socket it had data to write to for a peer
it could no longer write to. With poll() a hang-up comes once both sides
are shut down, so that held. WSAPoll reports one as soon as the peer has
half-closed, still reading, and a half-closed client got nothing on
Windows: the test added with the previous fix failed there.
The write is now tried, and a peer really gone makes it fail, which
ends the connection as before. WSAPoll keeps reporting the hang-up
whether the socket has room or not, so on Windows room is waited for
with select() first unless poll() reported it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WSAPoll reports a hang-up as soon as the peer has half-closed, rather
than once both sides are shut down as poll() does elsewhere, and
sockmap() takes a hang-up on a socket it is writing to for a peer it
can no longer write to: a half-closed client gets nothing there. That
is a separate problem, Windows only; the case added with the fix for
the other platforms is skipped there until it is dealt with.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-stream* and tls-init-stream* run iperf3 streams through the TLS
relays. iperf3 has no TLS of its own, so bench/mix gets a TLS wrapper
mode, run on the client to begin TLS towards a proxy ending it and at the
origin to end the TLS a proxy begins.
--proxy envoy measures envoy where nginx and haproxy are measured: as an
HTTP proxy for the origin, as a TCP relay and in the TLS scenarios, its
circuit breakers lifted for the scenarios opening more than 1024
connections.
report.py --machine with a single machine gives its medians alone, with
the client- or origin-limited ones marked; --overall marks only the best
of each row. The mix client's -errlog prints every failed exchange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxychild() allocated its 64 KB buffer for headers once and kept it
for the life of the connection: for a CONNECT tunnel all the time data
was relayed, and for a kept-alive connection between its requests.
With 2000 tunnels that was 128 MB, with 20000 mostly idle connections
over a gigabyte.
The buffer is now freed before a tunnel starts relaying, and when a
kept-alive connection has waited a second for its next request; it is
allocated again once the next request starts coming. logurl() takes a
copy of its own for a request it logs after its buffer was freed.
2000 tunnels: 339 to 213 MB; 20000 connections, 3000 exchanges a
second: 2.65 to 1.63 GB through the HTTP proxy, 2.66 to 1.42 GB through
CONNECT; busy kept-alive connections within 1.5% of before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read-ahead was off, so OpenSSL read every record in two calls, its
header and then its body. It was off because ssl_poll() asks
ssl_pending(), and SSL_pending() does not count data read ahead and not
yet decrypted: poll() would wait while data waits in OpenSSL.
SSL_has_pending() does count it. With OpenSSL 1.1.0 and later, which
have it, ssl_pending() uses it and records are read ahead; with older
ones and wolfSSL nothing changes.
Decrypting TLS streams takes 9-14% less time for the same data (ARM).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixed buffers hold 16384 + 64 bytes, a whole TLS record and its
header, which tlspr needs. sockmap() read all of that from one side and
wrote it to the other, so a side using TLS was written a full record and
another of 64 bytes for every full buffer: twice the records, twice the
writes. Data relayed into the fixed buffers is now read 16384 bytes at
most; a buffer already holding more, as tlspr can leave it, or a larger
one taken for a filter, is used whole.
Ending TLS for 1 MB answers this takes 25% more throughput from the
same CPU (Intel, 4.4 to 5.5 Gbit/s); plain relaying is not affected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the client shuts down its sending side, sockmap() shuts down the
server's in turn, and once the server has answered and closed, poll()
reports a hang-up on its socket. A socket sockmap() had nothing to ask
of at that moment, its buffer being full for a slow client, was still
polled, and a hang-up there ended both directions: whatever of the
answer was still unread in the kernel was dropped and the client got an
end of stream in the middle of it, megabytes short with large answers.
A hang-up on such a socket now only marks it: what is left is read
once the buffer has room, up to the end of the stream as usual. Errors
still end the connection. Since poll() reports a hang-up whatever is
asked, the socket is left out of poll() while nothing is asked of it.
The same holds for the client's side, for an upload cut short after the
server half-closes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tls-term-* and tls-init-* measure TLS ended or begun by the proxy on a
TCP relay: 3proxy tcppm under ssl_serv/ssl_cli, nginx stream, haproxy
mode tcp and stunnel, with a certificate made for each run and an HTTPS
listener on the origin. nginx::halfclose turns on proxy_half_close.
bench/mix is the Go load generator and origin for the mix-* scenarios.
A run stops when the origin exits (e.g. killed by the OOM killer)
instead of recording failures for everything after it.
RESULTS.md gets the HTTPS and mix tables; data/ the full suite with 16
KB buffers and -s by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside Linux nssocket() was a macro, and two of its callers have #ifdef
WITH_UN among its arguments. Preprocessor directives in macro arguments
are undefined behaviour: gcc and clang take them, MSVC does not. It is a
function on every system now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A kept-alive client connection reuses the connection to the server while
requests go to the same host. The check compared the start of the request
line up to the end of the host, method included, so a POST after a GET to
the same host, or a GET after a POST, closed the server connection and
opened another one. Only the host and port are compared now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
