The part after the @ of a port in a rule names the service: by its
port, by the address it listens on, or by both, the address in
brackets when IPv6 —
connlim 5 60 * * * @1080
connlim 5 60 * * * @10.0.0.1:1080
deny * * * @[::1]
A slash, as the delimiter of the first take, would have read as a
CIDR habit the day a service is named inside an address list, and
@ cannot appear in a port or an address. Digits alone are a port, a
dashed pair a port range, anything else an address with a port behind
a colon; an @ with nothing after it is refused. The destination port
before the @ is unchanged, and so is every rule without one.
A port in a rule's targetportlist may carry a service port after a
slash, X-Y/S, or, for any destination, */S: the rule matches only
requests that arrived through the service listening on the port S,
the port of its proxy -p or socks -p, whatever their destination. The
limiters are shared by all the services of a configuration, so this is
how a limit is given to the connections of one service alone:
connlim 5 60 * * * */1080
denies the sixth concurrent connection of the service on port 1080.
The destination port and the service port are both ranges or single
ports, a slash with nothing after it is refused, and a rule without a
slash is what it always was.
mark <number> extends the allow or deny rule above it, as nolog and
weight do: a connection the rule lets through carries the mark on its
outgoing sockets, towards the destination and towards the parents, so
policy routing and firewall rules can tell its packets apart (SO_MARK
of Linux, SO_USER_COOKIE of FreeBSD; on a system with no mark the
command sets it to nothing). The UDP sockets of a session towards a
destination are marked the same, the SOCKS UDP ASSOCIATE relays among
them, for the rule that let the session through is looked up for each
destination. A mark with no rule above it, or not a number, is refused.
Once reserved, a port is dropped when a configuration comes in force
that neither reserves it anew nor has a service on it: after chroot no
reservation can be made again, only the usage of the socket a service
takes over keeps it. The reservation of the configuration before no
longer waits for its last client to let it go: the reload releases it
at once, the socket living on where a service of the new one has
claimed it, closing where none has.
port tcp|udp [ip:]port [socket_options] opens, binds and listens at
the time the line is read, so a privileged port can be taken before
setuid and chroot drop the rights to open it: a service of the
configuration on the same address, of the same type, takes the socket
over when it starts, instead of binding one of its own, and a port no
service takes stays reserved until the configuration is left. A
reservation of the configuration before holds the port over a reload:
the new one shares its socket, each dropping its hold with its
configuration.
The socket is what the reservation made of it: the options which act
only before bind() cannot follow a service's wishes on a socket already
bound, so the service takes it as it is, its other options set on it.
The reservation is built from the same default address a service is, so
it matches in every byte a listener is compared by.
The port was promised by the man page but never parsed: a server
address with a port was refused, and only the default 1812 could be
used. The address is parsed with parsehost() now, as nserver and
authnserver are, so a name is resolved when the configuration is read,
an IPv6 address may be written in brackets, and the port is the one
authentication is asked on, the accounting of the same server going to
the port after it. The man page says which name servers resolve the
name.
An action word of a pcre or pcre_rewrite rule may end with ,log: every
match writes a record of its own carrying the matched part of the
buffer, so a header can be watched without changing the request's fate
(dunno,log), or a refusal can leave its trace (deny,log). The matched
text reaches the log through %T, like any service text. The pcre
options apply to such a pattern as to any other, so it can be written
case-insensitively and by the line.
Every request to a RADIUS server carries an Event-Timestamp (RFC 2869,
attribute 55), the time it was made in seconds since the epoch, so a
server no longer has to fish the time out of Acct-Session-Id.
A reply with no answers and an error code - FORMERR, SERVFAIL, NOTIMP,
REFUSED - ended the resolution as if the name had no address, and the
servers after the one that answered were never asked, so one broken
server made every name it broke fail. Such an answer is passed over for
the next server now; a reply without data, and one that says the name is
not there, still end the search.
Datagrams on a TCP connection, each with a two byte length in network
order in front of it, which is how udptunnel (the rfc1036 one) and
ProxyGuard (eduVPN) put them there, the framing alone or after a
handshake: udptunnel's 32 bytes, or ProxyGuard's HTTP Upgrade for
UoTLV/1 answered with 101 Switching Protocols.
tcptoudp takes the connections, a UDP socket of its own to the target
for each, the handshake a client speaks recognised by its first bytes
and answered; a parent the ACL gives is a SOCKS5 UDP relay, as for
udppm. udptotcp takes the datagrams, each client a session of the
service's with a TCP connection of its own, connected as the service's
loop sees it writable (dnspr asks a name server over TCP the same way);
-m sends udptunnel's handshake to the target, -m2 ProxyGuard's, -m0
(the default) none, so both ends of either tool are stood in for.
The service names take two places of the string table, the default
configuration path moving two later (stringtable[27]).
Built by the Makefiles and CMake, with the standalone binaries, and
installed by them; a case of the test suite speaks all three handshakes.
Authorization reports 10 when a limit of the service was exceeded,
which every other child answers with 503; the http server took it as
success and answered the request anyway. It now says 503 for itself,
after the body of the request is drained as any answer here does.
(cherry picked from commit 5a2e9d27c3)
A path ending in a backslashed dot-dot, or carrying one in a form the
checks did not name, was taken as it stood. Any dot-dot segment is now
refused wherever it is and whichever separator carries it, and a path
ending in a bare dot of either kind.
(cherry picked from commit c0eb7a65c6)
A rule whose ACL was missing was handed to the filters as the
connection's data, which they read as a match data block and freed:
a pointer into the configuration, dereferenced as another type and
given back to malloc. A rule without an ACL now skips the check and
goes on with a block of its own, and nothing a filter is given is
ever the rule itself.
(cherry picked from commit a7e5a8b48b)
Every connection matching a pcre rule matched on the one match data
block of the rule, which pcre2_match() writes into: two connections
matching at the same time wrote the same block, a race that corrupts
memory and brings the process down after a while under load. A
connection that passes the rule's ACL now makes a match data block of
its own when it starts and frees it when it ends; the match context is
kept, as it is only read while matching.
ssize_t does not exist under MSVC, and the socket function table
already spells its send and recv results int there and ssize_t
elsewhere. That type is now named SOCKRET, beside SASIZETYPE, and the
event loop's copies of it use the name, so the MSVC build compiles
again.
3proxy --help (and the usage message of a bad invocation) now lists
what the build was made with: TLS and its backend, RADIUS, PCRE, IPv6,
UNIX sockets, the built-in HTTP server, transparency, ODBC and the
optional proxy services. The test runner reads the list before the
first case and the cases state the features they need with
t.skip_unless(), so a build without a feature skips its cases instead
of failing them; a binary too old to report anything skips the cases
with needs and runs the rest.
The reply digest copied the shared secret right after the received
bytes, so a reply of 4100 bytes with a secret of 17 or more characters
wrote up to 47 bytes of it past the reply buffer on the stack, before
the reply was authenticated. The digest is now taken over the reply
and the secret in place, with nothing written past the reply, and it
fails closed when the hash cannot be made. A reply is also taken only
from the address and port of the server the request went to, and only
when its identifier and length field match the request, all before any
digest work; the reply's sender was never checked before.
The label walk of a DNS PTR answer ran to the end of the whole message,
not to the end of the record's data, and took a compression pointer for
a label length, so it could write past the end of the answer buffer on
the stack. An answer of 3841 bytes or more reached past the buffer
itself. A record that ended exactly at the end of the message was also
read one byte past it. The walk now stops at the end of the record's
data, a name that does not end there, or one that is compressed or
empty is not accepted, and the answer loop reads nothing past the
message.
3proxy 0.9.9.2, 1.0.2, devel (c3702866) and the experimental head (5de7ae2)
against nginx (with and without proxy_half_close), haproxy (plain and
splice-auto), envoy, squid, tinyproxy and dante on the M4 and the Skylake
host: the plain suite and the mix over a shaped link (20 Mbit/s, 5 ms,
0.5% loss), three runs a cell, proxy memory capped at 6 GB (--prx-mem-mb,
new). dante re-measured on the SOCKS scenarios only; its mix behaviour
stays in its own smaller-pool table. RESULTS.md rebuilt around six uniform
tables: throughput, memory and errors, p75/p90/p99.
A connection's buffers came from malloc(), which keeps what it is given
back: a buffer let go while the connection waits (HTTP keep-alive, an
idle relay) left its memory with the process. They now come from a
pool of memory mapped for them, in chunks aligned to their size, the
first unit of a chunk its header. A unit is a page, or a buffer if pages
are smaller: with pages of 4 or 16 KB a buffer is a unit of its own,
with pages of 64 KB a unit holds four. Free units are kept for the next
buffer, the most recently freed first; more than 64 and an eighth of
those in use, and the least recently freed are given back to the system
(MADV_DONTNEED on Linux, MADV_FREE_REUSABLE on macOS, MADV_FREE
elsewhere, MEM_RESET on Windows), and a chunk given back whole is
unmapped. Built for AddressSanitizer the buffers stay malloc()'s, which
it checks for overruns, unless WITHBUFPOOL is defined; the pool then
poisons the free ones.
Buffers are 16384 bytes, not 16384 + 64: a size of pages. The 64 bytes
were room for a TLS record's header, which tlspr needs to have a record
whole; tlspr now takes a larger buffer when a record does not fit, up to
TLS's largest (5 + 16384 + 2048), given back once it is emptied, and
relays read the whole buffer again, a record at most as before. A record
larger than the buffer set with -b is no longer refused.
HTTP proxy, 2000 idle keep-alive connections, after a request (Linux
container): 39.6 KB a connection before, 19.5 KB now; a thread a
connection, 44.2 KB before, 24.1 KB now; a TCP relay, 13.0 KB before,
11.6 KB now.
Incompatible for plugins: CLIENTBUFSIZE is 16384; clibufdata and
srvbufdata are the pool's, never to be passed to free() or realloc().
bufpoolinit() is called before any service starts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
-b, which made a service read less at once into a connection's buffer,
takes the size of its pipes after a comma: -b,262144, -b8192,32768.
From 4096 to 64M (MAXPIPESIZE); a pipe the system will not make so
large (pipe-max-size for an unprivileged process) is not used, and the
data is copied, as with -DPIPESIZE, which stays the default for a
service given none. sockmap() and the callbacks' relay fill a pipe up
to the service's size (SRVPIPESIZE()), and only a pipe of other than
64K costs an fcntl() as it is made.
The man page tells of -b, which it did not before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NTLM authentication of proxy clients was gone but for what was left of
it: the 407 answer offering NTLM before Basic (12, under NOCRYPT, which
nothing else used) and the one for its handshake (13), sent by nothing,
and the prototypes of genchallenge() and mschap(), defined nowhere. The
strings after them in proxy_stringtable move up two: 403 with <pre> is
12, 503 13, 401 of the FTP server 14, 100 Continue 15.
rus.3ps, StringsPlugin's translation, follows: the two strings go, the
403 with <pre> is UTF-8 as the rest, the XML of the running services
has its Content-disposition, and the table of counters the columns of
its rows (Action and #/Description were one).
NT: passwords, MD4 hashes as Windows keeps them, stay. The documents
no longer tell to give proxy -n to turn off NTLM.
Plugin binary compatibility: a plugin indexing proxy_stringtable finds
the strings from 12 on two places earlier, and a .3ps file of its own
with the NTLM strings has two more than 3proxy (StringsPlugin warns).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
load_string() put a NUL where a line read from the file had its CR or
LF, and stepped over it as well as over the character after: past the
line's NUL terminator, into what was left in the buffer from longer
lines before, or past its end, writing NULs wherever it found CR or LF.
The line is cut at its first CR or LF instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() under WITHLOG, and the SOCKS proxy under SOCKSTRACE, wrote
what they did to the log or to stderr, a build for debugging asking for
it: some 200 lines in the middle of the code, never built otherwise.
They go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Names are resolved with getaddrinfo(), thread-safe everywhere: what was
left of gethostbyname() (my_gethostbyname() over gethostbyname_r(), in
its glibc and Solaris forms, GETHOSTBYNAME_R, the mutex for the plain
one) served only the IPv4-only resolver already gone. Makefile.Solaris
(Sun's compiler, _SOLARIS) and Makefile.unix go: CMake builds for the
other systems; the documents say so, and their examples use
Makefile.FreeBSD.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows CE (_WINCE: WinMain() and its window, the command line made
into argv, the paths and functions it went without) and Windows 9x
(--install and --remove through the RunServices key of the registry,
which is what GetVersionEx() was asked for) are no longer built for.
--install and --remove use the Service Manager alone; Winsock 2.2 is
asked for, which WSAPoll() needs, rather than 1.1.
The service command of a configuration, deprecated, still does
nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Win32 "lite" build with Open Watcom (Makefile.watcom, its release
workflow, the CMake branch for it) was the only one built without IPv6
(NOIPV6), without fibers and event loops, with a select() of its own
for poll() (mypoll()), and with the calling conventions of Watcom's
pragmas in the plugins. They are gone with it: every build has IPv6,
and polls with poll() on Unix and WSAPoll() on Windows, which were
what every other build asked for (WITH_POLL, WITH_WSAPOLL, the
3PROXY_USE_POLL and 3PROXY_USE_WSAPOLL options of CMake, FD_SETSIZE
for select()). getip(), the IPv4-only resolver of NOIPV6, goes too.
Plugin binary compatibility: plugins built with Open Watcom, and their
calling convention, are no longer supported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tcpnext() and srvspawn() slept a little after accept() failed for want
of descriptors or memory, and after a client's memory could not be had,
with usleep(): accepting on a fiber of a loop, that held up every client
of the loop. srvsleep() waits in the loop there (sockpoll() with no
sockets), as tcploop() already did waiting for a slot, and sleeps as
before in a thread.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A service whose clients are served on loops took them in a thread of
its own, woken by the system for each and handing it to a loop: some
5000 switches of threads in 5 seconds under the mix-http benchmark.
Its tcploop() now runs on a fiber of the first loop (loopcall(),
tcploopcall()), its thread waiting for it to return, so connections
are accepted where they are polled for: some 40 switches.
On a loop, a connection taken, the next one is tried at once, until
accept() finds none (EAGAIN, now NEXT_RETRY without a log line), 64 at
most before the others of the loop are let go first (fiberlater(),
which resumes the fiber in the loop's next turn: a poll of a socket
known to be ready does not yield). Waiting for a slot of maxchild does
not block the loop: the slot is tried again every 100 ms.
The listening socket is polled through a dup() of its own: a loop knows
a socket by its number, and on a reload the socket handed over is
polled by the service taking it over too, whose wait would replace this
one's. Not on Windows, where a socket has no dup(), nor with -r.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
loopmode is cb unless a configuration says otherwise, as -L2 had it
for a service: its clients on event loops, a connection's data relayed
by the loop itself where nothing but the sockets is between its sides.
A service whose clients cannot share a thread (threadperclient: PAM,
LDAP, FilePlugin, names resolved by the system, -ne) or listening for
connections back (-R) still gives each a thread, as do the standalone
services and builds without fibers. loopmode on and -L keep the loops
with the relay on the client's stack; loopmode off and -L0 a thread
for each client.
The man page tells of cb. tests/run.py --loopmode takes the mode
(cb, on, off) to give every configuration, with two loops; the Linux
workflow runs the suite once more with a thread for each client, which
is no longer what a configuration gets by default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() and the callbacks' relay each had their own 64K for what a
pipe is given at most (MAXSPLICE, CBSPLICE). It is PIPESIZE now, one
for both, 64K as before: what a pipe holds unless asked otherwise.
Built with -DPIPESIZE=<bytes>, pipes are made that large with
F_SETPIPE_SZ (pipenew()); one the system will not make so large is not
used, and the data is copied, as when there is no pipe to be had.
Measured on ARM and Intel (mix-http, mix-relay), 16K and 32K were
slower, 128K and 256K no faster than 64K, which stays the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fiber's stack was a thread's, PTHREAD_STACK_MIN included: 128K on
arm64 glibc, which is there for a thread's signals, and a loop's thread
takes none. It is BASESTACKSIZE with the service's stacksize added, and
the room kept for wolfSSL's handshake (TLSSTACKSIZE), as a thread's but
for that. Only reserved either way: measured with OpenSSL, a fiber holds
8K to 12K of it, TLS handshakes included.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
glibc's getaddrinfo() sizes what it takes with alloca(), IPv6 most, by
the thread's stack, not by the fiber's it may be called on: a client
moved to a new configuration on a reload goes on on its fiber, and the
new configuration's services may resolve with the system resolver,
which otherwise is never called on a loop (threadperclient). A fiber
yields to have it called on the loop thread's own stack
(onthreadstack()), and goes on with its answer; the loop waits for it
as the fiber would have.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() made both buffers again as it returned, for whatever came
after it, and the callbacks before the end: after the last mapping there
is only the end, the log line and a refusal sent from a buffer of its
own, with no use for them. They are made again only for a mapping that
is not the last, and for a fiber mapping on after a reload; the
functions reading a side's buffer make it if it is not there, and a
worker taken from the pool for its next connection gets its own again.
proxy.c's other last mappings, a CONNECT through a parent and an answer
ending as the server closes, are marked so too (maplast(), proxytail),
as the CONNECT of its own was: their end, and with callbacks the fiber,
is the loop's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fiber's wait asks the system again what each socket was reported ready
for and answered with once already: one poll() for each of them, two for
an HTTP connection kept alive waiting on both of its sides. They are
asked all at once now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forwarded and Via asked gethostname(), a uname() syscall, for each
request. The name is kept, asked again after a reload.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An answer's headers were sent by themselves, and what hdrread() had read
of the body after them by sockmap() next: two segments where one does,
for most small answers. The headers, still where hdrread() put them in
the server's buffer, move up against the body, and go with what of it
was read in one send; an answer whose whole body came with its headers
is not mapped at all. Not for chunked answers, answers with no body, or
where a data filter is to see the body; the bytes were counted as they
were read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A connection kept alive waited for its next request holding both of its
buffers, empty: most of the memory of an HTTP proxy under load. They are
let go before the wait and made again as something comes, but for a
request httpsrv keeps there to hand over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sockmap() of fibers and threads kept both buffers for as long as it
mapped, even with a direction spliced. On a connection's last mapping
(maptail set), as the callbacks already do, a direction spliced lets its
empty buffer go: bufdrop(), shared with the callbacks now. It is made
again before a read into it, back to copying or draining what TLS held,
and both are there again once sockmap() returns, for whatever comes
after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UDP service's thread waited for its sockets with an epoll set of its
own on Linux and poll() over every socket on each turn elsewhere. It
runs a loop of loop.c now: epoll, kqueue or AFD, the sockets registered
once and changed as the sessions change, everywhere. The handlers are
called by the loop as a socket is ready, a datagram read before the
socket is polled again; the sessions ended meanwhile are let go after
the turn, as before. The deadlines keep their heap.
A session is allocated with what the loop waits for, private to
mainudp.c. The standalone udppm is linked with loop.c.
Plugins: struct udpsession loses regctrl and regevents.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A logger whose thread could not be started is written by its clients
themselves. A client of a loop, a fiber or a callback on the loop's
thread, would hold the loop while a database or a server takes its
line, so on a loop's thread the line is dropped instead, counted as the
log's lines dropped are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A logger's lines are written by a thread of its own, its clients only
queueing them, so a logger whose writes wait cannot hold a client's
thread or a loop: its blocks flag, giving its services a thread for each
client, is gone. ODBC was the logger setting it; its services are served
on loops now too. Only a logger whose thread could not be started is
written by its clients themselves, as before.
Plugins: struct logtype loses its last member, blocks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>