bench.py:
- the proxy gets one CPU, the client four, the origin three (8 CPUs), so
the proxy is what runs out first
- CPU time is the busy time of a role's CPUs from /proc/stat, softirqs
included: with IRQ time accounting the cgroup's own figure leaves out
much of the network work
- every run records which side set its pace (proxy, client, origin), by
the busy share of the role's CPUs and of its busiest thread
- a 50 KB keep-alive scenario with 2000 clients
- haproxy with splice: haproxy::splice-auto and haproxy::splice
- wrk timeouts counted apart from errors; tools found by absolute path;
leftovers of an interrupted run are cleaned up
report.py turns runs into Markdown tables: throughput, the proxy's CPU
per gigabyte or per request, its peak memory, splice and copying proxies
in groups of their own, the best of each group marked.
RESULTS.md has the comparison of 0.9.4, 1.0.1, devel, nginx and haproxy,
with and without splice, on an Apple M4 and an Intel Skylake virtual
machine, and how it was measured; data/ has the measurements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench/build.sh builds git refs or the working tree, each with the
Makefile it shipped with. bench/bench.py runs the client, the proxy and
the origin in network namespaces of their own, linked by veth pairs,
and in cgroups pinned to separate CPUs, then measures TCP relaying
(iperf3) and HTTP proxying (wrk) for 3proxy builds, nginx, haproxy and
the kernel forwarding alone, reporting throughput, latency and the
proxy's CPU time and peak memory per unit of work.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reload asked for through /R happens on the next tick of the main
loop, and between the old services stopping and the new ones starting a
request is refused: on Windows the new listener cannot bind the port
until the old one has closed it, and retries with a growing delay. When
the tick fell inside the case, every request after it failed with
ConnectionRefusedError (MSVC run of 98e8dfb). The case now waits for
both services to have restarted after each /R.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On receive, the struct sockaddr is zeroed and does not have an address
family until recvfrom fills it in. SASIZE assumes that if there is no
address family, it is INET. This means that for IPv6 UDP sockets, the
sockaddr is truncated and the reply is forwarded to the wrong address.
Instead, use sizeof(PROXYSOCKADDRTYPE).
extport binds a local port out of its range and connects from it. A port
which is free to bind may still be half of a pair which is closing from an
earlier connection to the same destination, and Windows fails the connect
with WSAEADDRINUSE where other systems fail the bind and fall back. The
request then failed with error 13 for a reason another port would have
avoided, which is what the range is searched for in the first place.
Retry the connect with another port from the range. Only for that error, so
a destination which is simply unreachable still costs one attempt, and out of
the same ten tries the bind already spends, so the attempts parentretries
makes are not multiplied. EADDRINUSE joins the errno values proxy.h maps to
their Winsock numbers, without which the test would never match on Windows.
parent_ports gave its three services one window between them, which is how
the case met the collision on the Windows runner. Each has its own now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nserver and authnserver parse their argument with parsehost(), which reads
the first colon as the address/port separator, so a bare IPv6 address picks
up a bogus port number. Document the bracket form in 3proxy.cfg(5) and in
both HOWTOs.
Refs #1275, #460
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Package maintainers, Gentoo among them, build from source and had nothing
signed to verify against: the source archive GitHub generates for a tag is
neither signed nor guaranteed to stay byte-identical.
Add release-tarball.yml. It builds 3proxy-<version>.tar.gz with git archive
from the release tag, signs the tarball and SHA256SUMS-src with the release
key, attests build provenance and uploads all four files to the release.
Also drop the SECURITY.md line about Authenticode signing, left over from the
removal of self-signed Windows binaries.
Closes#1270
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signing the release binaries with a self-signed certificate gives no
trust benefit: Windows does not recognize the issuer, so SmartScreen
warns anyway, and the signature is a frequent trigger for antivirus
false positives.
Drop the Decode Certificate / Extract public certificate / Sign steps
from the MSVC and Watcom workflows, stop shipping the extracted
3proxy.crt in the distribution, and remove the certificate URL from the
build version string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pop3p, imapp and smtpp are built with MAILPROXY=true, and ftppr and the
ftp:// scheme of the HTTP proxy with FTP=true; with CMake the switches are
3PROXY_USE_MAILPROXY and 3PROXY_USE_FTP. Neither is in a default build, and
the standalone binaries follow what was built.
A configuration naming one of them is still read either way. The three mail
protocols amount to a STARTTLS negotiation now that mail is carried over
TLS, so without a proxy of their own those names are tlspr speaking the
protocol: the file holds a stand-in which sets the protocol for the
connection and returns tlspr for the caller to run, which serves the
service name and a parent chain alike and leaves conf.c and the redirect
table untouched. FTP has no such fallback, so the service is known, answers
nothing and logs the refusal.
The Linux workflow builds both, so all of it is still compiled and run.
bindwithrange() honours a range by setting IP_LOCAL_PORT_RANGE where it
exists, which leaves the kernel to pick a port inside it and skip the ones
still closing. Everywhere else it binds a port out of the range at random,
ten times, and lets the system choose when all ten are taken.
A port which carried a connection cannot be bound again until that close
completes - four minutes of it on Windows - so a window of fifty ports
shared by three services in one configuration can have too few left, and
the connection then binds outside the range and the case fails. The window
is five hundred ports there instead. The Linux one is unchanged: the
kernel picks from it and needs no room to spare.
op_rewrite checked what it built with targetunsafe(), which describes a
path on this machine: on Windows it must name a drive or a share, so a
rewritten request path was refused and every rewrite rule failed there.
A rewrite produces a request path and is checked as one.
The header loop stopped at HTTPSRV_MAXHDR and answered anyway, leaving
the rest of the request in the stream for the next one to be read out
of. It refuses the request instead. A header longer than the buffer
arrives as several lines, so the count bounds what is read rather than
what a client may send in one header.
The test for the buffer growth after a PCRE rewrite sent its request
through to an httpsrv origin, which stops reading at that same cap; it
uses an origin which reads whatever it is sent, since what is under test
is the proxy in the middle.
41 of 112 configuration commands appear in a test. Record the rest as a
plan, ordered by how much of the product each gap covers against how much
of a fixture it needs, so the next person does not have to redo the survey.
Includes the two limitations left deliberately unasserted, and why pinning
the current behaviour there would be wrong.
tcppm and udppm forward a port each, so test both directions of each: a
request through the TCP mapper reaching its target, a datagram through the
UDP one coming back answered, and a mapper whose rules deny the client
answering nothing.
tlspr takes its destination from the name in the handshake, so point that
name at 127.0.0.1 with nsrecord and give the certificate the same name: the
name then both chooses where the request goes and is what the client checks.
dnspr answers from its cache, including the documented 0.0.0.0 record, which
is handed out as an address rather than withheld.
auto is asked to serve an HTTP proxy request, SOCKS4, SOCKS5 and a TLS
handshake on one port, and to make nothing of a request that is none of
them. Its protocols reach different places, so there are two origins.
The SOCKS UDP association was only checked for the port it binds. Send
datagrams through it as well, large and small, and check a second
association gets its own port - and that one bound inside an intport range
still relays.
A UDP service has no socket to connect to, so readiness is found by asking
until it answers rather than racing it.
A rewrite only reached the copy of the request kept for logging. On a
direct connection the server is sent the request line held in the header
buffer, which was parsed and reduced to its path before the filters ran, so
the rewrite showed up in the log and nowhere else. Through an HTTP parent
the rewritten copy is what goes out, which is why it appeared to work
there.
Splice the new path back into that buffer. The destination was chosen, and
the access rules applied to it, before the rewrite happened, so a rewrite
that changes the method or the authority is left alone: acting on it would
send the request somewhere the rules never saw. Say so in the manual.
The copy needed to notice a rewrite is only taken when a request filter
exists, so a proxy without one does no extra work.
The filter walked its list of access rules but tested the first entry each
time round, so anything pcre_extend appended was never consulted and the
command did nothing at all. Test the entry the loop is on.
pcre_extend takes an ACE and no FILTER_ACTION - the rule keeps the action it
was given - so correct the manual, which documented an argument the command
does not read.
The tests cover matching and denial, rule order, client headers, matching
options, the access rule a pcre rule carries, pcre_extend, and rewriting a
reply header and reply data. Request rewriting is covered through an HTTP
parent, which is the path where it reaches the wire.
The CA was created with no extensions, so it is not usable as a CA and
clients report that they cannot get the local issuer certificate. Add
basicConstraints, keyCertSign and a subject key identifier, in a file
rather than through -addext, which LibreSSL - the openssl on macOS and some
BSDs - does not apply the same way.
Ask for the key identifiers on the signed certificates too: OpenSSL 3 adds
them when it signs and LibreSSL does not, and Python has verified strictly
since 3.13, refusing a chain whose certificate carries no
authorityKeyIdentifier. Finish with openssl verify -x509_strict, which is
the check the client will make.
Both recipes were run against OpenSSL 3.6 and LibreSSL 3.3: the old one
fails strict verification, the new one passes on both.
OpenSSL 3 adds a subject and authority key identifier when it signs;
LibreSSL, which is the openssl on a stock macOS, does not. Python has
verified strictly since 3.13 and rejects a chain whose certificate has no
Authority Key Identifier, so the macOS runners refused a certificate the
Linux ones accepted. Ask for both by name, and make the self-check strict
so the next such gap is caught before a handshake.
httpsrv parsed Content-Length and never read what followed. The reply is
followed by a close, and closing a socket that still holds unread data
resets the connection instead of ending it, so a POST could cost the client
the reply it was about to read. Windows does that reliably; the same test
passes on Linux and macOS, which is why it looked flaky. Drain the body,
bounded at a megabyte.
The test CA was built with -addext, which LibreSSL - the openssl on a stock
macOS - does not apply the same way, leaving a certificate that is not
usable as a CA and a client that cannot build a chain to it. Put the
extensions in a file both accept, and verify the generated chain before any
of it is handed to a proxy, so a failure there is not read as a fault in
the proxy.
The MITM case reached its origin by address, so it depended on the
certificate 3proxy spoofs carrying an IP alternative name. It does when the
upstream certificate is copied, which is what happens on Linux and macOS
but not on Windows, where the client then refused the connection. Point a
name at 127.0.0.1 with nsrecord instead, and check the chain rather than
the name: an intercepted certificate names the upstream host, not the one
that was asked for. The log assertions gain from it too, since the name is
better evidence than a port that the request was seen.
nsrecord needs nserver as well as nscache, and has to follow nscache, so
say that in the manual: the record goes into the table nscache allocates,
and the table is only consulted when nserver is set.
The port-range fallback was exercised with a range the same case had
already used, so on a busy machine it could fail to bind for the ordinary
reason rather than the one under test. Use privileged ports, which nothing
can take.
When a case fails, print what its servers wrote: the reason usually goes to
the server's stderr, which was captured and then thrown away.
Three arrangements, with key material generated for the run rather than
kept in the tree: a proxy wrapped in TLS, a proxy that reaches a TLS parent
and verifies it against the CA, and MITM.
The MITM case checks what interception is for: the decrypted request line,
URI and all, reaches the log, where the same request through a plain
CONNECT tunnel leaves only the host and port.
The origin runs in its own process there so the proxy log holds only what
the proxy saw, and log assertions wait, since a record is written when the
connection finishes rather than when the reply arrives.
Verification of the spoofed certificate is deliberately not strict: 3proxy
issues those without an Authority Key Identifier, which Python rejects
under its 3.13 defaults.
