bughz/3proxy
1
0
Fork 0
mirror of https://github.com/3proxy/3proxy.git synced 2026-10-16 00:25:48 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity

Support = in socket options, make TCP_NODELAY default

Browse Source
This commit is contained in:
Vladimir Dubrovin 2026-10-11 20:56:49 +03:00
parent 0e8d757464
commit c827f77137
7 changed files with 144 additions and 81 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

43
doc/html/highload.html
View File

@ -34,7 +34,8 @@ Each connected client requires:
<ul>
<li>1 thread (process)
<li>2 sockets (file descriptors). For FTP, 4 sockets are required.
<br>Up to 128K of kernel buffer memory. This is the theoretical maximum; actual numbers depend on connection quality and traffic amount.
<br>With splice() (Linux, -s), 2 more pipes (file descriptors) = 4 file descriptors, 6 for FTP.
<br>Up to 128K (up to 256K with splice()) of kernel buffer memory. This is the theoretical maximum; actual numbers depend on connection quality and traffic amount.
<br>1 additional socket (file descriptor) during name resolution for non-cached names
<br>1 additional socket during authentication or logging for RADIUS authentication or logging.
<li>1 ephemeral port (3 ephemeral ports for FTP connections).
@ -279,6 +280,8 @@ tuning anything else.
If 3proxy is used in a VPS environment, there can be additional limitations.
For example, kernel resources, system CPU usage, and IOCTLs can be limited differently, and this can become a bottleneck.
splice() (see below) moves work into the kernel, so where the hosting limits kernel resources
(seen as nearly 100% CPU usage without any real activity of the application) do not enable it.
<h4>Extending the Ephemeral Port Range</h4>
@ -641,17 +644,37 @@ monitor "/some/path/3proxy/3proxy.ver"
e.g., with "touch /some/path/3proxy/3proxy.ver".
</ol>
<h4>Use TCP_NODELAY to Speed Up Connections with Small Amounts of Data</h4>
<h4>TCP_NODELAY for Connections with Small Amounts of Data</h4>
If most requests require an exchange with a small amount of data in both directions
without the need for bandwidth, e.g., messengers or small web requests,
you can eliminate Nagle's algorithm delay with the TCP_NODELAY flag. Usage example:
TCP_NODELAY is set by default on client (oc) and server (os)
connections: without it, Nagle's algorithm holds back a short write until the
previous one is acknowledged, and a peer delays its acknowledgement, so a reply
written in more than one part (e.g. HTTP headers, then a short body) waits tens
of milliseconds. On slow connections with high delays, where the bandwidth is the
bottleneck and fewer packets matter more than latency, it can be turned off:
<pre>
proxy -osTCP_NODELAY -ocTCP_NODELAY
proxy -osTCP_NODELAY=0 -ocTCP_NODELAY=0
</pre>
sets TCP_NODELAY for client (oc) and server (os) connections.
<p>Do not use TCP_NODELAY on slow connections with high delays when
connection bandwidth is a bottleneck.
<h4>Use Splice to Speed Up Large Data Transfers</h4>
On Linux, splice() passes data between the client and the server connections
through a kernel pipe, without copying it into the 3proxy process. Linux builds
include it, but a service uses it only when given the -s option:
<pre>
proxy -s
socks -s
tcppm -s 8080 192.168.1.1 80
</pre>
-s0 turns it off explicitly. In builds without it (CMake option 3PROXY_USE_SPLICE=OFF,
or -DWITHSPLICE removed from the Makefile) -s is accepted and ignored.
<p>splice() pays off when most of the traffic is large transfers: in the
<a href="https://github.com/3proxy/3proxy/tree/master/bench">benchmarks</a> it moved
1.4 to 1.7 times more data than the read/write path at about half the CPU per gigabyte.
It does not help small requests and short connections, which go slower with it, and
every spliced connection needs two more pipes (file descriptors) and more kernel
buffer memory. Data filters are not applied to spliced data.
<p>TCP_NODELAY and splice() do not conflict and can be combined on high-speed connections.
<h4>Add Grace Delay to Reduce System Calls</h4>
@ -661,7 +684,7 @@ the minimal number of packets in the same direction to apply the algorithm,
and the last value is the delay added after polling and prior to reading data.
The example above adds a 10-millisecond delay before reading data if the average
polling size is below 8000 bytes and 3 read operations have been made in the same
direction. <pre>logdump 1 1</pre> is useful
direction. It's especially useful with splice. <pre>logdump 1 1</pre> is useful
to see how grace delays work; choose a delay value to avoid filling the read
buffer (typically 64K) but keep the request sizes close to the chosen average
on large file uploads/downloads.

16
man/3proxy.cfg.5
View File

@ -184,8 +184,10 @@ connect to given remote HOST:port instead of listening local connection on -p or
.br
.B -oc\fIOPTIONS\fB, -os\fIOPTIONS\fB, -ol\fIOPTIONS\fB, -or\fIOPTIONS\fB, -oR\fIOPTIONS\fR
options for proxy-to-client (\fB-oc\fR), proxy-to-server (\fB-os\fR), proxy listening (\fB-ol\fR), connect back client (\fB-or\fR), connect back listening (\fB-oR\fR) sockets.
\fIOPTIONS\fR is a comma separated list of \fINAME\fR or \fINAME\fR=\fIVALUE\fR; a name alone sets the option to 1 (TCP_MAXSEG to the value of \fBmaxseg\fR), and \fINAME\fR=0 turns it off. Options add to the defaults and to the options given earlier for the same socket. An unknown name or a value which is not a number is an error.
TCP_NODELAY is set by default on proxy-to-client and proxy-to-server sockets, so a reply written in several parts is not delayed by Nagle's algorithm; use \fB-ocTCP_NODELAY=0\fR or \fB-osTCP_NODELAY=0\fR to turn it off, e.g. to send fewer packets on slow links.
Options like TCP_CORK, TCP_NODELAY, TCP_DEFER_ACCEPT, TCP_QUICKACK, TCP_TIMESTAMPS, TCP_FASTOPEN, SO_REUSEADDR, SO_REUSEPORT, SO_EXCLUSIVEADDRUSE, SO_PORT_SCALABILITY, SO_REUSE_UNICASTPORT, SO_KEEPALIVE, SO_DONTROUTE may be supported depending on OS.
SO_REUSEADDR and SO_REUSEPORT are set on the listening socket by default on Unix. On Windows SO_REUSEADDR is not set: it is not required to rebind a listening port and it only lets another local process bind the same address and port. Use SO_EXCLUSIVEADDRUSE (Windows) on the listening socket (\fB-ol\fR) to prevent that.
SO_REUSEADDR and SO_REUSEPORT are set by default on the listening sockets (\fB-ol\fR, \fB-oR\fR) on Unix; \fB-olSO_REUSEPORT=0\fR turns one off. On Windows SO_REUSEADDR is not set: it is not required to rebind a listening port and it only lets another local process bind the same address and port. Use SO_EXCLUSIVEADDRUSE (Windows) on the listening socket (\fB-ol\fR) to prevent that.
.br
.B -H
(for all services) Expect HAProxy PROXY protocol v1 header on incoming connection.
@ -200,11 +202,13 @@ delay GRACE_DELAY milliseconds before polling if average polling size is below G
(for admin) secure, allow only secure operations, currently only traffic counters
view without ability to reset.
.br
(for TCP services, Linux) enable splice(). splice() is not built by default and
is disabled even when built, because current Linux does not implement
SPLICE_F_MOVE, so no real zero-copy takes place and the read/write path is faster
for most traffic. Rebuild with -DWITHSPLICE to make -s available, -s0 disables it
explicitly.
(for TCP services, Linux) use splice() to pass data between the client and the
server without copying it through 3proxy. It is off unless -s is given (-s0 turns
it off explicitly). splice() suits large transfers: they go faster and use less
CPU. Small requests and short connections go slower with it, every connection
needs two more pipes (file descriptors) and more kernel memory, and data filters
are not applied to spliced data. Available in builds with WITHSPLICE, the default
on Linux.
.br
.B -C
(for TCP services) a connection closed by any of the sides terminates the

2
src/common.c
View File

@ -746,7 +746,7 @@ int doconnect(struct clientparam * param){
param->sinsl = param->srv->extsa;
}
*SAPORT(&param->sinsl) = 0;
setopts(param->remsock, param->srv->srvsockopts);
setopts(param->remsock, &param->srv->srvsockopts);
#ifndef _WIN32
/* not on Windows, see the accepting side in proxymain.c */

3
src/proxy.h
View File

@ -501,7 +501,8 @@ extern struct socketoptions {
int opt;
char * optname;
} sockopts[];
void setopts(SOCKET s, int opts);
int getopts(const char *s, struct sockoptset *opts);
void setopts(SOCKET s, const struct sockoptset *opts);
char * printopts(char *sep);
#ifdef _WINCE

149
src/proxymain.c
View File

@ -224,27 +224,62 @@ char * printopts(char *sep){
}
int getopts(const char *s){
int i=0, ret=0;
for(; sockopts[i].optname; i++)if(strstr(s,sockopts[i].optname)) ret |= (1<<i);
return ret;
/* A comma separated list of options, each NAME or NAME=VALUE; a name alone
stands for 1, or for TCP_MAXSEG for the value of maxseg. Options are added
to what is already set, so a default is changed only by naming it, and
turned off with NAME=0. An unknown name or a value which is not a number
makes the whole list fail. */
int getopts(const char *s, struct sockoptset *opts){
while(*s){
const char *end = strchr(s, ',');
const char *eq;
size_t len;
int i, value = 1;
if(!end) end = s + strlen(s);
eq = memchr(s, '=', (size_t)(end - s));
len = (size_t)((eq? eq : end) - s);
if(eq){
char *num;
if(!len) return -1;
value = (int)strtol(eq + 1, &num, 0);
if(num == eq + 1 || num != end) return -1;
}
else if(len) {
#ifdef TCP_MAXSEG
if(len == 10 && !strncmp(s, "TCP_MAXSEG", 10)) value = 0;
#endif
}
if(len){
for(i = 0; sockopts[i].optname; i++)
if(strlen(sockopts[i].optname) == len && !strncmp(s, sockopts[i].optname, len)) break;
if(!sockopts[i].optname || i >= MAXSOCKOPTS) return -1;
opts->set |= (1u << i);
opts->value[i] = value;
}
s = *end? end + 1 : end;
}
return 0;
}
void setopts(SOCKET s, int opts){
int i, opt, set;
for(i = 0; opts >= (opt = (1<<i)); i++){
set = 1;
void setopts(SOCKET s, const struct sockoptset *opts){
int i, value;
for(i = 0; sockopts[i].optname && i < MAXSOCKOPTS; i++){
if(!(opts->set & (1u << i))) continue;
value = opts->value[i];
#ifdef TCP_MAXSEG
if(sockopts[i].opt == TCP_MAXSEG){
if(!conf.maxseg) continue;
set = conf.maxseg;
if(sockopts[i].opt == TCP_MAXSEG && *sockopts[i].optname == 'T'){
if(!value) value = conf.maxseg;
if(!value) continue;
}
#endif
if(opts & opt) setsockopt(s, *sockopts[i].optname == 'T'? IPPROTO_TCP:
setsockopt(s, *sockopts[i].optname == 'T'? IPPROTO_TCP:
#ifdef SOL_IP
*sockopts[i].optname == 'I'? SOL_IP:
#endif
SOL_SOCKET, sockopts[i].opt, (char *)&set, sizeof(set));
SOL_SOCKET, sockopts[i].opt, (char *)&value, sizeof(value));
}
}
@ -356,7 +391,9 @@ int MODULEMAINFUNC (int argc, char** argv){
" -64 Prefer IPv6 for outgoing connections, use both IPv4 and IPv6\n"
" -ocOPTIONS, -osOPTIONS, -olOPTIONS, -orOPTIONS -oROPTIONS - options for\n"
" to-client (oc), to-server (os), listening (ol) socket, connect back client\n"
" (or) socket, connect back server (oR) listening socket\n"
" (or) socket, connect back server (oR) listening socket, a comma separated\n"
" list of NAME or NAME=VALUE (NAME=0 turns an option off); TCP_NODELAY is\n"
" set on to-client and to-server sockets by default\n"
" where possible options are: ";
#endif
@ -401,6 +438,34 @@ int MODULEMAINFUNC (int argc, char** argv){
srvinit(&srv, &defparam);
srv.pf = childdef.pf;
isudp = childdef.isudp;
/* Default socket options, which -oc, -os, -ol and -oR change by naming
them, NAME=0 to turn one off.
Nagle's algorithm holds back a short write until the previous one is
acknowledged, and a peer delays its acknowledgement, which stalls every
reply written in more than one piece.
SO_REUSEADDR is not set on Windows: it is not needed to rebind a
listening port there, and it only allows another local process to bind
the same address and port, with undefined behaviour as to which socket
receives the connections. Use -olSO_EXCLUSIVEADDRUSE to prevent that
instead. A Windows build which does not share the listening socket is the
exception: its UDP services bind a second socket to the same address to
answer from, and Windows only allows that when both sockets ask for it.
*/
(void)getopts("TCP_NODELAY", &srv.clisockopts);
(void)getopts("TCP_NODELAY", &srv.srvsockopts);
#ifndef _WIN32
(void)getopts("SO_REUSEADDR", &srv.lissockopts);
(void)getopts("SO_REUSEADDR", &srv.cbssockopts);
#elif !defined SHARE_UDP_SOCKET
if(isudp) (void)getopts("SO_REUSEADDR", &srv.lissockopts);
#endif
#ifdef SO_REUSEPORT
(void)getopts("SO_REUSEPORT", &srv.lissockopts);
(void)getopts("SO_REUSEPORT", &srv.cbssockopts);
#endif
#ifndef NOUDPMAIN
if(isudp) {
if(!udp_table.ihashtable)inithashtable(&udp_table, 64, 256, 65536);
@ -638,19 +703,19 @@ int MODULEMAINFUNC (int argc, char** argv){
case 'o':
switch(argv[i][2]){
case 's':
srv.srvsockopts = getopts(argv[i]+3);
error = getopts(argv[i]+3, &srv.srvsockopts);
break;
case 'c':
srv.clisockopts = getopts(argv[i]+3);
error = getopts(argv[i]+3, &srv.clisockopts);
break;
case 'l':
srv.lissockopts = getopts(argv[i]+3);
error = getopts(argv[i]+3, &srv.lissockopts);
break;
case 'r':
srv.cbcsockopts = getopts(argv[i]+3);
error = getopts(argv[i]+3, &srv.cbcsockopts);
break;
case 'R':
srv.cbcsockopts = getopts(argv[i]+3);
error = getopts(argv[i]+3, &srv.cbssockopts);
break;
default:
error = 1;
@ -829,42 +894,13 @@ int MODULEMAINFUNC (int argc, char** argv){
perror("socket()");
return -2;
}
setopts(sock, srv.lissockopts);
setopts(sock, &srv.lissockopts);
#ifdef _WIN32
ioctlsocket(sock, FIONBIO, &ul);
#else
fcntl(sock,F_SETFL,O_NONBLOCK | fcntl(sock,F_GETFL));
#endif
srv.srvsock = sock;
#ifdef WITH_UN
if(*SAFAMILY(&srv.intsa) != AF_UNIX)
#endif
{
/* SO_REUSEADDR is not set on Windows: it is not needed to rebind a listening
port there, and it only allows another local process to bind the same
address and port, with undefined behaviour as to which socket receives the
connections. Use -olSO_EXCLUSIVEADDRUSE to prevent that instead.
A Windows build which does not share the listening socket is the exception:
its UDP services bind a second socket to the same address to answer from,
and Windows only allows that when both sockets ask for it.
*/
#ifndef _WIN32
opt = 1;
if(srv.so._setsockopt(srv.so.state, sock, SOL_SOCKET, SO_REUSEADDR, (char *)&opt, sizeof(int)))perror("setsockopt()");
#else
#ifndef SHARE_UDP_SOCKET
if(isudp){
opt = 1;
if(srv.so._setsockopt(srv.so.state, sock, SOL_SOCKET, SO_REUSEADDR, (char *)&opt, sizeof(int)))perror("setsockopt()");
}
#endif
#endif
#ifdef SO_REUSEPORT
opt = 1;
srv.so._setsockopt(srv.so.state, sock, SOL_SOCKET, SO_REUSEPORT, (char *)&opt, sizeof(int));
#endif
}
#if defined SO_BINDTODEVICE || defined IP_BOUND_IF
if(bindtodevice(&srv.so, srv.so.state, sock, srv.ibindtodevice, *SAFAMILY(&srv.intsa))) {
dolog(&defparam, (unsigned char *)"failed to bind device");
@ -957,16 +993,7 @@ int MODULEMAINFUNC (int argc, char** argv){
freesrvstrings(&srv, cbc_string, cbl_string);
return -6;
}
#ifndef _WIN32
opt = 1;
srv.so._setsockopt(srv.so.state, srv.cbsock, SOL_SOCKET, SO_REUSEADDR, (char *)&opt, sizeof(int));
#endif
#ifdef SO_REUSEPORT
opt = 1;
srv.so._setsockopt(srv.so.state, srv.cbsock, SOL_SOCKET, SO_REUSEPORT, (char *)&opt, sizeof(int));
#endif
setopts(srv.cbsock, srv.cbssockopts);
setopts(srv.cbsock, &srv.cbssockopts);
if(srv.so._bind(srv.so.state, srv.cbsock, (struct sockaddr*)&cbsa, SASIZE(&cbsa))==-1) {
dolog(&defparam, (unsigned char *)"Failed to bind connect back socket");
@ -1025,7 +1052,7 @@ int MODULEMAINFUNC (int argc, char** argv){
if(iscbc){
new_sock=so._socket(so.state, SASOCK(&defparam.sincr), SOCK_STREAM, IPPROTO_TCP);
if(new_sock != INVALID_SOCKET){
setopts(new_sock, srv.cbcsockopts);
setopts(new_sock, &srv.cbcsockopts);
parsehost(srv.family, cbc_string, (struct sockaddr *)&defparam.sincr);
#if defined SO_BINDTODEVICE || defined IP_BOUND_IF
@ -1102,7 +1129,7 @@ int MODULEMAINFUNC (int argc, char** argv){
}
continue;
}
setopts(new_sock, srv.clisockopts);
setopts(new_sock, &srv.clisockopts);
}
size = sizeof(defparam.sincl);
if(srv.so._getsockname(srv.so.state, new_sock, (struct sockaddr *)&defparam.sincl, &size)){

2
src/socks.c
View File

@ -184,7 +184,7 @@ void * sockschild(struct clientparam* param) {
param->operation = command == 2?BIND:UDPASSOC;
if(command == 2){
if ((param->remsock=param->srv->so._socket(param->sostate, SASOCK(&param->req), SOCK_STREAM, IPPROTO_TCP)) == INVALID_SOCKET) {RETURN (11);}
setopts(param->remsock, param->srv->lissockopts);
setopts(param->remsock, &param->srv->lissockopts);
if(extbindtodevice(param, param->remsock)) {RETURN (12);}
}
break;

10
src/structures.h
View File

@ -529,6 +529,14 @@ struct filterp {
#define MAX_FILTERS 16
/* Socket options for one kind of socket, -oc and the like: which of
sockopts[] are given, by their index there, and the value of each. */
#define MAXSOCKOPTS 32
struct sockoptset {
unsigned set;
int value[MAXSOCKOPTS];
};
struct sockfuncs {
void * state;
@ -613,7 +621,7 @@ struct srvparam {
int stacksize;
int noforce;
int anonymous;
int clisockopts, srvsockopts, lissockopts, cbcsockopts, cbssockopts;
struct sockoptset clisockopts, srvsockopts, lissockopts, cbcsockopts, cbssockopts;
int gracetraf, gracenum, gracedelay;
int requirecert;
int haproxy;