Listeners: socket options, several services on one address, connecting back

Only the -ol options which act when a socket is bound (SO_REUSEADDR,
SO_REUSEPORT, SO_EXCLUSIVEADDRUSE, SO_PORT_SCALABILITY,
SO_REUSE_UNICASTPORT, IP_TRANSPARENT, TCP_FASTOPEN) must be the same
for a socket to be taken over. The others are set on the socket taken
over, those no longer given turned off; TCP_MAXSEG, whose default is
not 0, still needs a new socket when it is dropped.

Several services may still share an address, in one configuration or
across reloads: each new one takes over a listener of its own, one of
an old service of its type first, or opens another.

A service connecting back (-r) has no socket to share, but now a
listener without one for the place it connects to: on a reload the new
service connecting to the same place takes it over, and the old
service's clients are re-authenticated as its clients, as a listening
service's are.

Nothing moves from srvparam to the listener: its socket is what accept
and UDP sessions use, and the rest is the service's configuration,
which the listener only copies to compare.

Tests: a live option changed keeps the socket, a bind-time one makes
another, two services on one address are both taken over, and a
download through a connect-back service goes on or is cut as its user
is kept or removed. 3proxy(8) describes the takeover and
re-authentication.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

doc/html/man8/3proxy.8.html

@@ -159,10 +159,19 @@ it has an error, the configuration in force stays as it is, its services
 running. Otherwise the new services start and the old ones stop accepting
 within a second; connections already open finish under the old
 configuration. A new service listening on the same address and port, in
-the same network namespace and on the same device, with the same <b>-ol</b>
+the same network namespace and on the same device, takes over the old
-options, takes over the old service's TCP listening socket, so no
+service's TCP or UDP socket, so no connection is refused or lost
-connection is refused or lost meanwhile; any other listening socket is
+meanwhile, and UDP sessions go on through it; a service of the same type
-closed by the old service and opened by the new one. Settings left out of
+is preferred where several share an address, as two services of one
+configuration may. Of the <b>-ol</b> options only those acting when the socket is
+bound must be the same (SO_REUSEADDR, SO_REUSEPORT, SO_EXCLUSIVEADDRUSE,
+IP_TRANSPARENT, TCP_FASTOPEN); the others are set on the socket taken over,
+those no longer given turned off. Otherwise the old service closes its
+socket and the new one opens another. Connections of the old services are
+re-authenticated as clients of the new service on their address, or on the
+place a service connecting back (<b>-r</b>) connects to, unless <b>noforce</b> is set;
+they are closed if their service was removed or is of another type, or if
+the new one has no free slot of its maxconn. Settings left out of
 the new file go back to their defaults.</p>
 
 <p style="margin-left:9%; margin-top: 1em">Under Windows,

man/3proxy.8

@@ -120,10 +120,19 @@ it has an error, the configuration in force stays as it is, its services
 running. Otherwise the new services start and the old ones stop accepting
 within a second; connections already open finish under the old
 configuration. A new service listening on the same address and port, in
-the same network namespace and on the same device, with the same -ol
+the same network namespace and on the same device, takes over the old
-options, takes over the old service's TCP listening socket, so no
+service's TCP or UDP socket, so no connection is refused or lost
-connection is refused or lost meanwhile; any other listening socket is
+meanwhile, and UDP sessions go on through it; a service of the same type
-closed by the old service and opened by the new one. Settings left out of
+is preferred where several share an address, as two services of one
+configuration may. Of the -ol options only those acting when the socket is
+bound must be the same (SO_REUSEADDR, SO_REUSEPORT, SO_EXCLUSIVEADDRUSE,
+IP_TRANSPARENT, TCP_FASTOPEN); the others are set on the socket taken over,
+those no longer given turned off. Otherwise the old service closes its
+socket and the new one opens another. Connections of the old services are
+re-authenticated as clients of the new service on their address, or on the
+place a service connecting back (-r) connects to, unless noforce is set;
+they are closed if their service was removed or is of another type, or if
+the new one has no free slot of its maxconn. Settings left out of
 the new file go back to their defaults.
 .PP
 Under Windows, if

src/proxymain.c

@@ -883,9 +883,14 @@ struct listener {
 	struct listener *next;
 	SOCKET sock;
 	int isudp;
+	/* a service connecting back (-r) has no socket to share, but its
+	   clients find the newest service through a listener all the same:
+	   one without a socket, for the place it connects to */
+	char *cbdest;
 	PROXYSOCKADDRTYPE sa;
 	char *netns, *device;
 	struct sockoptset opts;
+	int maxseg;
 	struct srvparam *owner;	/* the service accepting on it, if any */
 	int users;		/* the services holding it: the one accepting,
 				   and those which did until their last client
@@ -895,8 +900,9 @@ struct listener {
 static struct listener *listeners = NULL;
 _3proxy_mutex_t listener_mutex;
 
-static int listenerkeep(struct srvparam *srv, SOCKET sock, int isudp);
+static int listenerkeep(struct srvparam *srv, SOCKET sock, int isudp, const char *cbdest);
 static void listenerstop(struct srvparam *srv);
+static void takeopts(struct listener *l, struct srvparam *srv);
 static void listenerrelease(struct srvparam *srv);
 #endif
 
@@ -915,6 +921,7 @@ static int srvlisten(struct srvstart *s){
 	/* taken over from a service of the configuration before: listening
 	   already, only the backlog may be new */
 	sock = srv->srvsock = srv->listener->sock;
+	takeopts(srv->listener, srv);
 #ifndef NOUDPMAIN
 	if(s->isudp) defparam->clisock = sock;
 	else
@@ -988,7 +995,7 @@ static int srvlisten(struct srvstart *s){
 #ifndef STDMAIN
 	/* kept for a service of a later configuration to take over; without
 	   memory for it the service closes the socket itself */
-	listenerkeep(srv, sock, 0);
+	listenerkeep(srv, sock, 0, NULL);
 #endif
  }
 #endif
@@ -996,7 +1003,7 @@ static int srvlisten(struct srvstart *s){
  if(s->isudp){
 	defparam->clisock = sock;
 #ifndef STDMAIN
-	listenerkeep(srv, sock, 1);
+	listenerkeep(srv, sock, 1, NULL);
 #endif
  }
 #endif
@@ -1350,6 +1357,10 @@ static int srvrun(struct srvstart *s){
 #endif
 
  if(!connectback && (res = srvlisten(s))) goto EXIT;
+#ifndef STDMAIN
+ /* connecting back there is no socket to share, only the place */
+ if(connectback && !srv->listener && s->cbc_string) listenerkeep(srv, INVALID_SOCKET, 0, (char *)s->cbc_string);
+#endif
 #ifndef NOTCPMAIN
  if(!s->isudp && s->iscbl && (res = cblisten(s))) goto EXIT;
 #endif
@@ -1535,8 +1546,60 @@ static int samestr(const char *a, const char *b){
  return (!a && !b) || (a && b && !strcmp(a, b));
 }
 
+/* -ol options that act only when set before bind(): a listening socket
+   differing in them is another socket. The rest are set on a socket taken
+   over, those no longer asked for turned off. */
+static int bindtimeopt(const char *name){
+ return !strncmp(name, "SO_REUSE", 8) || !strcmp(name, "SO_EXCLUSIVEADDRUSE") ||
+	!strcmp(name, "SO_PORT_SCALABILITY") || !strcmp(name, "IP_TRANSPARENT") ||
+	!strcmp(name, "TCP_FASTOPEN");
+}
+
+/* Whether a socket with the options of l can be made one with those of srv:
+   the same bind-time ones, with the same values; TCP_MAXSEG, whose default
+   is not 0, cannot be turned off again. */
+static int sameopts(struct listener *l, struct srvparam *srv){
+ const struct sockoptset *a = &l->opts, *b = &srv->lissockopts;
+ int i;
+
+ for(i = 0; sockopts[i].optname && i < MAXSOCKOPTS; i++){
+	unsigned bit = 1u << i;
+
+	if(bindtimeopt(sockopts[i].optname)){
+		if((a->set & bit) != (b->set & bit)) return 0;
+		if((a->set & bit) && a->value[i] != b->value[i]) return 0;
+	}
+#ifdef TCP_MAXSEG
+	else if(sockopts[i].opt == TCP_MAXSEG && *sockopts[i].optname == 'T' &&
+		(a->set & bit) && !(b->set & bit)) return 0;
+#endif
+ }
+ return 1;
+}
+
+/* Sets the options of srv on the socket of l taken over: those given, and
+   off those which l had and srv has not. */
+static void takeopts(struct listener *l, struct srvparam *srv){
+ struct sockoptset off;
+ int i;
+
+ memset(&off, 0, sizeof(off));
+ for(i = 0; sockopts[i].optname && i < MAXSOCKOPTS; i++){
+	unsigned bit = 1u << i;
+
+	if((l->opts.set & bit) && !(srv->lissockopts.set & bit) && !bindtimeopt(sockopts[i].optname))
+		off.set |= bit;
+ }
+ if(off.set) setopts(l->sock, &off, 0);
+ setopts(l->sock, &srv->lissockopts, srv->maxseg);
+ _3proxy_mutex_lock(&listener_mutex);
+ l->opts = srv->lissockopts;
+ l->maxseg = srv->maxseg;
+ _3proxy_mutex_unlock(&listener_mutex);
+}
+
 static int samelistener(struct listener *l, struct srvparam *srv, int isudp){
- if(l->isudp != isudp) return 0;
+ if(l->isudp != isudp || l->cbdest) return 0;
 #ifdef __linux__
  if(!samestr(l->netns, srv->inetns)) return 0;
 #endif
@@ -1545,46 +1608,55 @@ static int samelistener(struct listener *l, struct srvparam *srv, int isudp){
 #endif
  return SASIZE(&l->sa) == SASIZE(&srv->intsa) &&
 	!memcmp(&l->sa, &srv->intsa, SASIZE(&srv->intsa)) &&
-	!memcmp(&l->opts, &srv->lissockopts, sizeof(l->opts));
+	sameopts(l, srv);
 }
 
 /* On a reload, before the old services are told to stop: a prepared
-   service takes over the listening socket of an older configuration's
+   service takes over the listener of an older configuration's service on
-   service on the same address, if there is one. */
+   the same address, if there is one, of the same type first. A service
+   connecting back takes over the pseudo-listener of one connecting to the
+   same place. Two services of one configuration may share an address:
+   each takes a listener of its own, or opens one. */
 void mainclaim(void *service){
  struct srvstart *s = (struct srvstart *)service;
  struct srvparam *srv = &s->srv;
- struct listener *l;
+ struct listener *l, *found = NULL;
+ int cb = !s->isudp && s->iscbc;
 
- if(!s->isudp && s->iscbc) return;	/* connects back: no socket of its own */
  _3proxy_mutex_lock(&listener_mutex);
  for(l = listeners; l; l = l->next){
-	if(l->owner && l->sock != INVALID_SOCKET && l->owner->config != srv->config && samelistener(l, srv, s->isudp)){
+	if(!l->owner || l->owner->config == srv->config) continue;
-		l->owner = srv;
+	if(cb? !(l->cbdest && s->cbc_string && !strcmp(l->cbdest, (char *)s->cbc_string)) :
-		l->users++;
+		(l->sock == INVALID_SOCKET || !samelistener(l, srv, s->isudp))) continue;
-		srv->listener = l;
+	if(!found || (l->owner->service == srv->service && found->owner->service != srv->service))
-		break;
+		found = l;
  }
+ if(found){
+	found->owner = srv;
+	found->users++;
+	srv->listener = found;
  }
  _3proxy_mutex_unlock(&listener_mutex);
 }
 
 /* A socket a service opened and bound itself, kept for whoever may take it
    over; 0, or 1 for want of memory, when the service closes it itself. */
-static int listenerkeep(struct srvparam *srv, SOCKET sock, int isudp){
+static int listenerkeep(struct srvparam *srv, SOCKET sock, int isudp, const char *cbdest){
  struct listener *l;
 
  if(!(l = malloc(sizeof(*l)))) return 1;
  memset(l, 0, sizeof(*l));
  l->sock = sock;
  l->isudp = isudp;
+ l->maxseg = srv->maxseg;
+ if(cbdest && !(l->cbdest = strdup(cbdest))) { free(l); return 1; }
  l->sa = srv->intsa;
  l->opts = srv->lissockopts;
 #ifdef __linux__
  if(srv->inetns && !(l->netns = strdup(srv->inetns))) { free(l); return 1; }
 #endif
 #if defined SO_BINDTODEVICE || defined IP_BOUND_IF
- if(srv->ibindtodevice && !(l->device = strdup(srv->ibindtodevice))) { free(l->netns); free(l); return 1; }
+ if(srv->ibindtodevice && !(l->device = strdup(srv->ibindtodevice))) { free(l->netns); free(l->cbdest); free(l); return 1; }
 #endif
  l->owner = srv;
  l->users = 1;
@@ -1641,6 +1713,7 @@ static void listenerrelease(struct srvparam *srv){
  }
  if(l->netns) free(l->netns);
  if(l->device) free(l->device);
+ if(l->cbdest) free(l->cbdest);
  free(l);
 }
 

tests/cases/reattach.py

@@ -2,6 +2,8 @@
 the rules of the newest service on its listener, taking one of its maxconn
 slots, and is dropped where that cannot be."""
 
+import base64
+import socket
 import threading
 import time
 
@@ -81,3 +83,58 @@ def run(t):
         t.fail("a proxy download is cut when a socks service takes the port",
                "less than 2000000 bytes", results[0].get("length"))
     t.contains(server.output(), " 00026 ", "as its service is another one now (26)")
+
+    # --- a service connecting back (-r) has no socket, but a listener for the
+    # place it connects to: its clients are re-authenticated all the same ---
+    peer = socket.socket()
+    peer.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+    peer.bind(("127.0.0.1", 0))
+    peer.listen(16)
+    peer.settimeout(10)
+    cb = peer.getsockname()[1]
+
+    def cbconfig(users):
+        return config(f"proxy -p{proxy}").replace(
+            "flush\n            maxconn 50\n            auth iponly\n            allow *\n            admin",
+            f"flush\n            users {users}\n            auth strong\n            allow *\n"
+            f"            proxy -r127.0.0.1:{cb}\n\n            flush\n            maxconn 50\n"
+            "            auth iponly\n            allow *\n            admin")
+
+    def cb_download(result):
+        """A request over a connection the -r service made: the peer is its
+        client once it has sent C."""
+        try:
+            c, _ = peer.accept()
+            c.settimeout(20)
+            c.sendall(b"C")
+            auth = base64.b64encode(b"alice:one").decode()
+            c.sendall((f"GET http://127.0.0.1:{origin}/data?size=2000000&block=65536&delay=60 HTTP/1.1\r\n"
+                       f"Host: 127.0.0.1:{origin}\r\nProxy-Authorization: Basic {auth}\r\n"
+                       "Connection: close\r\n\r\n").encode())
+            got = b""
+            while True:
+                b = c.recv(65536)
+                if not b:
+                    break
+                got += b
+            c.close()
+            result["body"] = len(got.split(b"\r\n\r\n", 1)[1]) if b"\r\n\r\n" in got else 0
+        except OSError as e:
+            result["error"] = str(e)
+
+    reload(cbconfig("alice:CL:one"))
+    for keep, users, want in ((True, "alice:CL:one bob:CL:two", "finishes whole"),
+                              (False, "bob:CL:two", "is cut")):
+        result = {}
+        th = threading.Thread(target=cb_download, args=(result,))
+        th.start()
+        time.sleep(0.5)
+        reload(cbconfig(users))
+        th.join(25)
+        whole = result.get("body") == 2000000
+        name = f"a download through a connect-back service {want} when alice is " + \
+            ("kept" if keep else "removed")
+        t.ok(name) if whole == keep else t.fail(name, "whole" if keep else "cut", result)
+        if keep:
+            reload(cbconfig("alice:CL:one"))
+    peer.close()

tests/cases/takeover.py

@@ -118,6 +118,36 @@ def run(t):
     t.contains(server.output(), f"{udp} 0.0.0.0:0 0 0 0 Accepting connections, taken over",
                "the UDP service takes the socket over")
 
+    # --- options: a live -ol option changed is set on the socket taken over,
+    # a bind-time one makes another socket; two services of one
+    # configuration on one address are each taken over ----------------------
+    def taken_for(port, since):
+        return server.output()[since:].count(f":{port} 0.0.0.0:0 0 0 0 Accepting connections, taken over")
+
+    def reload_conf(c):
+        mark = len(server.output())
+        t.write_config("takeover", c)
+        t.http(f"http://127.0.0.1:{adm}/R")
+        time.sleep(1.5)
+        return mark
+
+    oconf = conf.replace(f"proxy -p{proxy}", f"proxy -olTCP_NODELAY -p{proxy}")
+    reload_conf(oconf)
+    mark = reload_conf(oconf.replace("-olTCP_NODELAY", "-olTCP_NODELAY,SO_KEEPALIVE"))
+    t.eq(1, taken_for(proxy, mark), "a live socket option changed: the socket is taken over")
+    t.eq(200, t.http(f"http://127.0.0.1:{origin}/data?size=100", proxy=f"127.0.0.1:{proxy}").status,
+         "and serves")
+    mark = reload_conf(oconf.replace("-olTCP_NODELAY", "-olTCP_NODELAY,SO_KEEPALIVE,SO_REUSEADDR=0"))
+    t.eq(0, taken_for(proxy, mark), "a bind-time socket option changed: another socket")
+    t.ok("which serves") if t.wait_port(proxy) and t.http(
+        f"http://127.0.0.1:{origin}/data?size=100", proxy=f"127.0.0.1:{proxy}").status == 200 else \
+        t.fail("which serves", 200, "no answer")
+    twice = conf.replace(f"proxy -p{proxy}", f"proxy -p{proxy}\n        proxy -p{proxy}")
+    reload_conf(twice)
+    mark = reload_conf(twice)
+    t.eq(2, taken_for(proxy, mark), "two services on one address are both taken over")
+    reload_conf(conf)
+
     # a changed port is a new socket: the old one closes, the new one opens
     moved = t.free_port()
     t.write_config("takeover", conf.replace(f"proxy -p{proxy}", f"proxy -p{moved}"))