Logs: a thread and a queue for each log; build-time defaults

Once 3proxy runs (after the first configuration is in force, so after a
daemon's fork), each logger has a thread of its own writing from a
queue, and a client never waits for a log: ODBC, RADIUS, a full disk or
a pipe nobody reads hold up only that log's thread.

A queued line is fixed size: the part of struct clientparam logging
reads (everything before the filters: addresses, counters, names and
credentials, which hold no pointer it follows but srv), the time it was
logged, and its text, 256 bytes inline or a copy of its own if longer.
The thread formats it: struct tm is per call now, timechanged() and
dologname() included, and the type's format() takes the time.

minlogqueue (64) slots are allocated as the thread starts and kept,
maxlogqueue (4096) lines are queued at most. A line which finds the
queue full is dropped and counted on the last one queued; once that is
written, "Log X has its queue full, N records dropped" goes to where
errors go. A log which fails is no longer written to the error log
instead: its lines are dropped and counted while it is retried every
minute, and the count is told when it is open again, or when it was
reopened after failed writes.

A service which ended is stopped at once (srvstop(): sockets, listener,
its clients gone), but freed (srvrelease()) through its log's queue,
behind the lines which refer to it and to its configuration: the event
is part of struct srvparam, so a full queue never drops it. logfinish()
writes what is queued, 10 seconds at most, also as 3proxy exits on Unix.
Standalone services write as before, as each line comes.

LOGQUEUEMIN, LOGQUEUEMAX, DEFAULT_MAXCHILD, DEFAULT_POOLMAX,
DEFAULT_POOLTIMEOUT and POOLPERSERVICE (the automatic thread pool cap
for each service, which was 32 in the code) can be set with -D at build
time.

Tests: a log nobody reads (a FIFO) holds no client up, drops past its
queue and says so, and lines queued for a service a reload removed are
written as it formatted them; /dev/full says what it dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Vladimir Dubrovin 2026-10-23 19:41:48 +03:00
parent a3c417c82a
commit 2052dc0a4f
13 changed files with 930 additions and 158 deletions

View File

@ -349,12 +349,23 @@ log or by -l, and shared by every service using it; it is kept
open over a configuration reload while some service still uses
it, and closed once none does. stdout, stderr and syslog are
opened at startup and never closed. rotate and archiver apply
to the file of the last log command. A log which fails to be
written is reopened, and if it cannot be, its records go to
stderr while it is retried every minute. Errors and warnings
to the file of the last log command. Errors and warnings
with no connection go to stderr; after daemon (or as a Windows
service) nothing goes to stdout or stderr: logging to stdout is
none, and errors go to the last log configured. <br>
Once 3proxy runs, each log is written by a thread of its own
from a queue (see <b>minlogqueue</b> and <b>maxlogqueue</b>):
a client never waits for a log, a slow one (ODBC, RADIUS, a
full disk) included. A record which finds the queue full is
dropped. A log which fails to be written is reopened; if it
cannot be, its records are dropped while it is retried every
minute. Each time records are dropped, how many is said where
errors go: &quot;Log <i>name</i> has its queue full,
<i>N</i> records dropped&quot;, &quot;could not be written,
opened again&quot;, &quot;failed&quot;, &quot;is open
again&quot;. A service a reload removed is freed only once the
records logged for it are written. Standalone services write
their log as each record comes. <br>
log command supports same format specifications for filename
template as &quot;logformat&quot; (if filename contains
&acute;%&acute; sign it&acute;s believed to be template). As
@ -367,6 +378,17 @@ specificators.</p>
<i>&lt;n&gt;</i> <br>
how many archived log files to keep</p>
<p style="margin-left:9%; margin-top: 1em"><b>minlogqueue</b>
<i>&lt;n&gt;</i> <br>
<b>maxlogqueue</b> <i>&lt;n&gt;</i> <br>
The records a log&acute;s queue keeps room for, allocated as
its thread starts (64 by default), and holds at most, records
past it being dropped (4096 by default). Room taken by a burst
beyond minlogqueue is freed as the queue empties. A record
takes about 2 KB. The values of the configuration in force
apply to every log. Other defaults can be given at build time
with -DLOGQUEUEMIN=<i>n</i> and -DLOGQUEUEMAX=<i>n</i>.</p>
<p style="margin-left:9%; margin-top: 1em"><b>logformat</b>
<i>&lt;format&gt;</i> <br>

View File

@ -353,9 +353,19 @@ alone is stdout.
A log is opened once for the same argument, whether given by log or by -l, and shared by every service using it; it
is kept open over a configuration reload while some service still uses it, and closed once none does. stdout,
stderr and syslog are opened at startup and never closed. rotate and archiver apply to the file of the last log
command. A log which fails to be written is reopened, and if it cannot be, its records go to stderr while it is
retried every minute. Errors and warnings with no connection go to stderr; after daemon (or as a Windows service)
command. Errors and warnings with no connection go to stderr; after daemon (or as a Windows service)
nothing goes to stdout or stderr: logging to stdout is none, and errors go to the last log configured.
.br
Once 3proxy runs, each log is written by a thread of its own from a queue (see
.BR minlogqueue
and
.BR maxlogqueue ):
a client never waits for a log, a slow one (ODBC, RADIUS, a full disk) included. A record which finds the
queue full is dropped. A log which fails to be written is reopened; if it cannot be, its records are dropped
while it is retried every minute. Each time records are dropped, how many is said where errors go: "Log
\fIname\fR has its queue full, \fIN\fR records dropped", "could not be written, opened again", "failed",
"is open again". A service a reload removed is freed only once the records logged for it are written.
Standalone services write their log as each record comes.
.br
log command supports same format specifications for filename template
as "logformat" (if filename contains \'%\' sign it\'s believed to be template).
@ -367,6 +377,18 @@ Grinwitch time zone for all time-based format specificators.
\fI<n>\fR
how many archived log files to keep
.br
.BR minlogqueue
\fI<n>\fR
.br
.BR maxlogqueue
\fI<n>\fR
.br
The records a log's queue keeps room for, allocated as its thread starts (64 by default), and holds at
most, records past it being dropped (4096 by default). Room taken by a burst beyond minlogqueue is freed
as the queue empties. A record takes about 2 KB. The values of the configuration in force apply to every
log. Other defaults can be given at build time with -DLOGQUEUEMIN=\fIn\fR and -DLOGQUEUEMAX=\fIn\fR.
.br
.BR logformat
\fI<format>\fR
@ -1478,7 +1500,11 @@ instead of ending, and keeps the memory it had for its connection: a new
connection starts without creating a thread. At most \fImax_idle\fR threads
wait, each at most \fIidle_seconds\fR (60 by default; 0 waits without a limit).
By default up to 32 threads wait for each running service, but no more than
the services' maxchild together. \fBthreadpool 0\fR ends every thread with its
the services' maxchild together. Each waiting thread keeps its stack and
its connection's structure, about 60 KB with the buffers. Other defaults can
be given at build time: -DDEFAULT_POOLMAX=\fIn\fR (-1 is the automatic
cap), -DPOOLPERSERVICE=\fIn\fR (the 32), -DDEFAULT_POOLTIMEOUT=\fIseconds\fR
and, for maxconn, -DDEFAULT_MAXCHILD=\fIn\fR (500). \fBthreadpool 0\fR ends every thread with its
connection. A thread serves a service only if its stack is at least as large as
the service needs (see \fBstacksize\fR), otherwise a new one is created.
The admin service shows how many threads wait, how many were created and how

View File

@ -538,12 +538,17 @@ int WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPWSTR lpCmdLine, int
{ (LPSTR)conf->stringtable[1], (LPSERVICE_MAIN_FUNCTION)ServiceMain},
{ NULL, NULL }
};
if(!StartServiceCtrlDispatcher( ste ))cyclestep();
if(!StartServiceCtrlDispatcher( ste )){
cyclestep();
logfinish();
}
}
else
#endif
{
cyclestep();
/* what is queued to the logs written */
logfinish();
}
@ -552,6 +557,8 @@ int WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPWSTR lpCmdLine, int
signal(SIGTERM, mysigterm);
signal(SIGUSR1, mysigusr1);
cyclestep();
/* what is queued to the logs written */
logfinish();
#endif

View File

@ -401,6 +401,18 @@ static int h_log(int argc, unsigned char ** argv){
return 0;
}
static int h_minlogqueue(int argc, unsigned char **argv){
conf->minlogqueue = atoi((char *)argv[1]);
if(conf->minlogqueue < 0) conf->minlogqueue = 0;
return 0;
}
static int h_maxlogqueue(int argc, unsigned char **argv){
conf->maxlogqueue = atoi((char *)argv[1]);
if(conf->maxlogqueue < 1) conf->maxlogqueue = 1;
return 0;
}
static int h_stacksize(int argc, unsigned char **argv){
conf->stacksize = atoi((char *)argv[1]);
return 0;
@ -2114,6 +2126,8 @@ struct commands commandhandlers[]={
{NULL, "delimchar",h_delimchar, 2, 2},
{NULL, "authnserver", h_authnserver, 2, 2},
{NULL, "stacksize", h_stacksize, 2, 2},
{NULL, "minlogqueue", h_minlogqueue, 2, 2},
{NULL, "maxlogqueue", h_maxlogqueue, 2, 2},
{NULL, "threadpool", h_threadpool, 2, 3},
{NULL, "force", h_force, 1, 1},
{NULL, "noforce", h_noforce, 1, 1},

View File

@ -22,12 +22,14 @@
.trafcounter = NULL,
.services = NULL,
.stacksize = 0,
.poolmax = -1,
.pooltimeout = 60,
.poolmax = DEFAULT_POOLMAX,
.pooltimeout = DEFAULT_POOLTIMEOUT,
.counterd = -1,
.rotate = 0,
.paused = 0,
.archiverc = 0,
.minlogqueue = LOGQUEUEMIN,
.maxlogqueue = LOGQUEUEMAX,
.demon = 0,
.maxchild = DEFAULT_MAXCHILD,
.backlog = 0,

777
src/log.c

File diff suppressed because it is too large Load Diff

View File

@ -159,14 +159,14 @@ static void freeworker(struct clientparam *param){
free(param);
}
/* The automatic cap: 32 idle workers for each running service, but never
more than all of them could have connections. */
/* The automatic cap: POOLPERSERVICE idle workers for each running service,
but never more than all of them could have connections. */
static unsigned poollimit(void){
unsigned limit;
if(runconf->poolmax >= 0) limit = (unsigned)runconf->poolmax;
else {
limit = 32 * tpool.services;
limit = POOLPERSERVICE * tpool.services;
if(limit > tpool.maxchild) limit = tpool.maxchild;
}
tpool.limit = limit;

View File

@ -145,13 +145,28 @@ void setopts(SOCKET s, const struct sockoptset *opts, int maxseg){
}
}
/* Releases what the preparation and the run took, s itself excepted. */
static void srvdiscard(struct srvstart *s){
srvfree(&s->srv);
static void srvstrings(struct srvstart *s){
if(s->cbc_string) free(s->cbc_string);
if(s->cbl_string) free(s->cbl_string);
}
/* Releases what the preparation and the run took, s itself excepted. */
static void srvdiscard(struct srvstart *s){
srvfree(&s->srv);
srvstrings(s);
}
#ifndef STDMAIN
/* A service which ran, released once its lines are written, s with it. */
static void srvlater(void *p){
struct srvstart *s = (struct srvstart *)p;
srvrelease(&s->srv);
srvstrings(s);
free(s);
}
#endif
#ifndef STDMAIN
static void srvunlink(struct srvparam *srv){
if(srv->next)srv->next->prev = srv->prev;
@ -673,11 +688,9 @@ int mainprepare(int argc, char** argv, void **service){
return 0;
}
/* service is freed once its lines are written */
int mainrun(void *service){
int res = srvrun((struct srvstart *)service);
free(service);
return res;
return srvrun((struct srvstart *)service);
}
#endif
@ -782,7 +795,9 @@ void srvinit2(struct srvparam * srv, struct clientparam *param){
param->sinsr = srv->extsa;
}
void srvfree(struct srvparam * srv){
/* A service ended: its sockets closed, its clients gone. What its lines
refer to is kept for srvrelease(). */
void srvstop(struct srvparam * srv){
int i;
#ifndef STDMAIN
@ -810,6 +825,11 @@ void srvfree(struct srvparam * srv){
for(i = srv->maxchild; i; i--) _3proxy_sem_lock(srv->childslots);
_3proxy_mutex_lock(&srv->counter_mutex);
_3proxy_mutex_unlock(&srv->counter_mutex);
}
/* What a stopped service holds: once the lines logged for it, which refer
to it and to its configuration, are written. */
void srvrelease(struct srvparam * srv){
#ifndef STDMAIN
if(srv->filter){
while(srv->nfilters){
@ -861,6 +881,11 @@ void srvfree(struct srvparam * srv){
#endif
}
void srvfree(struct srvparam * srv){
srvstop(srv);
srvrelease(srv);
}
#ifndef STDMAIN
static void * itcopy (void * from, size_t size){
void * ret;

View File

@ -34,7 +34,22 @@
#define MAXUSERNAME 128
#define _PASSWORD_LEN 256
/* Defaults, -D at build time sets others: maxconn; threadpool, the idle
workers kept (-1 automatic: POOLPERSERVICE for each service, no more than
their maxconn together) and how long one waits, in seconds. Each idle
worker keeps its thread and its struct clientparam. */
#ifndef DEFAULT_MAXCHILD
#define DEFAULT_MAXCHILD 500
#endif
#ifndef DEFAULT_POOLMAX
#define DEFAULT_POOLMAX -1
#endif
#ifndef DEFAULT_POOLTIMEOUT
#define DEFAULT_POOLTIMEOUT 60
#endif
#ifndef POOLPERSERVICE
#define POOLPERSERVICE 32
#endif
/* How much of a connection's buffer is read into at once: all of it, or
less if the service was told so with -b. */
@ -342,6 +357,7 @@ void dolog(struct clientparam * param, const unsigned char *s);
int dobuf(struct clientparam * param, unsigned char * buf, const unsigned char *s, const unsigned char * doublec);
int dobuf2(struct clientparam * param, unsigned char * buf, const unsigned char *s, const unsigned char * doublec, struct tm* tm, char * format);
int logbuf(struct clientparam * param, unsigned char * buf, const unsigned char *s, const unsigned char * doublec);
int logbuft(struct clientparam * param, unsigned char * buf, const unsigned char *s, const unsigned char * doublec, time_t sec, unsigned msec);
void logdo(struct clientparam * param, const unsigned char *s);
void logerror(const unsigned char *s);
void loginit(void);
@ -355,6 +371,7 @@ void logstdclose(void);
size_t logselen(const unsigned char *selector);
void logtick(time_t t);
void logfinish(void);
void logcall(struct logger *lg, struct logevent *ev, void (*func)(void *), void *arg);
int timechanged (time_t oldtime, time_t newtime, ROTATION lt);
extern struct logger lognull;
#ifdef WITH_ODBC
@ -500,6 +517,8 @@ FILTER_ACTION handledatfltsrv(struct clientparam *param, unsigned char ** buf_p,
void srvinit(struct srvparam * srv, struct clientparam *param);
void srvinit2(struct srvparam * srv, struct clientparam *param);
void srvfree(struct srvparam * srv);
void srvstop(struct srvparam * srv);
void srvrelease(struct srvparam * srv);
unsigned char * dologname (unsigned char *buf, unsigned char *name, const unsigned char *ext, ROTATION lt, time_t t);
int readconfig(FILE * fp);
void initcommands(void);

View File

@ -50,6 +50,10 @@ struct udpstate;
Their functions of file scope are declared here, so that they can be in
any order. */
static void srvdiscard(struct srvstart *s);
static void srvstrings(struct srvstart *s);
#ifndef STDMAIN
static void srvlater(void *p);
#endif
static void srvunlink(struct srvparam *srv);
static int srvprepare(struct srvstart *s, int argc, char **argv);
static int samestr(const char *a, const char *b);
@ -163,12 +167,19 @@ EXIT:
#endif
if(!srv->silent) srv->logfunc(defparam, (unsigned char *)"Exiting thread");
/* srvfree() waits for every slot to come back, this one included */
/* srvstop() waits for every slot to come back, this one included */
if(haveslot) _3proxy_sem_unlock(srv->childslots);
srvdiscard(s);
srvstop(srv);
#ifndef _WIN32
pthread_attr_destroy(&s->pa);
#endif
#ifndef STDMAIN
/* freed, s with it, once its log has written the lines which refer to it */
logcall(srv->logger, &srv->logrelease, srvlater, s);
#else
srvrelease(srv);
srvstrings(s);
#endif
return res;
}

View File

@ -105,8 +105,8 @@ static int odbcinit(struct logger *lg){
return odbcconnect(lg)? 0 : -1;
}
static int odbcformat(struct logger *lg, struct clientparam *param, const unsigned char *s, unsigned char *buf){
return logbuf(param, buf, s, (unsigned char *)"\'");
static int odbcformat(struct logger *lg, struct clientparam *param, const unsigned char *s, time_t sec, unsigned msec, unsigned char *buf){
return logbuft(param, buf, s, (unsigned char *)"\'", sec, msec);
}
static int odbcwrite(struct logger *lg, struct clientparam *param, const unsigned char *buf, int len){

View File

@ -204,6 +204,14 @@ int
#define UDPBUFSIZE 16384
#define CLIENTBUFSIZE (16384 + 64)
#define CLIENTTMPBUFSIZE 8192
/* The defaults of minlogqueue and maxlogqueue: the lines a log's queue
keeps slots for, and holds at most; -D at build time sets others. */
#ifndef LOGQUEUEMIN
#define LOGQUEUEMIN 64
#endif
#ifndef LOGQUEUEMAX
#define LOGQUEUEMAX 4096
#endif
#define HOSTNAMELEN 256
#define USERNAMELEN 128
@ -638,20 +646,35 @@ struct logtype {
/* opens what it writes to, lg->data: 0, an error (an errno value), or
< 0 if it cannot be reached now but is to be tried again */
int (*init)(struct logger *lg);
/* the line for a connection into buf, CLIENTTMPBUFSIZE: its length, -1
for nothing to write; NULL if write() takes the connection itself */
int (*format)(struct logger *lg, struct clientparam *param, const unsigned char *s, unsigned char *buf);
/* the line for a connection, logged at sec and msec, into buf,
CLIENTTMPBUFSIZE: its length, -1 for nothing to write; NULL if
write() takes the connection itself */
int (*format)(struct logger *lg, struct clientparam *param, const unsigned char *s, time_t sec, unsigned msec, unsigned char *buf);
/* 0, or an error */
int (*write)(struct logger *lg, struct clientparam *param, const unsigned char *buf, int len);
void (*close)(struct logger *lg);
/* every second or so: rotation, flushing */
void (*tick)(struct logger *lg, time_t t);
int stacksize; /* the client thread stack its writing needs */
int stacksize; /* the stack its writing needs */
int standard; /* kept open for the life of the process */
int text; /* writes a line of text: errors can go there */
int params; /* the string after a comma is parameters */
};
/* What a log's queue holds: a line (log.c's struct logslot, of which this
is the head), or something to do once the lines before it are written,
such as freeing a service they refer to. */
struct logevent {
struct logevent *next;
int kind;
unsigned dropped; /* lines dropped after this one, the queue full */
void (*func)(void *arg);
void *arg;
};
/* Once 3proxy runs, a logger has a thread of its own, which writes what its
queue holds: the clients do not wait for a log. Until then, and in a
standalone service, lines are written as they come. */
struct logger {
struct logger *next;
const struct logtype *type;
@ -660,20 +683,38 @@ struct logger {
void *data; /* the type's own */
int refs;
int errors; /* writes failed in a row */
int failed; /* could not be reopened: lines go to standard error */
int failed; /* could not be reopened: its lines are dropped */
time_t failtime;
unsigned faildropped; /* lines dropped as it failed, not yet told */
ROTATION rotation;
int rotate, archiverc;
unsigned char **archiver;
time_t rotated; /* when the file in use was opened */
int reopen; /* the rotation changed: the file name too */
_3proxy_mutex_t mutex; /* writing, reopening */
_3proxy_mutex_t mutex; /* the queue; writing, while there is no thread */
/* the queue: the lines in it counted against maxlogqueue, the slots
allocated kept down to minlogqueue as they come back */
struct logevent *head, *tail;
void *free;
int lines, slots;
_3proxy_sem_t wake;
int running, /* has its thread */
stop, /* its thread is to end once the queue is empty */
more; /* for its thread: lines follow the one written */
/* given while its thread runs, for the thread to take */
int newrot, newparams;
ROTATION nrotation;
int nrotate, narchiverc;
unsigned char **narchiver;
unsigned char *nparams;
};
struct srvparam {
struct sockfuncs so;
struct srvparam *next;
struct srvparam *prev;
/* freeing it, queued to its log behind the lines which refer to it */
struct logevent logrelease;
struct listener *listener; /* its listening socket, which a newer
service may take over on a reload */
struct srvparam *lisnext; /* the next service on the listener */
@ -941,6 +982,7 @@ struct extparam {
struct srvparam *services;
int stacksize,
counterd, rotate, paused, archiverc,
minlogqueue, maxlogqueue,
demon, maxchild, backlog, version, noforce, bandlimver, parentretries;
unsigned authcachetype, authcachetime;
int filtermaxsize;

View File

@ -44,6 +44,81 @@ def _opened(pid, path):
return n
def _stalled(t):
"""A log nobody reads: a FIFO whose pipe is full. Clients are not held
up, the lines past the queue are dropped and counted, and lines queued
for a service a reload removed are still written as it formatted them."""
fifo = os.path.join(t.tmpdir, "logs-fifo")
os.mkfifo(fifo)
# opened for reading before 3proxy opens it for writing, then not read
rfd = os.open(fifo, os.O_RDONLY | os.O_NONBLOCK)
origin, port, adm = t.free_port(), t.free_port(), t.free_port()
# lines of 2 KB, made so by the service's format: the pipe's 64 KB fill
# before the requests are done
pad = "z" * 2000
def conf(proxyline):
return f"""
log
maxlogqueue 8
auth iponly
allow *
http data * /data
httpsrv -p{origin}
flush
admin -p{adm}
flush
logformat "LQ {pad} %T"
auth iponly
allow *
{proxyline}
"""
server = t.start("logs_stall", conf(f'proxy -p{port} -l{fifo}'),
ports=[origin, port, adm])
began = time.time()
ok = 0
for i in range(120):
r = t.http(f"http://127.0.0.1:{origin}/data?size=10&n{i:04d}",
proxy=f"127.0.0.1:{port}")
ok += r.status == 200
t.eq(120, ok, "a log nobody reads holds no client up")
t.eq(True, time.time() - began < 60, "and they are not slowed to its pace")
# the service goes: its lines still queued are written as it made them
t.write_config("logs_stall", conf("flush"))
t.http(f"http://127.0.0.1:{adm}/R")
got = b""
deadline = time.time() + 15
while time.time() < deadline and "has its queue full" not in server.output():
try:
chunk = os.read(rfd, 65536)
except BlockingIOError:
chunk = b""
got += chunk
if not chunk:
time.sleep(0.05)
t.eq(True, "has its queue full" in server.output(),
"the lines past its queue are dropped, and said so")
# what is left in the pipe and the queue, read to the end
deadline = time.time() + 5
while time.time() < deadline:
try:
chunk = os.read(rfd, 65536)
except BlockingIOError:
chunk = b""
if chunk:
got += chunk
deadline = time.time() + 1
else:
time.sleep(0.05)
os.close(rfd)
lines = got.decode("utf-8", "replace").splitlines()
t.eq(True, 0 < len(lines) < 120, "some lines are written, not all")
t.eq(True, all(l.startswith("Q " + pad) for l in lines),
"each as its service formats it, the service gone or not")
def config(origin, p1, p2, p3, adm, main, shared=None):
"""p2 and p3 log to shared with -l, if it is given, or are not there."""
lines = [f"log {main}", 'logformat "LLOG %N %T"', "auth iponly", "allow *",
@ -108,7 +183,7 @@ def run(t):
t.eq(True, _wait(lambda: _proxied(dated, "five")), "goes to the dated file")
t.eq(False, _proxied(second, "five"), "not to the undated one")
# --- a log which fails ------------------------------------------------
# --- a log which fails: its lines dropped, and said so -----------------
if os.path.exists("/dev/full"):
o2, q = t.free_port(), t.free_port()
full = t.start("logs_full", f"""
@ -124,8 +199,14 @@ def run(t):
for i in range(80):
t.http(f"http://127.0.0.1:{o2}/data?size=10&{'x' * 40}{i}",
proxy=f"127.0.0.1:{q}")
t.eq(True, _wait(lambda: "LOG PROXY" in full.output()),
"the lines a full device refuses go to standard error")
t.eq(True, _wait(lambda: "Log /dev/full could not be written" in full.output()),
"a log a full device refuses says so on standard error")
t.eq(True, "records dropped" in full.output(), "with the records it dropped")
t.eq(False, "LOG PROXY" in full.output(), "its lines are not written elsewhere")
# --- a log which stalls: clients go on, the queue is bounded ----------
if hasattr(os, "mkfifo"):
_stalled(t)
# --- a log which cannot be opened is an error in the configuration ---
out = t.run_config("logs_bad", f"""