feat: supervise the daemon, and restart one that has lost every peer
Build docker image and push to registry.bitdeals.org / main-build-job (push) Successful in 7m12s

A PyBitmessage daemon that has dropped to zero network connections does not find
its way back. It can sit there for days — testnet1 did, and every BitDeals deal
completed on that stand since 2026-08-30 left its escrow unspent, because the
guarantor never received a CH3 the node could no longer publish. A daemon that
has just started, by contrast, dials hard and reconnects within seconds. The
cure was already known; what was missing was anything to notice and apply it.

Docker will not: it reacts to a process exiting, never to a healthcheck, and
health-driven restarts exist only under Swarm. Doing it from outside means
handing a container the docker socket, which is root on the host and a poor
trade for a relay with a published port. So the container supervises itself.

`run.sh` no longer ends at `exec gosu bitmessage pybitmessage -d`. That made the
daemon PID 1, and it is a bad PID 1: daemonize() double-forks and parks the
grandfather in `while True: time.sleep(1)`, the final child SIGTERMs it to say
"ready", and PID 1 drops that signal for want of a handler. The grandfather slept
for ever; `docker stop` therefore reached the real daemon only as the SIGKILL ten
seconds later, cutting a startup VACUUM in half — the way a node gets trapped
retrying one it can never finish; and a daemon that died on its own left the
container Up around a corpse, because what PID 1 was doing had nothing to do with
whether the daemon lived.

Away from PID 1 that grandfather does die on the ready signal — measured here:
the start call returns at once with status 143 and leaves exactly one
pybitmessage process behind. That makes starting the daemon an ordinary blocking
call, and the supervisor ordinary shell: a trap that stops the daemon through its
own API, a restart when it is gone, and the peer rule.

What the supervisor does not do is act on a daemon whose API is not answering at
all. That is the trapped-VACUUM node; a restart does not cure it and cuts the
next VACUUM short as well. watchdog.py reports that case as its own exit code so
the loop can leave it to a person. The healthcheck is untouched: it reports, and
does not act.

Four settings, on by default: BITMESSAGE_WATCHDOG, and _PERIOD, _AFTER,
_COOLDOWN. All validated at start, where a typo is visible, rather than hours
later as a supervisor that spins or one that never acts.

Callers must raise the stop grace period — 90s in compose, or --stop-timeout 90.
A clean shutdown took 17.5 s on a small database and grows with it, so under
Docker's default ten the daemon is killed mid-write anyway and the supervisor
buys nothing. An image cannot set this for itself.
This commit is contained in:
2026-09-04 09:53:07 +00:00
parent 58e3f22982
commit 88b5b896e1
5 changed files with 309 additions and 1 deletions
+31
View File
@@ -84,6 +84,10 @@ Container images are configured using parameters passed at runtime.
|-e BITMESSAGE_TRUSTED_PEER|`host:port` of the one peer this node may connect out to; it dials nothing else. Default: empty — the node chooses its own peers. See Notes|
|-e BITMESSAGE_SEND_OUTGOING|Whether the node dials out at all, `True` or `False`. Default: `True`. `False` gives a node that only accepts inbound connections — the hub of a private contour|
|-e BITMESSAGE_KNOWN_NODES|Comma-separated `host:port` list, written into `knownnodes.dat` on every start in place of whatever was there. Default: empty — the file is left as it is. Also switches the DNS bootstrap off, see Notes|
|-e BITMESSAGE_WATCHDOG|Whether to restart a daemon that has lost every peer, `True` or `False`. Default: `True`. See Notes|
|-e BITMESSAGE_WATCHDOG_PERIOD|Seconds between peer checks. Default: `60`|
|-e BITMESSAGE_WATCHDOG_AFTER|How many peerless checks in a row it takes to act. Default: `5`, so five minutes at the default period|
|-e BITMESSAGE_WATCHDOG_COOLDOWN|Floor between two restarts, in seconds. Default: `900`. `0` removes it|
# Notes
@@ -96,6 +100,33 @@ Container images are configured using parameters passed at runtime.
- The container turns healthy once the daemon has a network connection, which
on a new node takes a few minutes. The start period also covers the startup
`VACUUM` of `messages.dat`.
- **A peerless daemon is restarted; a silent one is not.** A daemon that has
lost every peer does not find its way back — it can sit at zero connections
for as long as you leave it — while one that has just started dials hard and
does. So the container supervises its own daemon: it asks the API for the
connection count every `BITMESSAGE_WATCHDOG_PERIOD`, and after
`BITMESSAGE_WATCHDOG_AFTER` answers of zero in a row it stops the daemon
through its own API and starts it again. The *container* is not restarted and
nothing outside it is involved, so a node that depends on this one keeps its
addresses and only sees the API blink.
A daemon whose API does not answer at all is left alone, deliberately. That is
the node trapped retrying a startup `VACUUM` it cannot finish, restarting does
not cure it, and restarting anyway cuts the next `VACUUM` in half too. The
healthcheck reports both cases as unhealthy; only one of them is something to
do anything about, and a person has to look at the other.
**Give the container `stop_grace_period: 90s`** (compose) or
`--stop-timeout 90` (`docker run`). Closing `messages.dat` properly takes
longer than Docker's default ten seconds — measured at about thirteen on a
small database, and it grows with the file — so without this the daemon is
SIGKILLed mid-write, which is the very thing the supervisor is there to
prevent. An image cannot set this for itself; only the caller can.
`BITMESSAGE_WATCHDOG=False` turns the peer rule off. The supervisor stays
either way — it is also what stops the daemon cleanly on `docker stop`, and
what restarts one that died outright rather than leaving the container up
around a corpse.
- **The P2P port must be published as `8444:8444`.** The daemon tells peers the
port from its own config (`port` in `keys.dat`), not the port you mapped it
to, so `8555:8444` advertises a port nobody can reach. A different host port