bitdeals
88b5b896e1
Build docker image and push to registry.bitdeals.org / main-build-job (push) Successful in 7m12s
A PyBitmessage daemon that has dropped to zero network connections does not find its way back. It can sit there for days — testnet1 did, and every BitDeals deal completed on that stand since 2026-08-30 left its escrow unspent, because the guarantor never received a CH3 the node could no longer publish. A daemon that has just started, by contrast, dials hard and reconnects within seconds. The cure was already known; what was missing was anything to notice and apply it. Docker will not: it reacts to a process exiting, never to a healthcheck, and health-driven restarts exist only under Swarm. Doing it from outside means handing a container the docker socket, which is root on the host and a poor trade for a relay with a published port. So the container supervises itself. `run.sh` no longer ends at `exec gosu bitmessage pybitmessage -d`. That made the daemon PID 1, and it is a bad PID 1: daemonize() double-forks and parks the grandfather in `while True: time.sleep(1)`, the final child SIGTERMs it to say "ready", and PID 1 drops that signal for want of a handler. The grandfather slept for ever; `docker stop` therefore reached the real daemon only as the SIGKILL ten seconds later, cutting a startup VACUUM in half — the way a node gets trapped retrying one it can never finish; and a daemon that died on its own left the container Up around a corpse, because what PID 1 was doing had nothing to do with whether the daemon lived. Away from PID 1 that grandfather does die on the ready signal — measured here: the start call returns at once with status 143 and leaves exactly one pybitmessage process behind. That makes starting the daemon an ordinary blocking call, and the supervisor ordinary shell: a trap that stops the daemon through its own API, a restart when it is gone, and the peer rule. What the supervisor does not do is act on a daemon whose API is not answering at all. That is the trapped-VACUUM node; a restart does not cure it and cuts the next VACUUM short as well. watchdog.py reports that case as its own exit code so the loop can leave it to a person. The healthcheck is untouched: it reports, and does not act. Four settings, on by default: BITMESSAGE_WATCHDOG, and _PERIOD, _AFTER, _COOLDOWN. All validated at start, where a typo is visible, rather than hours later as a supervisor that spins or one that never acts. Callers must raise the stop grace period — 90s in compose, or --stop-timeout 90. A clean shutdown took 17.5 s on a small database and grows with it, so under Docker's default ten the daemon is killed mid-write anyway and the supervisor buys nothing. An image cannot set this for itself.
76 lines
2.6 KiB
Python
76 lines
2.6 KiB
Python
#!/usr/bin/python
|
|
|
|
"""Two errands the supervisor in run.sh cannot do in shell: ask, and stop.
|
|
|
|
`peers` reports how many network connections the daemon has, as an exit code the
|
|
shell can branch on. `shutdown` asks the daemon to stop through its own API,
|
|
which runs doCleanShutdown -- the database is closed rather than cut off.
|
|
|
|
The streak, the cooldown and the restart itself are deliberately not here. This
|
|
process is short-lived, so a counter would need a file to live in, and then two
|
|
places would decide one thing. run.sh counts; this only reports.
|
|
|
|
Exit codes are the interface:
|
|
|
|
0 the daemon answered and has peers
|
|
1 the daemon answered and has none (`peers`)
|
|
or the shutdown was asked for (`shutdown`)
|
|
2 the daemon did not answer at all
|
|
|
|
2 is not 1 on purpose. A daemon whose API is refusing connections is the trapped
|
|
node of the startup-VACUUM failure, and restarting it neither cures that nor is
|
|
harmless: it drops the next VACUUM half-done as well. The supervisor must be
|
|
able to tell "no peers" from "no answer", and only act on the first.
|
|
"""
|
|
|
|
import json
|
|
import os
|
|
import sys
|
|
import urllib
|
|
import xmlrpclib
|
|
|
|
# Credentials go into a URL, so they must be percent-encoded: '@' splits the
|
|
# userinfo, '#' truncates the rest, '/' and ':' change what is parsed as host
|
|
# and port. healthy_check.py carries the same six lines for the same reason --
|
|
# these are one-shot scripts with no package around them to share a module.
|
|
API_USER = os.getenv('BITMESSAGE_API_USER', 'bitmessage_api_user')
|
|
API_PASSWORD = os.getenv('BITMESSAGE_API_PASSWORD', 'bitmessage_api_password')
|
|
API_PORT = os.getenv('BITMESSAGE_API_PORT', '8442')
|
|
|
|
API_LINK = "http://{}:{}@127.0.0.1:{}/".format(
|
|
urllib.quote(API_USER, safe=''), urllib.quote(API_PASSWORD, safe=''), API_PORT)
|
|
|
|
NO_ANSWER = 2
|
|
|
|
|
|
def peers(api):
|
|
"""How many peers the daemon holds, printed for the container log."""
|
|
count = json.loads(api.clientStatus())['networkConnections']
|
|
print "networkConnections:", count
|
|
return 0 if count > 0 else 1
|
|
|
|
|
|
def shutdown(api):
|
|
"""Ask the daemon to stop itself cleanly (api.py: HandleShutdown)."""
|
|
api.shutdown()
|
|
print "shutdown requested"
|
|
return 1
|
|
|
|
|
|
COMMANDS = {'peers': peers, 'shutdown': shutdown}
|
|
|
|
|
|
def main(argv):
|
|
if len(argv) != 2 or argv[1] not in COMMANDS:
|
|
print >> sys.stderr, "usage: watchdog.py peers|shutdown"
|
|
return NO_ANSWER
|
|
try:
|
|
return COMMANDS[argv[1]](xmlrpclib.ServerProxy(API_LINK))
|
|
except Exception as exc: # the API is down, or answering something else
|
|
print "no answer from the daemon:", exc
|
|
return NO_ANSWER
|
|
|
|
|
|
if __name__ == '__main__':
|
|
sys.exit(main(sys.argv))
|