escape the sed replacements, and stop the script failing silently

The sed pass that configures the daemon punished anyone setting a strong
password: '&' in a replacement means the whole match, so a password
containing one was silently rewritten into something else, and the '|'
delimiter made sed exit with 'unknown option to s'. Every setting rode in
one invocation and the script had no set -e, so that failure applied none
of them -- not apipassword, not apienabled, not apiinterface -- and the
daemon came up on whatever it had before without saying so.

esc() now escapes backslash, '&' and the delimiter via printf. The
expressions are anchored to the start of the line and name their key in the
replacement instead of using \1, so no backreference is involved and
nothing in another section can match. keys.dat also holds privsigningkey
and privencryptionkey for this node's Bitmessage identities; editing in
place leaves every byte outside [bitmessagesettings] untouched, which is
why this stays a line edit rather than a parse-and-rewrite.

The clients needed the other half of this: credentials go into an XML-RPC
URL, where '@' splits the userinfo and '#' truncates the rest, so a strong
password wrote correctly and still failed to connect. Both now
percent-encode. A '%' in the password remains unusable -- it breaks
PyBitmessage's own config reader and every API call returns 500.

Also here, all found while making the above safe:
- set -eu, with the keys.dat chown guarded. A misconfiguration now stops
  the container instead of passing unnoticed.
- The seed-address retry loop used bash brace expansion under CMD ["sh"],
  where /bin/sh is dash and {1..4} is a literal, so it ran once, not four
  times.
- apt-get update shared a layer with nothing, letting a cached update feed
  install months-stale package lists.
- HEALTHCHECK gained a start period; the startup VACUUM takes tens of
  seconds on a large messages.dat and the container reported unhealthy for
  all of it.
- Removed the AppImage systemd unit, AppArmor profile and updater script.
  Nothing referenced them -- not the image, the deployment or the ansible
  roles -- and the updater fetched a binary with no signature or checksum
  check.

README.md

@@ -4,6 +4,12 @@
 
 PyBitmessage client running as a daemon in docker container with XML-RPC API enabled.
 
+This repository covers the docker deployment only. It used to also carry an
+AppImage systemd unit, an AppArmor profile and an updater script; none of them
+were referenced by the image, the deployment or the ansible roles, and the
+updater fetched a binary with no signature or checksum check, so they were
+removed rather than left looking usable.
+
 # Usage
 
 The container generates a Bitmessage Deterministic Addresses based on a `BITMESSAGE_SEED_PHRASE` variable.

docker/Dockerfile

@@ -3,10 +3,11 @@ FROM ubuntu:bionic
 
 SHELL ["/bin/bash", "-exo", "pipefail", "-c"]
 
-RUN apt-get update
+# Install dependencies. update and install share a layer on purpose: split
-
+# across two, a cached update feeds install package lists that may be months
-# Install dependencies
+# stale, and the install then fails or pulls something unintended.
-RUN apt-get install -yq --no-install-suggests --no-install-recommends \
+RUN apt-get update \
+ && apt-get install -yq --no-install-suggests --no-install-recommends \
     build-essential libcap-dev libssl-dev \
     python-all-dev python-msgpack python-pip python-setuptools \
     git
@@ -72,7 +73,10 @@ RUN su bitmessage -c "pybitmessage -t"
 
 CMD ["sh", "/usr/local/bin/run.sh"]
 
-## Check PyBitmessage active network connections
+## Check PyBitmessage active network connections.
-HEALTHCHECK --retries=0 --interval=15s \
+## The start period covers the startup VACUUM of messages.dat, which takes tens
+## of seconds once the database reaches a few hundred MB; without it the
+## container reports unhealthy for that whole window on every restart.
+HEALTHCHECK --retries=0 --interval=15s --start-period=180s \
   CMD ["python", "/usr/local/bin/healthy_check.py"]
 

docker/healthy_check.py

@@ -2,6 +2,7 @@
 
 import sys
 import os
+import urllib
 import xmlrpclib
 import json
 
@@ -9,7 +10,13 @@ api_user=os.getenv('BITMESSAGE_API_USER', 'bitmessage_api_user') ;
 api_password=os.getenv('BITMESSAGE_API_PASSWORD', 'bitmessage_api_password') ;
 api_port=os.getenv('BITMESSAGE_API_PORT', '8442') ;
 
-api_link="http://{}:{}@127.0.0.1:{}/".format(api_user, api_password, api_port)
+# Credentials go into a URL, so they must be percent-encoded: '@' splits the
+# userinfo, '#' truncates the rest, '/' and ':' change what is parsed as host
+# and port. Without this a strong password fails here while being perfectly
+# valid in keys.dat. (A '%' in the password is a separate matter -- it breaks
+# PyBitmessage's own config reader and makes every API call return 500.)
+api_link="http://{}:{}@127.0.0.1:{}/".format(
+    urllib.quote(api_user, safe=''), urllib.quote(api_password, safe=''), api_port)
 
 api = xmlrpclib.ServerProxy(api_link)
 

docker/run.sh

@@ -1,5 +1,7 @@
 #!/bin/sh
 
+set -eu
+
 export BITMESSAGE_API_USER="${BITMESSAGE_API_USER:-bitmessage_api_user}"
 export BITMESSAGE_API_PASSWORD="${BITMESSAGE_API_PASSWORD:-bitmessage_api_password}"
 export BITMESSAGE_SEED_ADDRESSES="${BITMESSAGE_SEED_ADDRESSES:-0}"
@@ -8,33 +10,56 @@ export BITMESSAGE_TTL="${BITMESSAGE_TTL:-172800}"
 export BITMESSAGE_STOPRESENDINGAFTERXDAYS="${BITMESSAGE_STOPRESENDINGAFTERXDAYS:-30}"
 export BITMESSAGE_APIVARIANT="${BITMESSAGE_APIVARIANT:-legacy}"
 
-if [ -z "$BITMESSAGE_SEED_PHRASE" ]
+if [ -z "${BITMESSAGE_SEED_PHRASE:-}" ]
 then
   BITMESSAGE_SEED_PHRASE="$(cat /dev/random | tr -dc "a-z" | head -c32)"
   export BITMESSAGE_SEED_PHRASE
 fi
 
-# this command must be run as root (for bind mounts to container)
+# Escape a value for use on the right-hand side of the sed expressions below.
-chown bitmessage:bitmessage keys.dat
+# There, a backslash starts an escape, "&" stands for the whole match, and "|"
-chmod 600 keys.dat
+# ends the replacement because it is the delimiter. Unescaped, a password
+# containing "&" was silently rewritten into something else and one containing
+# "|" made sed fail outright.
+esc() {
+	printf '%s' "$1" | sed -e 's/[\\&|]/\\&/g'
+}
 
-# set config values
+# this command must be run as root (for bind mounts to container)
-gosu bitmessage sed -i -e "s|\(apiinterface = \).*|\10\.0\.0\.0|g" \
+if [ -f keys.dat ]
-	-e "s|\(apivariant = \).*|\1$BITMESSAGE_APIVARIANT|g" \
+then
-	-e "s|\(apiusername = \).*|\1$BITMESSAGE_API_USER|g" \
+	chown bitmessage:bitmessage keys.dat
-	-e "s|\(apipassword = \).*|\1$BITMESSAGE_API_PASSWORD|g" \
+	chmod 600 keys.dat
-	-e "s|\(apiport = \).*|\1$BITMESSAGE_API_PORT|g" \
+fi
-	-e "s|\(apienabled = \).*|\1True|g" \
+
-	-e "s|\(ttl = \).*|\1$BITMESSAGE_TTL|g" \
+# Set config values. Every expression is anchored to the start of the line and
-	-e "s|\(stopresendingafterxdays = \).*|\1$BITMESSAGE_STOPRESENDINGAFTERXDAYS|g" \
+# names its key in the replacement, so no backreference is involved and nothing
-	-e "s|\(udp = \).*|\1False|g" keys.dat
+# in another section can match. With set -e a failure here now stops the
+# container instead of leaving the daemon on its previous settings unnoticed --
+# including the case of a bind mount with no keys.dat at all.
+gosu bitmessage sed -i \
+	-e "s|^apiinterface = .*|apiinterface = 0.0.0.0|" \
+	-e "s|^apivariant = .*|apivariant = $(esc "$BITMESSAGE_APIVARIANT")|" \
+	-e "s|^apiusername = .*|apiusername = $(esc "$BITMESSAGE_API_USER")|" \
+	-e "s|^apipassword = .*|apipassword = $(esc "$BITMESSAGE_API_PASSWORD")|" \
+	-e "s|^apiport = .*|apiport = $(esc "$BITMESSAGE_API_PORT")|" \
+	-e "s|^apienabled = .*|apienabled = True|" \
+	-e "s|^ttl = .*|ttl = $(esc "$BITMESSAGE_TTL")|" \
+	-e "s|^stopresendingafterxdays = .*|stopresendingafterxdays = $(esc "$BITMESSAGE_STOPRESENDINGAFTERXDAYS")|" \
+	-e "s|^udp = .*|udp = False|" keys.dat
 
 # generate address from seed
-test "$BITMESSAGE_SEED_ADDRESSES" -gt 0 && for i in {1..4}
+if [ "$BITMESSAGE_SEED_ADDRESSES" -gt 0 ]
-do
+then
+	# Four attempts, not a bash {1..4}: this runs under dash, where brace
+	# expansion is literal and the loop would have run once. The call is
+	# idempotent (createDeterministicAddresses returns nothing for an address
+	# that already exists), so these are retries while the API comes up.
+	for i in 1 2 3 4
+	do
 		sleep 15
 		gosu bitmessage /usr/bin/python /usr/local/bin/seed_addr_gen.py
-done &
+	done &
+fi
 
 exec gosu bitmessage pybitmessage -d
-

docker/seed_addr_gen.py

@@ -1,6 +1,7 @@
 #!/usr/bin/python
 
 import os
+import urllib
 import xmlrpclib
 
 api_user=os.getenv('BITMESSAGE_API_USER', 'bitmessage_api_user') ;
@@ -9,7 +10,10 @@ api_port=os.getenv('BITMESSAGE_API_PORT', '8442') ;
 addr_num=os.getenv('BITMESSAGE_SEED_ADDRESSES', '0') ;
 addr_seed=os.getenv('BITMESSAGE_SEED_PHRASE') ;
 
-api_link="http://{}:{}@127.0.0.1:{}/".format(api_user, api_password, api_port)
+# Percent-encode the credentials before they go into the URL -- see the same
+# note in healthy_check.py.
+api_link="http://{}:{}@127.0.0.1:{}/".format(
+    urllib.quote(api_user, safe=''), urllib.quote(api_password, safe=''), api_port)
 
 api = xmlrpclib.ServerProxy(api_link)
 

etc/apparmor.d/PyBitmessage.AppImage

@@ -1,94 +0,0 @@
-# Last Modified: Fri Oct 13 05:01:46 2023
-include <tunables/global>
-
-/**/PyBitmessage*.AppImage {
-  include <abstractions/apparmor_api/find_mountpoint>
-  include <abstractions/base>
-  include <abstractions/fonts>
-  include <abstractions/openssl>
-  include <abstractions/user-tmp>
-
-  capability dac_read_search,
-  capability sys_admin,
-
-  network inet dgram,
-  network inet stream,
-  network inet6 dgram,
-  network inet6 stream,
-  network netlink raw,
-
-  mount fstype=fuse.PyBitmessage*.AppImage options=(ro, nosuid, nodev),
-  umount,
-
-  /dev/fuse rw,
-  /etc/fuse.conf r,
-  /etc/gai.conf r,
-  /etc/host.conf r,
-  /etc/hosts r,
-  /etc/nsswitch.conf r,
-  /etc/python2.7/sitecustomize.py r,
-  /etc/resolv.conf r,
-  /etc/xdg/Trolltech.conf rk,
-  /proc/filesystems r,
-  /sys/devices/system/cpu/online r,
-
-  /tmp/*/*/.mount_PyBitm*/ r,
-  /tmp/*/*/.mount_PyBitm*/** r,
-  /tmp/*/*/.mount_PyBitm*/AppRun mrix,
-  /tmp/*/*/.mount_PyBitm*/lib/x86_64-linux-gnu/lib*.so* mr,
-  /tmp/*/*/.mount_PyBitm*/usr/bin/pybitmessage mrix,
-  /tmp/*/*/.mount_PyBitm*/usr/bin/qt.conf mrk,
-  /tmp/*/*/.mount_PyBitm*/usr/lib/python2.7/**.so mr,
-  /tmp/*/*/.mount_PyBitm*/usr/bin/python2.7 rix,
-  /tmp/*/*/.mount_PyBitm*/usr/lib/x86_64-linux-gnu/**/lib*.so* mr,
-  /tmp/*/*/.mount_PyBitm*/usr/lib/x86_64-linux-gnu/lib*.so* mr,
-  /tmp/*/*/.mount_PyBitm*/lib/x86_64/lib*.so mr,
-  /proc/*/cmdline r,
-  /usr/share/themes/** r,
-  owner /run/*/*/sni-qt_python2*/ rw,
-  owner /run/*/*/sni-qt_python2*/icons/ rw,
-
-  /tmp/.mount_PyBitm*/ r,
-  /tmp/.mount_PyBitm*/** r,
-  /tmp/.mount_PyBitm*/AppRun mrix,
-  /tmp/.mount_PyBitm*/lib/x86_64-linux-gnu/lib*.so* mr,
-  /tmp/.mount_PyBitm*/usr/bin/pybitmessage mrix,
-  /tmp/.mount_PyBitm*/usr/bin/qt.conf mrk,
-  /tmp/.mount_PyBitm*/usr/lib/python2.7/**.so mr,
-  /tmp/.mount_PyBitm*/usr/lib/x86_64-linux-gnu/**/lib*.so* mr,
-  /tmp/.mount_PyBitm*/usr/lib/x86_64-linux-gnu/lib*.so* mr,
-  /usr/bin/dash mrix,
-  /usr/bin/fusermount mrix,
-  /usr/bin/fusermount3 mrix,
-  /usr/bin/python2.7 r,
-  /usr/bin/stat mrix,
-  /usr/sbin/ldconfig mrix,
-  /usr/share/icons/ r,
-  /usr/share/icons/Adwaita/* r,
-  /usr/share/icons/Adwaita/** r,
-  /usr/share/icons/gnome/* r,
-  /usr/share/icons/hicolor/* rk,
-  /usr/share/mime/* r,
-  /usr/share/pixmaps/ r,
-  /usr/share/themes/Adwaita/** r,
-  owner /**/PyBitmessage*.AppImage mr,
-  owner /etc/passwd r,
-  owner @{HOME}/tmp* w,
-  owner /run/systemd/userdb/ r,
-  owner /run/*/*/sni-qt_python2_*/ rw,
-  owner /run/*/*/sni-qt_python2_*/icons/ rw,
-  owner /usr/local/share/fonts/** r,
-  owner @{HOME}/.cache/fontconfig/*-le64.cache-7 r,
-  owner @{HOME}/.config/PyBitmessage/ r,
-  owner @{HOME}/.config/PyBitmessage/debug.log w,
-  owner @{HOME}/.config/PyBitmessage/keys.dat rw,
-  owner @{HOME}/.config/PyBitmessage/keys.dat.*.bak w,
-  owner @{HOME}/.config/PyBitmessage/knownnodes.dat rw,
-  owner @{HOME}/.config/PyBitmessage/messages.dat rwk,
-  owner @{HOME}/.config/PyBitmessage/messages.dat-journal rw,
-  owner @{HOME}/.config/PyBitmessage/pybitmessageqt.conf rwk,
-  owner @{HOME}/.config/PyBitmessage/singleton.lock rwk,
-  owner @{HOME}/.config/Trolltech.conf rwk,
-
-}
-

etc/systemd/system/bitmessage-appimage.service

@@ -1,28 +0,0 @@
-[Unit]
-Description=Bitmessage appimage service
-Documentation=https://bitmessage.org/wiki/API_Reference
-ConditionPathExists=/home/bitmessage/PyBitmessage.AppImage
-After=network.target
-Wants=network.target
-
-[Service]
-Type=forking
-ExecStart=/home/bitmessage/PyBitmessage.AppImage --daemon
-WorkingDirectory=/home/bitmessage
-PIDFile=/home/bitmessage/.config/PyBitmessage/singleton.lock
-KillMode=process
-Restart=always
-RestartSec=10
-#Nice=19
-
-SyslogIdentifier=bitmessage
-User=bitmessage
-Group=bitmessage
-PrivateTmp=yes
-
-#ProtectHome=tmpfs
-#BindPaths=/home/bitmessage/.config/PyBitmessage/
-
-[Install]
-WantedBy=multi-user.target
-

usr/local/bin/bitmessage-updater.sh

@@ -1,35 +0,0 @@
-#!/bin/bash
-
-#
-#Be careful when downloading form https://artifacts.bitmessage.at. It's just a dump of binaries, it doesn't separate the official with development binaries.
-#
-#To find out which is the latest official one, go to github and click on the icon indicating the latest official build result.
-#
-#Peter Surda
-#Bitmessage developer
-#
-
-set -ex
-
-IMAGE_LIST="https://artifacts.bitmessage.at/appimage"
-LAST_IMAGE="$(wget -q $IMAGE_LIST -O- | grep -oP '[[:digit:]]{5}' | tail -n1)"
-
-IMAGE_FILENAME="$(wget -q $IMAGE_LIST/$LAST_IMAGE -O- | grep -oE '"PyBitmessage-.*-x86_64.AppImage"')"
-IMAGE_FILENAME=${IMAGE_FILENAME:1:-1}
-
-URL="$IMAGE_LIST/$LAST_IMAGE/$IMAGE_FILENAME"
-
-DEST_FILE="$HOME/$IMAGE_FILENAME"
-HARD_LINK="$HOME/PyBitmessage.AppImage"
-
-if ! [ -f $DEST_FILE ]; then
-	wget -c $URL -O $DEST_FILE
-	chmod u+x $DEST_FILE
-	cp -v -fl $DEST_FILE $HARD_LINK
-fi
-
-#remove old appimage files
-ls -1 -tc $HOME/PyBitmessage-.*-x86_64.AppImage 2>/dev/null \
-	| tail -n +3 \
-	| xargs -I{} rm -f $HOME/{}
-