fix: repair issuance, verify installation, reach HAProxy over a unix socket

Setting CERTBOT_EMAIL broke certificate issuance outright. CERTBOT_OPTS was
built as "--email $CERTBOT_EMAIL" and passed quoted, so certbot received the
flag and its value as one argument and rejected it as unrecognised; `set -e`
then ended the pass, and the loop repeated the same failure every 12 hours
while the site served the self-signed placeholder. The arguments are built
with `set --` now, which keeps them two words.

Installation into the running HAProxy reported success whatever happened. The
runtime API answers a refusal in the reply text and still closes cleanly, so
socat exits 0 either way; the script sent set, commit and show in a row and
inspected none of them. It now matches the replies and stops at the first that
is not an acknowledgement, saying plainly that the running HAProxy is still on
its previous certificate. It also reaches the API over a unix socket on a
volume shared with haproxy rather than TCP 9999, so the private key it carries
no longer crosses a network in the clear.

Also:

- The self-signed placeholder wrote site.key, site.csr and site.crt to the
  working directory (/opt/certbot) and left the private key there for good.
  It is built in a subshell under umask 077 on the volume the key belongs on,
  the CSR step is gone, and `>>` is replaced by a temporary file and an atomic
  rename — HAProxy reads site.pem at start-up and must never find it partial.
- 2-concatenate-cert.sh changed into the live directory without checking, and
  on failure looked for fullchain.pem in whatever directory the caller was
  sitting in — the scripts are sourced, so that is a real possibility. It uses
  absolute paths and reports when there is nothing to concatenate.
- The wait for HAProxy watches for the runtime API socket instead of probing
  TCP 9999, which no longer exists.
- compose: /var/lib/letsencrypt is declared a VOLUME by the base image, so
  leaving it unnamed created a fresh anonymous volume on every container
  creation. It and the volumes section, which was missing entirely, are added.
  Build context is the repository root, as the Dockerfile's COPY expects.

Verified on testnet2 against a real unix socket: the happy path sends set,
commit and show; a refused set stops before commit; a refused commit and an
unreachable socket are both reported. The runtime API payload is byte-identical
to what the previous echo produced.

docker-compose.yml

@@ -2,8 +2,10 @@ services:
   #Let's Encrypt requires domain's public A/AAAA DNS records pointed at your machine.
   certbot:
     build:
-      context: ./docker
-      dockerfile: Dockerfile
+      # The repository root, not ./docker: the Dockerfile copies
+      # ./docker/scripts/, which a context of ./docker cannot see.
+      context: .
+      dockerfile: ./docker/Dockerfile
     image: registry.bitdeals.org/certbot
     restart: unless-stopped
     environment:
@@ -12,4 +14,18 @@ services:
     volumes:
       - certificates:/etc/certificates
       - letsencrypt:/etc/letsencrypt
+      # The base image declares /var/lib/letsencrypt as a VOLUME as well. Left
+      # unmounted it gets a fresh anonymous volume on every container creation,
+      # so a redeploy leaves an orphan behind each time. Naming it keeps the
+      # work directory in one place across recreations.
+      - letsencrypt_work:/var/lib/letsencrypt
+      # HAProxy's runtime API socket, through which the renewed certificate is
+      # installed. Shared with haproxy and with nothing else: reaching that
+      # socket is equivalent to holding the TLS private key.
+      - haproxy_admin:/var/lib/haproxy
 
+volumes:
+  certificates:
+  letsencrypt:
+  letsencrypt_work:
+  haproxy_admin:

docker/scripts/0-create-cert.sh

@@ -3,32 +3,54 @@
 set -e
 
 if [ ! -f /etc/certificates/site.pem ]; then
-  # Generate self-signed certificate
-  openssl genrsa -out site.key 2048
-  openssl req -new -key site.key -out site.csr -batch
-  openssl x509 -req -days 365 -in site.csr -signkey site.key -out site.crt
-  cat site.key site.crt >> /etc/certificates/site.pem
+  # Self-signed placeholder, so HAProxy can bind :443 before a real certificate
+  # exists. Built inside a subshell: this script is sourced, so a bare cd or
+  # umask would change the caller's shell too. Relative paths used to resolve
+  # against the working directory (/opt/certbot) and left the private key there
+  # for good; here they resolve against the volume the key belongs on, and the
+  # intermediates are removed before site.pem appears.
+  (
+    umask 077
+    cd /etc/certificates
+    openssl req -x509 -newkey rsa:2048 -nodes -days 365 -batch \
+      -subj "/CN=${CERTBOT_DOMAIN:-localhost}" \
+      -keyout site.key.tmp -out site.crt.tmp
+    cat site.key.tmp site.crt.tmp > site.pem.tmp
+    rm -f site.key.tmp site.crt.tmp
+    # Last and atomic: HAProxy reads site.pem at start-up and must never find
+    # it half-written.
+    mv site.pem.tmp site.pem
+  )
 fi
 
-# Wait for haproxy container
-while ! nc -z haproxy 9999 2>/dev/null; do
-    echo "Waiting for haproxy:9999..."
+# Wait for haproxy: Let's Encrypt reaches this container's challenge server only
+# through it, so asking for a certificate first would just fail validation. The
+# runtime API socket appearing is the signal — HAProxy creates it once the
+# configuration has parsed and the binds have succeeded. Waiting on the socket
+# rather than a TCP probe also means there is no port left to probe.
+HAPROXY_SOCKET=/var/lib/haproxy/admin.sock
+while [ ! -S "$HAPROXY_SOCKET" ]; do
+    echo "Waiting for $HAPROXY_SOCKET..."
     sleep 7
 done
 
-# check e-mail for letsencrypt notifications
-if [ -n "$CERTBOT_EMAIL" ]; then
-  CERTBOT_OPTS="--email $CERTBOT_EMAIL"
-else
-  CERTBOT_OPTS="--register-unsafely-without-email"
-fi
-
 if [ -n "$CERTBOT_DOMAIN" ]; then
 
+  # E-mail for Let's Encrypt expiry notices. Built with set -- so that the flag
+  # and its value stay two separate arguments: a single "$CERTBOT_OPTS" word
+  # reached certbot as one token ("--email you@example.org") and was rejected
+  # as an unknown argument, which failed every issuance with an address set.
+  # The entrypoint passes no positional parameters, so there are none to lose.
+  if [ -n "$CERTBOT_EMAIL" ]; then
+    set -- --email "$CERTBOT_EMAIL"
+  else
+    set -- --register-unsafely-without-email
+  fi
+
   # Request certificate
   certbot certonly --standalone \
     --non-interactive --agree-tos --http-01-port=380 \
-    "$CERTBOT_OPTS" \
+    "$@" \
     --cert-name "$CERTBOT_DOMAIN" \
     -d "$CERTBOT_DOMAIN"
 

docker/scripts/2-concatenate-cert.sh

@@ -1,8 +1,20 @@
 #!/bin/sh
 
-cd /etc/letsencrypt/live/"$CERTBOT_DOMAIN"
+# Join the issued certificate and its key into the single site.pem HAProxy
+# expects.
+#
+# Absolute paths rather than a cd: this script is sourced, so a cd would move
+# the caller's working directory as well, and an unchecked one that failed left
+# the tests below looking for fullchain.pem in whatever directory the caller
+# happened to be in.
 
-if [ -f fullchain.pem -a -f privkey.pem ]; then
-  cat fullchain.pem privkey.pem > /etc/certificates/site.pem
+live="/etc/letsencrypt/live/$CERTBOT_DOMAIN"
+
+if [ -f "$live/fullchain.pem" ] && [ -f "$live/privkey.pem" ]; then
+  # Through a temporary file, then an atomic rename: HAProxy reads site.pem at
+  # start-up and must never find it half-written.
+  cat "$live/fullchain.pem" "$live/privkey.pem" > /etc/certificates/site.pem.tmp
+  mv /etc/certificates/site.pem.tmp /etc/certificates/site.pem
+else
+  echo "2-concatenate-cert.sh: no certificate under $live, site.pem left as it is" >&2
 fi
-

docker/scripts/3-update-haproxy-cert.sh

@@ -1,12 +1,67 @@
 #!/bin/sh
 
-# Start transaction
-echo -e "set ssl cert /usr/local/etc/haproxy/certificates/site.pem <<\n$(cat /etc/certificates/site.pem)\n" \
-  | socat - tcp-connect:haproxy:9999,retry=30,interval=2,connect-timeout=5
+# Install /etc/certificates/site.pem into the *running* HAProxy over its runtime
+# API — no restart, no dropped connections.
+#
+# Every answer is inspected. The runtime API reports a refusal in the reply text
+# and still closes the connection cleanly, so socat exits 0 either way: the
+# unchecked version sent set, commit and show in a row and reported success
+# while HAProxy went on serving the old certificate. If HAProxy ever reworded
+# these replies the checks would raise a false alarm, which is the safe
+# direction for the error to point — site.pem on the shared volume is correct
+# regardless, and HAProxy loads it from there on its next restart.
+
+CERT_PATH=/usr/local/etc/haproxy/certificates/site.pem
+SRC=/etc/certificates/site.pem
+# A unix socket on a volume shared with HAProxy, not a TCP port: the private key
+# below travels this channel in the clear, and file permissions are the only
+# access control a container-to-container path can have.
+API="UNIX-CONNECT:/var/lib/haproxy/admin.sock"
+
+update_haproxy_cert() {
+  if [ ! -s "$SRC" ]; then
+    echo "3-update-haproxy-cert.sh: $SRC is missing or empty, nothing to install" >&2
+    return 1
+  fi
+
+  # Start transaction. The payload ends with a blank line — that is what closes
+  # a << block on the runtime API. retry= is what waits out a HAProxy that is
+  # still coming up.
+  answer=$(
+    printf 'set ssl cert %s <<\n%s\n\n' "$CERT_PATH" "$(cat "$SRC")" \
+      | socat - "$API,retry=30,interval=2" 2>&1
+  ) || {
+    echo "3-update-haproxy-cert.sh: cannot reach the HAProxy runtime API: $answer" >&2
+    return 1
+  }
+
+  case "$answer" in
+    *"Transaction created"*) ;;
+    *)
+      echo "3-update-haproxy-cert.sh: HAProxy refused the certificate: $answer" >&2
+      return 1
+      ;;
+  esac
 
   # Commit transaction
-echo "commit ssl cert /usr/local/etc/haproxy/certificates/site.pem" | socat tcp-connect:haproxy:9999 -
+  answer=$(echo "commit ssl cert $CERT_PATH" | socat - "$API" 2>&1) || {
+    echo "3-update-haproxy-cert.sh: cannot reach the HAProxy runtime API: $answer" >&2
+    return 1
+  }
 
-# Show certification info (not essential)
-echo "show ssl cert /usr/local/etc/haproxy/certificates/site.pem" | socat tcp-connect:haproxy:9999 -
+  case "$answer" in
+    *Success*) ;;
+    *)
+      echo "3-update-haproxy-cert.sh: HAProxy refused to commit the certificate: $answer" >&2
+      return 1
+      ;;
+  esac
+}
 
+if update_haproxy_cert; then
+  # Show certification info (not essential) — it is what puts the live
+  # certificate's dates in the container log.
+  echo "show ssl cert $CERT_PATH" | socat - "$API" 2>&1 || true
+else
+  echo "3-update-haproxy-cert.sh: the running HAProxy still serves its previous certificate" >&2
+fi