feat: harden the edge and move the runtime API off TCP

The runtime API was an unauthenticated `level admin` channel on TCP 9999.
`expose:` restricts nothing and docker networks have no per-port rules, so it
was reachable by every container sharing a network — nginx included — and
reaching it means installing your own certificate and key. It is now a unix
socket on a volume shared with certbot alone, which also keeps the private key
off the network on every renewal. `expose-fd listeners` is dropped: it hands
the listening sockets themselves to a client of that socket and only serves
seamless reloads, which this image never performs.

HAProxy binds a unix socket by creating `<path>.<pid>.tmp` and renaming it, so
it needs write access to the directory; the image now creates /var/lib/haproxy
owned by uid 1001 and docker carries that onto an empty named volume. Without
it HAProxy refuses to start.

Also in this commit:

- The visitor's address no longer leaves the process. `option forwardfor` is
  gone, X-Forwarded-For is deleted unconditionally, and X-Client-Id carries an
  HMAC of the address under the optional XFF_HMAC_KEY instead — one-to-one with
  the address, so a rate limit keyed on it is as precise, but not reversible.
  An empty key sends no header rather than one derived from an empty key.
- Logging, which was absent entirely. To stdout for `docker logs`, errors-only
  via the existing dontlog-normal. Both log-format and error-log-format are
  hand-written: the built-in formats open with %ci:%cp and would have logged
  the addresses the rest of this works to avoid. The pseudonym is computed by a
  tcp-request connection rule so a refused handshake has one too.
- TLS pinned: floor TLS 1.2, ECDHE-only in ECDSA and RSA variants, no session
  tickets, ALPN offering HTTP/2.
- HTTP redirects to HTTPS, excepting the ACME challenge path, plus HSTS at one
  day — short deliberately, since the header cannot be recalled once sent.
- timeout http-request, which `timeout client` cannot stand in for: that one
  resets on every byte, so a slow-drip client held a connection indefinitely.
- Backends re-resolve through the declared `resolvers docker`, which nothing
  referenced. Names were resolved once at boot, so a container recreated on a
  new IP was never noticed. init-addr libc,none lets HAProxy start anyway.
- The commented-out /ms deny rules are deleted. They would have blocked
  /ms/info and /ms/api/v2/whoami, both deliberately public.
- compose: build context is the repository root, as the Dockerfile's COPY
  expects and as CI already did; restart policy added, since an empty
  certificate volume is a fatal start-up error; the volumes section, which was
  missing, so the file can actually come up.

Verified on testnet2 against the real binary: config parses clean with the key
set, empty, and rejects a non-base64 key at parse time; a live stack confirms
the pseudonym is stable per address, differs between addresses, survives a
forged header, and that the client address appears nowhere in the log.

docker-compose.yml

@@ -1,14 +1,27 @@
 services:
   haproxy:
     build:
-      context: ./docker
-      dockerfile: Dockerfile
+      # The repository root, not ./docker: the Dockerfile copies
+      # ./docker/haproxy.cfg, which a context of ./docker cannot see.
+      context: .
+      dockerfile: ./docker/Dockerfile
     image: registry.bitdeals.org/haproxy
+    # `bind ... ssl crt` is resolved while the configuration is parsed, so an
+    # empty certificates volume is a fatal start-up error rather than a warning.
+    # certbot writes a self-signed placeholder on its own first start; until it
+    # has, HAProxy needs to keep retrying. Order it after certbot too — in a
+    # project that defines one, add: depends_on: [nginx, certbot]
+    restart: unless-stopped
     volumes:
       - certificates:/usr/local/etc/haproxy/certificates:ro
+      # The runtime API socket. Share this volume with certbot and with nothing
+      # else: reaching the socket is equivalent to holding the TLS private key.
+      - haproxy_admin:/var/lib/haproxy
     ports:
       - "80:80"
       - "443:443"
-    expose:
-      - "9999"
+
+volumes:
+  certificates:
+  haproxy_admin:
 

docker/Dockerfile

@@ -3,3 +3,16 @@ FROM bitnami/haproxy
 # Copy config
 COPY ./docker/haproxy.cfg /bitnami/haproxy/conf/haproxy.cfg
 
+# Directory for the runtime API socket. HAProxy runs as uid 1001 here and binds
+# a unix socket by creating `<path>.<pid>.tmp` and renaming it over the target,
+# so it needs write permission on the *directory*, not just the file — which is
+# also why a stale socket left by a previous run is harmless.
+#
+# /var/lib is owned by root, hence the explicit USER switch. Docker copies this
+# ownership onto an empty named volume when it initialises one here, so the
+# volume shared with certbot comes up writable by HAProxy without a chown at
+# runtime.
+USER root
+RUN mkdir -p /var/lib/haproxy && chown 1001:1001 /var/lib/haproxy
+USER 1001
+

docker/haproxy.cfg

@@ -1,6 +1,40 @@
 global
-    # Enable HAProxy runtime API
-    stats socket :9999 level admin expose-fd listeners
+    # The runtime API, through which certbot installs a renewed certificate.
+    #
+    # A unix socket on a volume shared with certbot alone, not a TCP port. This
+    # is an unauthenticated `level admin` channel — whoever reaches it can
+    # install their own certificate and private key, or point a backend
+    # somewhere else — and a TCP port is reachable by every container sharing a
+    # network, which here would include nginx. File permissions are the only
+    # access control available; a docker network has none.
+    #
+    # No expose-fd listeners: that would additionally hand a client of this
+    # socket the listening sockets themselves, and it is only needed for
+    # seamless reloads, which this image never performs.
+    stats socket /var/lib/haproxy/admin.sock mode 660 level admin
+
+    # To stdout, so `docker logs` collects it — a container has no syslog
+    # daemon, and without any `log` directive HAProxy emits nothing at all:
+    # no backend failures, no TLS handshake refusals, no 5xx.
+    log stdout format raw local0
+
+    # TLS policy for every `bind ... ssl` below. Left unset, the floor is
+    # whatever the image's OpenSSL happens to accept, which is not a decision
+    # this repository gets to make once and keep.
+    #
+    # ECDHE only, no DHE: every client that speaks TLS 1.2 has done ECDHE for
+    # over a decade, and admitting DHE would drag in the question of DH
+    # parameter size for no gain. Both -ECDSA- and -RSA- variants are listed
+    # because the certificate can be either — certbot issues ECDSA by default,
+    # while the self-signed placeholder it falls back to is RSA.
+    #
+    # no-tls-tickets: a resumption ticket is sealed with a key held for the life
+    # of the process, so a ticket captured today is readable by anyone who later
+    # obtains that key. Dropping tickets keeps forward secrecy whole, at the
+    # cost of a full handshake on resumption.
+    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
+    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305
+    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
 
 defaults
     mode http
@@ -9,43 +43,130 @@ defaults
     timeout connect 5s
     timeout client 1m
     timeout server 1m
+    # An absolute cap on the header-reading phase. `timeout client` cannot
+    # replace it: that one is an inactivity timeout and resets on every byte, so
+    # a client dripping one byte at a time holds a connection open for as long
+    # as it likes.
+    timeout http-request 10s
 
+    log global
+
+    # A hand-written format, never `option httplog`: that one's default begins
+    # with `%ci:%cp` and would put every visitor's address into `docker logs`,
+    # undoing the pseudonym the frontends go to the trouble of minting. The
+    # pseudonym takes that first field instead, and with no key configured the
+    # field is a bare `-` rather than something derived from an empty key.
+    #
+    # The request line is logged as method and path only. `%{+Q}r` would carry
+    # the query string too, and a token that ever appears in one would be
+    # written down for as long as the log is kept.
+    # Two loggers, and the second is easy to miss: a connection that dies before
+    # a transaction exists — a refused TLS handshake, and TLS 1.2 is now the
+    # floor — is written by the error logger, whose default format *also* opens
+    # with %ci:%cp. Leaving it alone would log the address of exactly the
+    # clients this configuration turns away.
+.if !streq("${XFF_HMAC_KEY}","")
+    log-format "%[var(sess.cid)] [%tr] %ft %b/%s %ST %B %TR/%Ta %HM %HP"
+    error-log-format "%[var(sess.cid)] [%tr] %ft %ac/%fc %[fc_err_str]"
+.else
+    log-format "- [%tr] %ft %b/%s %ST %B %TR/%Ta %HM %HP"
+    error-log-format "- [%tr] %ft %ac/%fc %[fc_err_str]"
+.endif
+
+    # Kept: with logging now on, this is what makes the log errors-only rather
+    # than a full access log. Remove it deliberately if every request is wanted.
     option dontlog-normal
     option tcp-smart-accept
     option tcp-smart-connect
-    #option forwardfor
+    # No `option forwardfor`, deliberately: it would write the visitor's real
+    # address into X-Forwarded-For, and the address must not leave this process.
+    # Each frontend deletes that header and mints X-Client-Id instead — see
+    # XFF_HMAC_KEY below.
     option http-keep-alive
 
     http-reuse safe
 
 frontend http
     bind *:80
-    #http-request redirect scheme https code 301
+
+.if !streq("${XFF_HMAC_KEY}","")
+    # A pseudonym instead of the address: one-to-one with it, so a rate limit
+    # keyed on it is exactly as precise, but not reversible without the key.
+    # HMAC rather than a bare digest — IPv4 is 2^32 values, and an unkeyed hash
+    # of an address is brute-forced in seconds. Unset key means no pseudonym at
+    # all, never one derived from an empty key.
+    #
+    # Computed on accept rather than in the http phase: a TLS handshake that
+    # fails never reaches an http-request rule, and the error logger still needs
+    # something to write in place of the address. `sess` scope so it outlives
+    # the transaction. Written above the http-request rules because that is the
+    # order it runs in, and HAProxy warns when the two disagree.
+    tcp-request connection set-var(sess.cid) src,hmac(sha256,"${XFF_HMAC_KEY}"),base64
+.endif
+
+    # The visitor's address stops here — X-Forwarded-For is dropped rather than
+    # filled in, so nothing downstream can log an address it was never given.
+    # Both deletes run unconditionally: a header a client sent must never be
+    # mistaken for one this proxy minted.
+    http-request del-header X-Forwarded-For
+    http-request del-header X-Client-Id
+.if !streq("${XFF_HMAC_KEY}","")
+    http-request set-header X-Client-Id %[var(sess.cid)]
+.endif
+    http-request set-header X-Forwarded-Proto http
 
     # ACL
     acl certbot path_beg /.well-known/acme-challenge/
-    use_backend certbot if certbot
 
-#    # Fool-proof: MS has no auth — never expose /ms publicly.
-#    http-request deny deny_status 404 if { path -i /ms } || { path -i -m beg /ms/ }
+    # Everything except the ACME challenge goes to HTTPS. The `unless` is
+    # load-bearing: Let's Encrypt validates over plain HTTP on port 80, so
+    # redirecting that path would break every renewal. Written above
+    # use_backend because that is the order it runs in — http-request rules are
+    # evaluated before backend selection whatever the order in the file, and
+    # HAProxy warns when the two disagree.
+    http-request redirect scheme https code 301 unless certbot
+
+    use_backend certbot if certbot
 
 frontend https
-    bind *:443 ssl crt /usr/local/etc/haproxy/certificates/site.pem
-    http-request add-header X-Forwarded-Proto https
+    # alpn: without it HAProxy negotiates HTTP/1.1 only. The backend stays
+    # HTTP/1.1 — HAProxy translates — so this is a client-side upgrade alone.
+    bind *:443 ssl crt /usr/local/etc/haproxy/certificates/site.pem alpn h2,http/1.1
+
+    # Same as the http frontend — see the comments there.
+.if !streq("${XFF_HMAC_KEY}","")
+    tcp-request connection set-var(sess.cid) src,hmac(sha256,"${XFF_HMAC_KEY}"),base64
+.endif
+
+    http-request del-header X-Forwarded-For
+    http-request del-header X-Client-Id
+.if !streq("${XFF_HMAC_KEY}","")
+    http-request set-header X-Client-Id %[var(sess.cid)]
+.endif
+    http-request set-header X-Forwarded-Proto https
+
+    # HSTS. max-age is deliberately one day, not the customary year: this is a
+    # one-way door — a browser that has seen the header refuses plain HTTP to
+    # this host until it expires, and no server-side change can call it back.
+    # One day keeps a mistake (a lapsed certificate, a host that must serve
+    # HTTP again) recoverable within a day. Raise it in steps once renewals have
+    # been seen to work for a while: 86400 -> 2592000 -> 31536000.
+    # No includeSubDomains and no preload: both would bind names this proxy does
+    # not serve, and preload is effectively permanent.
+    http-response set-header Strict-Transport-Security "max-age=86400"
 
     # ACL
     acl certbot path_beg /.well-known/acme-challenge/
     use_backend certbot if certbot
 
-#    # Fool-proof: MS has no auth — never expose /ms publicly.
-#    http-request deny deny_status 404 if { path -i /ms } || { path -i -m beg /ms/ }
-
 backend default-backend-http
-    http-request set-header X-Forwarded-Proto https if { ssl_fc }
-    server main nginx:80 check
+    # resolvers: without it the name is resolved once at start-up and kept, so a
+    # recreated nginx container on a new IP is never noticed. init-addr libc,none
+    # lets HAProxy start even when nginx is not up yet.
+    server main nginx:80 check resolvers docker resolve-prefer ipv4 init-addr libc,none
 
 backend certbot
-    server certbot certbot:380
+    server certbot certbot:380 resolvers docker resolve-prefer ipv4 init-addr libc,none
 
 resolvers docker
     nameserver dns1 127.0.0.11:53