The public door closed again this morning. haproxy exited at parse time on
every host with
'bind *:443' : inconsistencies between private key and certificate
loaded '/usr/local/etc/haproxy/certificates/site.pem'
on a leaf and a private key that openssl confirms are one pair — same public
key, and the same pair certbot has been serving since 2026-08-22. Nothing of
ours had changed: the image was rebuilt at 10:21 by the scheduled build, on a
base that had moved overnight.
The cause is the override this file added two days ago. Measured on testnet2,
same image, same certificate, only the variable changed:
OPENSSL_FIPS=no config check fails at `bind *:443`
OPENSSL_FIPS=yes config check passes, curl 200, TLSv1.3
`ENV OPENSSL_FIPS=no` was right for the 2026-09-02 base, where preferring the
FIPS provider left haproxy offering an X25519MLKEM768 group it could not use
and refusing every modern client. On today's base the same value stops it
loading an ECDSA certificate at all, while the shipped "yes" both loads it and
still negotiates TLS 1.3 — the thing the override existed to protect. So it is
removed rather than inverted; the comment keeps why, because the right value
depends on the base and the next base may want the other one.
Which is the second change, and the one that matters. `FROM bitnami/haproxy`
carries no tag and the build runs daily, so this image took whatever the base
was each morning: two outages in three days, neither with a commit behind it,
both diagnosed from scratch. The base is pinned by digest now — the one
measured working today. It stops moving on its own, and bumping it becomes a
deliberate act with a check behind it.
The public door has been refusing every modern client since 2026-09-02,
with no commit behind it. `FROM bitnami/haproxy` carries no tag and the
scheduled build runs daily, so the image is rebuilt each morning on
whatever base is `:latest` that day. That morning's base arrived with
OpenSSL's FIPS provider preferred.
Photon OS reads the preference straight out of the environment:
[alg_sect]
default_properties = ?fips=$ENV::OPENSSL_FIPS
With "yes", every algorithm fetch prefers the FIPS provider, and that
provider cannot produce an X25519MLKEM768 key — ssl_generate_pkey_group
fails. HAProxy offers the group anyway: OpenSSL 3.5 lists it first and
haproxy.cfg sets no ssl-default-bind-curves, so the server picks a group
it cannot then use and answers a fatal illegal_parameter. Anything that
offers post-quantum key exchange — curl, browsers, httpx — is refused;
only a client asking for a classical group gets in. МС talks to ДС over
HTTPS, so no deal could be created through it either.
Nothing here needs FIPS, and the base image itself provides the switch.
The provider stays loaded; only the preference is dropped.
Verified by building this image on testnet2 and serving the live
certificate from it: curl 200 where it was a TLS connect error, group
X25519MLKEM768 negotiated, ECDHE-ECDSA-CHACHA20-POLY1305 still offered,
TLS 1.1 still refused.
An empty key switches the pseudonym off: no X-Client-Id is sent, and a
rate limit downstream falls back to one bucket shared by every visitor.
That is the one state nobody chooses on purpose and the easiest to reach
by forgetting a line in a .env — so the entrypoint now fills the key in
with `openssl rand -base64 32` when nothing else did.
Nobody picks this value, nothing outside the container needs to know it,
and no two deployments need the same one, which is what makes generating
it the right default rather than a convenience. A fresh key per container
start costs a reset of the downstream rate-limit buckets — invisible
against a one-minute window — and makes pseudonyms from before and after
unlinkable, which is the property the key exists for rather than a loss.
Passing one explicitly still wins, for whoever wants pseudonyms stable
across restarts or identical on two proxies.
base64 and not hex, deliberately: HAProxy's hmac() decodes the key as
base64, and hex would be accepted and silently decoded into something
else — a usable key, but it would quietly cost the guarantee that a
malformed one stops the container at configuration parsing.
The base image's entrypoint is the haproxy binary with no shell in
between, so the wrapper is the whole chain and execs the same binary with
the same arguments. CMD is restated rather than inherited.
Verified by building the image and checking the config in all three
states: unset (wrapper reports it generated one, config parses), set and
valid (wrapper silent, config parses), set and not base64 (`[ALERT]
invalid args in converter 'hmac' : failed to parse key`, container
refuses to start).
READMEs updated in both languages, and "address" is spelled "IP address"
throughout — it was never anything else.
The runtime API was an unauthenticated `level admin` channel on TCP 9999.
`expose:` restricts nothing and docker networks have no per-port rules, so it
was reachable by every container sharing a network — nginx included — and
reaching it means installing your own certificate and key. It is now a unix
socket on a volume shared with certbot alone, which also keeps the private key
off the network on every renewal. `expose-fd listeners` is dropped: it hands
the listening sockets themselves to a client of that socket and only serves
seamless reloads, which this image never performs.
HAProxy binds a unix socket by creating `<path>.<pid>.tmp` and renaming it, so
it needs write access to the directory; the image now creates /var/lib/haproxy
owned by uid 1001 and docker carries that onto an empty named volume. Without
it HAProxy refuses to start.
Also in this commit:
- The visitor's address no longer leaves the process. `option forwardfor` is
gone, X-Forwarded-For is deleted unconditionally, and X-Client-Id carries an
HMAC of the address under the optional XFF_HMAC_KEY instead — one-to-one with
the address, so a rate limit keyed on it is as precise, but not reversible.
An empty key sends no header rather than one derived from an empty key.
- Logging, which was absent entirely. To stdout for `docker logs`, errors-only
via the existing dontlog-normal. Both log-format and error-log-format are
hand-written: the built-in formats open with %ci:%cp and would have logged
the addresses the rest of this works to avoid. The pseudonym is computed by a
tcp-request connection rule so a refused handshake has one too.
- TLS pinned: floor TLS 1.2, ECDHE-only in ECDSA and RSA variants, no session
tickets, ALPN offering HTTP/2.
- HTTP redirects to HTTPS, excepting the ACME challenge path, plus HSTS at one
day — short deliberately, since the header cannot be recalled once sent.
- timeout http-request, which `timeout client` cannot stand in for: that one
resets on every byte, so a slow-drip client held a connection indefinitely.
- Backends re-resolve through the declared `resolvers docker`, which nothing
referenced. Names were resolved once at boot, so a container recreated on a
new IP was never noticed. init-addr libc,none lets HAProxy start anyway.
- The commented-out /ms deny rules are deleted. They would have blocked
/ms/info and /ms/api/v2/whoami, both deliberately public.
- compose: build context is the repository root, as the Dockerfile's COPY
expects and as CI already did; restart policy added, since an empty
certificate volume is a fatal start-up error; the volumes section, which was
missing, so the file can actually come up.
Verified on testnet2 against the real binary: config parses clean with the key
set, empty, and rejects a non-base64 key at parse time; a live stack confirms
the pseudonym is stable per address, differs between addresses, survives a
forged header, and that the client address appears nowhere in the log.
bitdeals
bitdeals
private-user