feat: promote acme-dns renewal to production

docs/acme-dns-renewal-plan.md

@@ -0,0 +1,283 @@
+# Plan: restricted ACME DNS renewal for cc-ci
+
+## Outcome
+
+Replace the manually issued, sops-stored wildcard certificate with unattended
+DNS-01 renewal for these exact names:
+
+```text
+ci.commoninternet.net
+*.ci.commoninternet.net
+```
+
+The cc-ci host will run an authoritative `acme-dns` instance only for
+`acme.commoninternet.net`. It will never receive a Gandi credential or any
+credential that can edit the parent `commoninternet.net` zone.
+
+The only enduring delegation from the parent zone is:
+
+```text
+_acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.
+```
+
+That CNAME authorizes the generated acme-dns account to answer ACME TXT
+challenges for the ci wildcard, not to edit any parent-zone DNS record.
+
+## Project facts and constraints
+
+- The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the
+  orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6.
+- The wildcard currently points at the public gateway, which TLS-passthroughs
+  to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point
+  directly to `91.98.47.73`; the gateway is not involved in DNS.
+- Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and
+  443 only; any Hetzner Cloud firewall must also be checked before deployment.
+- TLS terminates in the Docker Swarm Traefik service. It currently reads
+  `ssl_cert` and `ssl_key` **Swarm secrets** populated from
+  `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by
+  `runner/warm_reconcile.py`. A normal host-service reload cannot install a
+  renewed certificate.
+- The existing certificate is expired: its served validity ended
+  `2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until
+  the new production certificate and Traefik rotation have both been verified.
+- `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS
+  `security.acme` uses Lego and supports a DNS provider plus an environment
+  file and post-renew hook. Confirm the pinned provider spelling with
+  `lego --help` during implementation; Lego's current documented provider code
+  is `acmedns`.
+
+## Security invariants
+
+1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any
+   parent-zone update credential on cc-ci or the orchestrator.
+2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP
+   because it is loopback-only; do not create a circular API TLS dependency.
+3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
+4. The generated acme-dns account data is a secret. Keep it as a root/acme-only
+   persistent state file under `/var/lib/acme/`; never put it in Nix text, the
+   Nix store, git, `.env.public`, or a log.
+5. After the account exists, set `disable_registration = true`. The existing
+   account must still be able to call `/update`.
+6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`.
+   This is defence in depth in addition to the loopback API binding.
+
+## Intended DNS design
+
+Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue
+ambiguity:
+
+```text
+ns-acme.commoninternet.net.  A   91.98.47.73
+acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
+```
+
+`acme-dns` itself serves the delegated zone and returns its matching NS record:
+
+```text
+acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
+```
+
+This host is authoritative for `acme.commoninternet.net` and its generated
+children only. It is not authoritative for `ci.commoninternet.net` or for
+`commoninternet.net`.
+
+## Implementation phases
+
+### 1. Preflight and safety checks
+
+Before changing Nix configuration, record:
+
+```bash
+ssh cc-ci 'ss -lntup "( sport = :53 )"'
+ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
+ssh cc-ci 'nft list ruleset'
+ssh cc-ci 'docker service ls'
+```
+
+Confirm that no service owns port 53, that the Traefik Swarm services are
+healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and
+53/udp. Do not replace an existing DNS service.
+
+Obtain the operator's ACME contact email before enabling `security.acme`.
+
+### 2. Add a dedicated acme-dns Nix module
+
+Create `nix/modules/acme-dns.nix` and import it from
+`nix/hosts/cc-ci-hetzner/configuration.nix`. The module should:
+
+- create a dedicated unprivileged `acme-dns` user and group;
+- run `${pkgs.acme-dns}/bin/acme-dns -c <public generated config>` with a
+  persistent working/state directory `/var/lib/acme-dns`;
+- grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53;
+- use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`;
+- bind DNS to `91.98.47.73:53` with `protocol = "both4"`;
+- set `domain = "acme.commoninternet.net"`,
+  `nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style
+  `nsadmin` value;
+- include the public NS record above in `general.records`;
+- bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive
+  CORS list, and initially leave `disable_registration = false`;
+- use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`,
+  `ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the
+  state directory as writable; and
+- open `networking.firewall.allowedTCPPorts = [ 53 ]` and
+  `allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration.
+
+The configuration file is public data and may be generated by Nix. It must not
+contain account credentials.
+
+Deploy this phase with the normal cc-ci deployment discipline: first
+`nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and
+the host remain healthy, then run the identical `switch` target. Verify local
+DNS on both transports:
+
+```bash
+dig @91.98.47.73 acme.commoninternet.net NS
+dig +tcp @91.98.47.73 acme.commoninternet.net NS
+```
+
+### 3. Operator gate: delegate the narrow DNS zone
+
+After the service is healthy, ask the operator to add exactly these records at
+Gandi (using its DNS UI, never a token on this host):
+
+```dns
+ns-acme.commoninternet.net.  A   91.98.47.73
+acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
+```
+
+If Gandi models delegation as a nameserver/glue form rather than ordinary zone
+records, use its equivalent UI flow. Do not proceed until public recursive DNS
+shows the delegation and direct queries work from an external network:
+
+```bash
+dig NS acme.commoninternet.net @1.1.1.1
+dig TXT test.acme.commoninternet.net @91.98.47.73
+dig +tcp TXT test.acme.commoninternet.net @91.98.47.73
+```
+
+### 4. Configure NixOS ACME in staging mode and obtain the account target
+
+Extend the new module with one `security.acme.certs` entry for the base name
+`ci.commoninternet.net`:
+
+```nix
+{
+  domain = "ci.commoninternet.net";
+  extraDomainNames = [ "*.ci.commoninternet.net" ];
+  dnsProvider = "acmedns"; # verify against the pinned Lego binary
+  environmentFile = "/etc/acme-dns/lego.env";
+  dnsResolver = "1.1.1.1:53";
+}
+```
+
+`/etc/acme-dns/lego.env` contains only non-secret wiring:
+
+```text
+ACME_DNS_API_BASE=http://127.0.0.1:8080
+ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
+ACME_DNS_ALLOWLIST=127.0.0.1/32
+```
+
+Lego registers and persists its per-domain acme-dns account in the storage
+path. The path is writable only by the ACME service user and is not Nix-managed
+content. Do not hand-create its JSON: let the pinned Lego provider establish
+the account format.
+
+Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate
+unit manually after the NS delegation is confirmed. The first staging run is
+expected to create the account and may fail validation because the CNAME is not
+yet present. Read the storage file only with a root-only helper that prints the
+generated **fulldomain** and never its username or password.
+
+### 5. Operator gate: permanent challenge CNAME
+
+Ask the operator to create the exact target reported in phase 4:
+
+```dns
+_acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.
+```
+
+This is a permanent record. It must not be created, changed, or removed by an
+agent. Confirm the complete chain through a public recursive resolver before
+continuing:
+
+```bash
+dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
+dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
+dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73
+```
+
+### 6. Staging issuance, then production issuance
+
+Run the NixOS ACME certificate unit against staging and verify all of the
+following:
+
+1. it updates only the generated acme-dns TXT target;
+2. public recursive DNS sees the CNAME and the TXT value;
+3. staging issues a certificate containing both requested names; and
+4. no Gandi variable, credential file, or API request appears in the unit.
+
+Only then select the production Let's Encrypt directory and issue the real
+certificate. Keep the old sops certificate live during both attempts.
+
+### 7. Make Traefik consume renewals safely
+
+Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and
+cannot see an updated host file. Add a root-only renewal handoff service,
+serialized with all other Traefik reconciliation, and call it from the ACME
+certificate's `postRun` hook.
+
+The handoff must:
+
+1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output
+   into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes;
+2. generate a new, content-derived **non-secret** Swarm secret version;
+3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe
+   environment to reference those versions, and reconcile/redeploy Traefik;
+4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI;
+5. retain the prior secret version until the new task is healthy, then remove
+   it; and
+6. record a failure clearly without deleting the last-known-good certificate.
+
+Implement this as a tested extension of `runner/warm_reconcile.py` (or a
+small, explicitly locked companion) rather than an ad-hoc shell command. The
+renewal path and the normal `deploy-proxy` path must share a lock so they cannot
+race over Swarm secret versions.
+
+After production issuance and a successful Traefik rotation, remove the
+`wildcard_cert` and `wildcard_key` sops declarations from
+`nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the
+renewed host files. Remove the obsolete encrypted values from the private
+`cc-ci-secrets` repository only after rollback is no longer needed.
+
+### 8. Lock registration and prove unattended renewal
+
+In a follow-up Nix change, set `api.disable_registration = true`, test that the
+existing account can still update its TXT record, and confirm `/register` is
+rejected. Then verify:
+
+```bash
+systemctl list-timers 'acme-*'
+systemctl start acme-ci.commoninternet.net.service
+journalctl -u acme-ci.commoninternet.net.service -b
+```
+
+Perform a controlled staging renewal after registration is disabled, observe
+the renewed Traefik secret version, and confirm the certificate served through
+the gateway has the expected names and a new validity window.
+
+## Final acceptance checklist
+
+- [ ] cc-ci and the orchestrator contain no Gandi API credential.
+- [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci.
+- [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone.
+- [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
+- [ ] External UDP and TCP authoritative DNS checks pass.
+- [ ] Registration is disabled after the one account is created.
+- [ ] The ACME account can update only its generated TXT record.
+- [ ] The certificate covers both `ci.commoninternet.net` and its wildcard.
+- [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior
+      version until the replacement passes health checks.
+- [ ] No credential or private key has been committed, logged, or written into
+      the Nix store.

nix/hosts/cc-ci-hetzner/configuration.nix

@@ -14,6 +14,7 @@
     ./networking.nix
     ../../modules/packages.nix
     ../../modules/secrets.nix
+    ../../modules/acme-dns.nix
     ../../modules/swarm.nix
     ../../modules/docker-prune.nix
     ../../modules/abra.nix

nix/modules/acme-dns.nix

@@ -0,0 +1,155 @@
+# Restricted DNS-01 certificate issuance for ci.commoninternet.net.
+#
+# This host is authoritative only for acme.commoninternet.net. Gandi continues
+# to own commoninternet.net; it delegates this narrow zone and one permanent
+# _acme-challenge CNAME manually. No Gandi credential is present here.
+{ pkgs, ... }:
+let
+  acmeDnsConfig = pkgs.writeText "cc-ci-acme-dns.conf" ''
+    [general]
+    listen = "91.98.47.73:53"
+    protocol = "both4"
+    domain = "acme.commoninternet.net"
+    nsname = "ns-acme.commoninternet.net"
+    nsadmin = "hostmaster.commoninternet.net"
+    records = [
+      "acme.commoninternet.net. NS ns-acme.commoninternet.net.",
+      "ns-acme.commoninternet.net. A 91.98.47.73",
+    ]
+    debug = false
+
+    [database]
+    # acme-dns 2.x registers the embedded driver under `sqlite` (not the
+    # legacy `sqlite3` identifier).
+    engine = "sqlite"
+    connection = "/var/lib/acme-dns/acme-dns.db"
+
+    [api]
+    ip = "127.0.0.1"
+    port = "8080"
+    tls = "none"
+    # The one Lego account was bootstrapped before this configuration was
+    # hardened. Updates authenticated by that account remain available.
+    disable_registration = true
+    corsorigins = []
+
+    [logconfig]
+    loglevel = "info"
+    logtype = "stdout"
+    logformat = "json"
+  '';
+
+  # These are wiring values only. The acme-dns account JSON is generated by
+  # Lego below /var/lib/acme and never enters Nix, git, or /etc.
+  legoEnvironment = pkgs.writeText "cc-ci-acme-dns-lego.env" ''
+    ACME_DNS_API_BASE=http://127.0.0.1:8080
+    ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
+    ACME_DNS_ALLOWLIST=127.0.0.1/32
+  '';
+in
+{
+  users.groups.acme-dns = { };
+  users.users.acme-dns = {
+    isSystemUser = true;
+    group = "acme-dns";
+    home = "/var/lib/acme-dns";
+  };
+
+  environment.etc."acme-dns/lego.env".source = legoEnvironment;
+
+  # The staging order has completed successfully. This marker permits the
+  # production ACME post-run hook to hand a renewed certificate to Traefik.
+  systemd.tmpfiles.rules = [
+    "f /var/lib/ci-certs/acme-production-enabled 0600 root root -"
+  ];
+
+  networking.firewall = {
+    allowedTCPPorts = [ 53 ];
+    allowedUDPPorts = [ 53 ];
+  };
+
+  systemd.services.acme-dns = {
+    description = "Restricted authoritative DNS for cc-ci ACME DNS-01";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      User = "acme-dns";
+      Group = "acme-dns";
+      StateDirectory = "acme-dns";
+      StateDirectoryMode = "0700";
+      WorkingDirectory = "/var/lib/acme-dns";
+      ExecStart = "${pkgs.acme-dns}/bin/acme-dns -c ${acmeDnsConfig}";
+      Restart = "on-failure";
+      RestartSec = "5s";
+      AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
+      CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
+      NoNewPrivileges = true;
+      PrivateTmp = true;
+      PrivateDevices = true;
+      ProtectHome = true;
+      ProtectSystem = "strict";
+      ReadWritePaths = [ "/var/lib/acme-dns" ];
+      RestrictAddressFamilies = [ "AF_INET" "AF_UNIX" ];
+    };
+  };
+
+  # Traefik consumes its wildcard as immutable Swarm secrets, so a renewed
+  # host certificate must be copied and reconciled rather than merely reloaded.
+  # This service is started only by the production-mode ACME postRun hook.
+  systemd.services.cc-ci-acme-traefik-handoff = {
+    description = "Install renewed cc-ci wildcard into Traefik Swarm secrets";
+    after = [ "docker.service" "deploy-proxy.service" ];
+    requires = [ "docker.service" ];
+    path = [ pkgs.coreutils pkgs.docker pkgs.systemd pkgs.gnugrep ];
+    serviceConfig = {
+      Type = "oneshot";
+      UMask = "0077";
+    };
+    script = ''
+      src=/var/lib/acme/ci.commoninternet.net
+      dst=/var/lib/ci-certs/live
+      test -s "$src/fullchain.pem"
+      test -s "$src/key.pem"
+      install -d -m 0700 "$dst"
+      install -m 0444 "$src/fullchain.pem" "$dst/fullchain.pem.new"
+      install -m 0400 "$src/key.pem" "$dst/privkey.pem.new"
+      mv -f "$dst/fullchain.pem.new" "$dst/fullchain.pem"
+      mv -f "$dst/privkey.pem.new" "$dst/privkey.pem"
+
+      # deploy-proxy performs the health-gated Swarm rollout. Its reconciler
+      # derives a fresh version from the public certificate chain and inserts
+      # the matching ssl_cert/ssl_key secrets before deploying Traefik.
+      systemctl restart deploy-proxy.service
+
+      # A successful rollout no longer references old wildcard versions. Best
+      # effort removal retains any secret Docker still reports as in use.
+      keep="v$(sha256sum "$dst/fullchain.pem" | cut -c1-16)"
+      docker secret ls --format '{{.Name}}' | \
+        grep -E '^traefik_ci_commoninternet_net_ssl_(cert|key)_v' | \
+        grep -v -E "_(ssl_cert|ssl_key)_$keep\$" | \
+        while IFS= read -r stale; do docker secret rm "$stale" || true; done
+    '';
+  };
+
+  security.acme = {
+    acceptTerms = true;
+    certs."ci.commoninternet.net" = {
+      domain = "ci.commoninternet.net";
+      extraDomainNames = [ "*.ci.commoninternet.net" ];
+      # Staging issuance succeeded using the permanent, narrowly delegated
+      # CNAME. Production uses the same restricted acme-dns account.
+      dnsProvider = "acmedns";
+      environmentFile = "/etc/acme-dns/lego.env";
+      dnsResolver = "1.1.1.1:53";
+      server = "https://acme-v02.api.letsencrypt.org/directory";
+      postRun = ''
+        # The production marker is deployed only after staging proves the
+        # permanent CNAME and restricted acme-dns account work end to end.
+        if [ -e /var/lib/ci-certs/acme-production-enabled ]; then
+          ${pkgs.systemd}/bin/systemctl --no-block start cc-ci-acme-traefik-handoff.service
+        fi
+      '';
+    };
+  };
+}

runner/warm_reconcile.py

@@ -24,6 +24,7 @@ Run as root on cc-ci (direct docker/volume access). CLI: `warm_reconcile.py <app
 
 from __future__ import annotations
 
+import hashlib
 import json
 import os
 import re
@@ -37,12 +38,29 @@ from harness import abra, lifecycle, warmsnap  # noqa: E402
 # --------------------------------------------------------------------------- specs
 
 
+CERT_DIR = "/var/lib/ci-certs/live"
+
+
+def wildcard_secret_version(cert_dir: str = CERT_DIR) -> str:
+    """Stable Swarm-secret version for the public certificate chain.
+
+    The certificate chain is public material, so its digest is safe to use as a
+    version label. The key is deliberately never read or hashed for logging.
+    """
+    chain = os.path.join(cert_dir, "fullchain.pem")
+    if not os.path.isfile(chain):
+        raise RuntimeError(f"FATAL: wildcard certificate missing at {chain}")
+    with open(chain, "rb") as certificate:
+        digest = hashlib.sha256(certificate.read()).hexdigest()
+    return "v" + digest[:16]
+
+
 def _traefik_setup(recipe: str, domain: str, version: str) -> None:
     """Per-app config for the traefik reverse-proxy reconcile — preserves EXACTLY what the prior
     proxy.nix bash reconcile did (wildcard/file-provider mode serving the pre-issued cert as
     ssl_cert/ssl_key swarm secrets; NO ACME). Uses the proven abra.env_set (newline-safe, unlike the
     bash set_env that bit keycloak)."""
-    cert_dir = "/var/lib/ci-certs/live"
+    cert_dir = CERT_DIR
     if not (
         os.path.isfile(f"{cert_dir}/fullchain.pem") and os.path.isfile(f"{cert_dir}/privkey.pem")
     ):
@@ -56,14 +74,15 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
     abra.env_set(domain, "DOMAIN", domain)
     abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
     abra.env_set(domain, "WILDCARDS_ENABLED", "1")
-    abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", "v1")
-    abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", "v1")
+    secret_version = wildcard_secret_version(cert_dir)
+    abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", secret_version)
+    abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", secret_version)
     abra.env_set(domain, "COMPOSE_FILE", '"compose.yml:compose.wildcard.yml"')
     stack = lifecycle._stack_name(domain)  # noqa: SLF001
     have = set(lifecycle._docker_names("secret", stack))  # noqa: SLF001
 
     def _has(name):
-        return any(s.endswith(f"_{name}_v1") for s in have)
+        return any(s.endswith(f"_{name}_{secret_version}") for s in have)
 
     if not _has("ssl_cert"):
         _run(
@@ -74,7 +93,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
                 "insert",
                 domain,
                 "ssl_cert",
-                "v1",
+                secret_version,
                 f"{cert_dir}/fullchain.pem",
                 "-f",
                 "-n",
@@ -91,7 +110,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
                 "insert",
                 domain,
                 "ssl_key",
-                "v1",
+                secret_version,
                 f"{cert_dir}/privkey.pem",
                 "-f",
                 "-n",

tests/unit/test_acme_dns.py

@@ -0,0 +1,21 @@
+"""Unit coverage for the public wildcard-secret version label."""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parents[2] / "runner"))
+import warm_reconcile as wr  # noqa: E402
+
+
+def test_wildcard_secret_version_is_stable_and_does_not_need_key(tmp_path):
+    (tmp_path / "fullchain.pem").write_text("public certificate chain\n")
+    assert wr.wildcard_secret_version(str(tmp_path)) == wr.wildcard_secret_version(str(tmp_path))
+    assert wr.wildcard_secret_version(str(tmp_path)).startswith("v")
+
+
+def test_wildcard_secret_version_changes_with_certificate_chain(tmp_path):
+    chain = tmp_path / "fullchain.pem"
+    chain.write_text("first public certificate chain\n")
+    first = wr.wildcard_secret_version(str(tmp_path))
+    chain.write_text("replacement public certificate chain\n")
+    assert wr.wildcard_secret_version(str(tmp_path)) != first