diff --git a/docs/acme-dns-renewal-plan.md b/docs/acme-dns-renewal-plan.md new file mode 100644 index 0000000..ac347c2 --- /dev/null +++ b/docs/acme-dns-renewal-plan.md @@ -0,0 +1,283 @@ +# Plan: restricted ACME DNS renewal for cc-ci + +## Outcome + +Replace the manually issued, sops-stored wildcard certificate with unattended +DNS-01 renewal for these exact names: + +```text +ci.commoninternet.net +*.ci.commoninternet.net +``` + +The cc-ci host will run an authoritative `acme-dns` instance only for +`acme.commoninternet.net`. It will never receive a Gandi credential or any +credential that can edit the parent `commoninternet.net` zone. + +The only enduring delegation from the parent zone is: + +```text +_acme-challenge.ci.commoninternet.net. CNAME .acme.commoninternet.net. +``` + +That CNAME authorizes the generated acme-dns account to answer ACME TXT +challenges for the ci wildcard, not to edit any parent-zone DNS record. + +## Project facts and constraints + +- The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the + orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6. +- The wildcard currently points at the public gateway, which TLS-passthroughs + to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point + directly to `91.98.47.73`; the gateway is not involved in DNS. +- Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and + 443 only; any Hetzner Cloud firewall must also be checked before deployment. +- TLS terminates in the Docker Swarm Traefik service. It currently reads + `ssl_cert` and `ssl_key` **Swarm secrets** populated from + `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by + `runner/warm_reconcile.py`. A normal host-service reload cannot install a + renewed certificate. +- The existing certificate is expired: its served validity ended + `2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until + the new production certificate and Traefik rotation have both been verified. +- `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS + `security.acme` uses Lego and supports a DNS provider plus an environment + file and post-renew hook. Confirm the pinned provider spelling with + `lego --help` during implementation; Lego's current documented provider code + is `acmedns`. + +## Security invariants + +1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any + parent-zone update credential on cc-ci or the orchestrator. +2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP + because it is loopback-only; do not create a circular API TLS dependency. +3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone. +4. The generated acme-dns account data is a secret. Keep it as a root/acme-only + persistent state file under `/var/lib/acme/`; never put it in Nix text, the + Nix store, git, `.env.public`, or a log. +5. After the account exists, set `disable_registration = true`. The existing + account must still be able to call `/update`. +6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`. + This is defence in depth in addition to the loopback API binding. + +## Intended DNS design + +Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue +ambiguity: + +```text +ns-acme.commoninternet.net. A 91.98.47.73 +acme.commoninternet.net. NS ns-acme.commoninternet.net. +``` + +`acme-dns` itself serves the delegated zone and returns its matching NS record: + +```text +acme.commoninternet.net. NS ns-acme.commoninternet.net. +``` + +This host is authoritative for `acme.commoninternet.net` and its generated +children only. It is not authoritative for `ci.commoninternet.net` or for +`commoninternet.net`. + +## Implementation phases + +### 1. Preflight and safety checks + +Before changing Nix configuration, record: + +```bash +ssh cc-ci 'ss -lntup "( sport = :53 )"' +ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"' +ssh cc-ci 'nft list ruleset' +ssh cc-ci 'docker service ls' +``` + +Confirm that no service owns port 53, that the Traefik Swarm services are +healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and +53/udp. Do not replace an existing DNS service. + +Obtain the operator's ACME contact email before enabling `security.acme`. + +### 2. Add a dedicated acme-dns Nix module + +Create `nix/modules/acme-dns.nix` and import it from +`nix/hosts/cc-ci-hetzner/configuration.nix`. The module should: + +- create a dedicated unprivileged `acme-dns` user and group; +- run `${pkgs.acme-dns}/bin/acme-dns -c ` with a + persistent working/state directory `/var/lib/acme-dns`; +- grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53; +- use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`; +- bind DNS to `91.98.47.73:53` with `protocol = "both4"`; +- set `domain = "acme.commoninternet.net"`, + `nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style + `nsadmin` value; +- include the public NS record above in `general.records`; +- bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive + CORS list, and initially leave `disable_registration = false`; +- use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`, + `ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the + state directory as writable; and +- open `networking.firewall.allowedTCPPorts = [ 53 ]` and + `allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration. + +The configuration file is public data and may be generated by Nix. It must not +contain account credentials. + +Deploy this phase with the normal cc-ci deployment discipline: first +`nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and +the host remain healthy, then run the identical `switch` target. Verify local +DNS on both transports: + +```bash +dig @91.98.47.73 acme.commoninternet.net NS +dig +tcp @91.98.47.73 acme.commoninternet.net NS +``` + +### 3. Operator gate: delegate the narrow DNS zone + +After the service is healthy, ask the operator to add exactly these records at +Gandi (using its DNS UI, never a token on this host): + +```dns +ns-acme.commoninternet.net. A 91.98.47.73 +acme.commoninternet.net. NS ns-acme.commoninternet.net. +``` + +If Gandi models delegation as a nameserver/glue form rather than ordinary zone +records, use its equivalent UI flow. Do not proceed until public recursive DNS +shows the delegation and direct queries work from an external network: + +```bash +dig NS acme.commoninternet.net @1.1.1.1 +dig TXT test.acme.commoninternet.net @91.98.47.73 +dig +tcp TXT test.acme.commoninternet.net @91.98.47.73 +``` + +### 4. Configure NixOS ACME in staging mode and obtain the account target + +Extend the new module with one `security.acme.certs` entry for the base name +`ci.commoninternet.net`: + +```nix +{ + domain = "ci.commoninternet.net"; + extraDomainNames = [ "*.ci.commoninternet.net" ]; + dnsProvider = "acmedns"; # verify against the pinned Lego binary + environmentFile = "/etc/acme-dns/lego.env"; + dnsResolver = "1.1.1.1:53"; +} +``` + +`/etc/acme-dns/lego.env` contains only non-secret wiring: + +```text +ACME_DNS_API_BASE=http://127.0.0.1:8080 +ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json +ACME_DNS_ALLOWLIST=127.0.0.1/32 +``` + +Lego registers and persists its per-domain acme-dns account in the storage +path. The path is writable only by the ACME service user and is not Nix-managed +content. Do not hand-create its JSON: let the pinned Lego provider establish +the account format. + +Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate +unit manually after the NS delegation is confirmed. The first staging run is +expected to create the account and may fail validation because the CNAME is not +yet present. Read the storage file only with a root-only helper that prints the +generated **fulldomain** and never its username or password. + +### 5. Operator gate: permanent challenge CNAME + +Ask the operator to create the exact target reported in phase 4: + +```dns +_acme-challenge.ci.commoninternet.net. CNAME .acme.commoninternet.net. +``` + +This is a permanent record. It must not be created, changed, or removed by an +agent. Confirm the complete chain through a public recursive resolver before +continuing: + +```bash +dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1 +dig TXT .acme.commoninternet.net @91.98.47.73 +dig +tcp TXT .acme.commoninternet.net @91.98.47.73 +``` + +### 6. Staging issuance, then production issuance + +Run the NixOS ACME certificate unit against staging and verify all of the +following: + +1. it updates only the generated acme-dns TXT target; +2. public recursive DNS sees the CNAME and the TXT value; +3. staging issues a certificate containing both requested names; and +4. no Gandi variable, credential file, or API request appears in the unit. + +Only then select the production Let's Encrypt directory and issue the real +certificate. Keep the old sops certificate live during both attempts. + +### 7. Make Traefik consume renewals safely + +Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and +cannot see an updated host file. Add a root-only renewal handoff service, +serialized with all other Traefik reconciliation, and call it from the ACME +certificate's `postRun` hook. + +The handoff must: + +1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output + into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes; +2. generate a new, content-derived **non-secret** Swarm secret version; +3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe + environment to reference those versions, and reconcile/redeploy Traefik; +4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI; +5. retain the prior secret version until the new task is healthy, then remove + it; and +6. record a failure clearly without deleting the last-known-good certificate. + +Implement this as a tested extension of `runner/warm_reconcile.py` (or a +small, explicitly locked companion) rather than an ad-hoc shell command. The +renewal path and the normal `deploy-proxy` path must share a lock so they cannot +race over Swarm secret versions. + +After production issuance and a successful Traefik rotation, remove the +`wildcard_cert` and `wildcard_key` sops declarations from +`nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the +renewed host files. Remove the obsolete encrypted values from the private +`cc-ci-secrets` repository only after rollback is no longer needed. + +### 8. Lock registration and prove unattended renewal + +In a follow-up Nix change, set `api.disable_registration = true`, test that the +existing account can still update its TXT record, and confirm `/register` is +rejected. Then verify: + +```bash +systemctl list-timers 'acme-*' +systemctl start acme-ci.commoninternet.net.service +journalctl -u acme-ci.commoninternet.net.service -b +``` + +Perform a controlled staging renewal after registration is disabled, observe +the renewed Traefik secret version, and confirm the certificate served through +the gateway has the expected names and a new validity window. + +## Final acceptance checklist + +- [ ] cc-ci and the orchestrator contain no Gandi API credential. +- [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci. +- [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone. +- [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public. +- [ ] External UDP and TCP authoritative DNS checks pass. +- [ ] Registration is disabled after the one account is created. +- [ ] The ACME account can update only its generated TXT record. +- [ ] The certificate covers both `ci.commoninternet.net` and its wildcard. +- [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior + version until the replacement passes health checks. +- [ ] No credential or private key has been committed, logged, or written into + the Nix store.