# Plan: restricted ACME DNS renewal for cc-ci ## Outcome Replace the manually issued, sops-stored wildcard certificate with unattended DNS-01 renewal for these exact names: ```text ci.commoninternet.net *.ci.commoninternet.net ``` The cc-ci host will run an authoritative `acme-dns` instance only for `acme.commoninternet.net`. It will never receive a Gandi credential or any credential that can edit the parent `commoninternet.net` zone. The only enduring delegation from the parent zone is: ```text _acme-challenge.ci.commoninternet.net. CNAME .acme.commoninternet.net. ``` That CNAME authorizes the generated acme-dns account to answer ACME TXT challenges for the ci wildcard, not to edit any parent-zone DNS record. ## Project facts and constraints - The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6. - The wildcard currently points at the public gateway, which TLS-passthroughs to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point directly to `91.98.47.73`; the gateway is not involved in DNS. - Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and 443 only; any Hetzner Cloud firewall must also be checked before deployment. - TLS terminates in the Docker Swarm Traefik service. It currently reads `ssl_cert` and `ssl_key` **Swarm secrets** populated from `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by `runner/warm_reconcile.py`. A normal host-service reload cannot install a renewed certificate. - The existing certificate is expired: its served validity ended `2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until the new production certificate and Traefik rotation have both been verified. - `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS `security.acme` uses Lego and supports a DNS provider plus an environment file and post-renew hook. Confirm the pinned provider spelling with `lego --help` during implementation; Lego's current documented provider code is `acmedns`. ## Security invariants 1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any parent-zone update credential on cc-ci or the orchestrator. 2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP because it is loopback-only; do not create a circular API TLS dependency. 3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone. 4. The generated acme-dns account data is a secret. Keep it as a root/acme-only persistent state file under `/var/lib/acme/`; never put it in Nix text, the Nix store, git, `.env.public`, or a log. 5. After the account exists, set `disable_registration = true`. The existing account must still be able to call `/update`. 6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`. This is defence in depth in addition to the loopback API binding. ## Intended DNS design Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue ambiguity: ```text ns-acme.commoninternet.net. A 91.98.47.73 acme.commoninternet.net. NS ns-acme.commoninternet.net. ``` `acme-dns` itself serves the delegated zone and returns its matching NS record: ```text acme.commoninternet.net. NS ns-acme.commoninternet.net. ``` This host is authoritative for `acme.commoninternet.net` and its generated children only. It is not authoritative for `ci.commoninternet.net` or for `commoninternet.net`. ## Implementation phases ### 1. Preflight and safety checks Before changing Nix configuration, record: ```bash ssh cc-ci 'ss -lntup "( sport = :53 )"' ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"' ssh cc-ci 'nft list ruleset' ssh cc-ci 'docker service ls' ``` Confirm that no service owns port 53, that the Traefik Swarm services are healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and 53/udp. Do not replace an existing DNS service. Obtain the operator's ACME contact email before enabling `security.acme`. ### 2. Add a dedicated acme-dns Nix module Create `nix/modules/acme-dns.nix` and import it from `nix/hosts/cc-ci-hetzner/configuration.nix`. The module should: - create a dedicated unprivileged `acme-dns` user and group; - run `${pkgs.acme-dns}/bin/acme-dns -c ` with a persistent working/state directory `/var/lib/acme-dns`; - grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53; - use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`; - bind DNS to `91.98.47.73:53` with `protocol = "both4"`; - set `domain = "acme.commoninternet.net"`, `nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style `nsadmin` value; - include the public NS record above in `general.records`; - bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive CORS list, and initially leave `disable_registration = false`; - use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`, `ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the state directory as writable; and - open `networking.firewall.allowedTCPPorts = [ 53 ]` and `allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration. The configuration file is public data and may be generated by Nix. It must not contain account credentials. Deploy this phase with the normal cc-ci deployment discipline: first `nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and the host remain healthy, then run the identical `switch` target. Verify local DNS on both transports: ```bash dig @91.98.47.73 acme.commoninternet.net NS dig +tcp @91.98.47.73 acme.commoninternet.net NS ``` ### 3. Operator gate: delegate the narrow DNS zone After the service is healthy, ask the operator to add exactly these records at Gandi (using its DNS UI, never a token on this host): ```dns ns-acme.commoninternet.net. A 91.98.47.73 acme.commoninternet.net. NS ns-acme.commoninternet.net. ``` If Gandi models delegation as a nameserver/glue form rather than ordinary zone records, use its equivalent UI flow. Do not proceed until public recursive DNS shows the delegation and direct queries work from an external network: ```bash dig NS acme.commoninternet.net @1.1.1.1 dig TXT test.acme.commoninternet.net @91.98.47.73 dig +tcp TXT test.acme.commoninternet.net @91.98.47.73 ``` ### 4. Configure NixOS ACME in staging mode and obtain the account target Extend the new module with one `security.acme.certs` entry for the base name `ci.commoninternet.net`: ```nix { domain = "ci.commoninternet.net"; extraDomainNames = [ "*.ci.commoninternet.net" ]; dnsProvider = "acmedns"; # verify against the pinned Lego binary environmentFile = "/etc/acme-dns/lego.env"; dnsResolver = "1.1.1.1:53"; } ``` `/etc/acme-dns/lego.env` contains only non-secret wiring: ```text ACME_DNS_API_BASE=http://127.0.0.1:8080 ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json ACME_DNS_ALLOWLIST=127.0.0.1/32 ``` Lego registers and persists its per-domain acme-dns account in the storage path. The path is writable only by the ACME service user and is not Nix-managed content. Do not hand-create its JSON: let the pinned Lego provider establish the account format. Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate unit manually after the NS delegation is confirmed. The first staging run is expected to create the account and may fail validation because the CNAME is not yet present. Read the storage file only with a root-only helper that prints the generated **fulldomain** and never its username or password. ### 5. Operator gate: permanent challenge CNAME Ask the operator to create the exact target reported in phase 4: ```dns _acme-challenge.ci.commoninternet.net. CNAME .acme.commoninternet.net. ``` This is a permanent record. It must not be created, changed, or removed by an agent. Confirm the complete chain through a public recursive resolver before continuing: ```bash dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1 dig TXT .acme.commoninternet.net @91.98.47.73 dig +tcp TXT .acme.commoninternet.net @91.98.47.73 ``` ### 6. Staging issuance, then production issuance Run the NixOS ACME certificate unit against staging and verify all of the following: 1. it updates only the generated acme-dns TXT target; 2. public recursive DNS sees the CNAME and the TXT value; 3. staging issues a certificate containing both requested names; and 4. no Gandi variable, credential file, or API request appears in the unit. Only then select the production Let's Encrypt directory and issue the real certificate. Keep the old sops certificate live during both attempts. ### 7. Make Traefik consume renewals safely Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and cannot see an updated host file. Add a root-only renewal handoff service, serialized with all other Traefik reconciliation, and call it from the ACME certificate's `postRun` hook. The handoff must: 1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes; 2. generate a new, content-derived **non-secret** Swarm secret version; 3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe environment to reference those versions, and reconcile/redeploy Traefik; 4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI; 5. retain the prior secret version until the new task is healthy, then remove it; and 6. record a failure clearly without deleting the last-known-good certificate. Implement this as a tested extension of `runner/warm_reconcile.py` (or a small, explicitly locked companion) rather than an ad-hoc shell command. The renewal path and the normal `deploy-proxy` path must share a lock so they cannot race over Swarm secret versions. After production issuance and a successful Traefik rotation, remove the `wildcard_cert` and `wildcard_key` sops declarations from `nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the renewed host files. Remove the obsolete encrypted values from the private `cc-ci-secrets` repository only after rollback is no longer needed. ### 8. Lock registration and prove unattended renewal In a follow-up Nix change, set `api.disable_registration = true`, test that the existing account can still update its TXT record, and confirm `/register` is rejected. Then verify: ```bash systemctl list-timers 'acme-*' systemctl start acme-ci.commoninternet.net.service journalctl -u acme-ci.commoninternet.net.service -b ``` Perform a controlled staging renewal after registration is disabled, observe the renewed Traefik secret version, and confirm the certificate served through the gateway has the expected names and a new validity window. ## Final acceptance checklist - [ ] cc-ci and the orchestrator contain no Gandi API credential. - [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci. - [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone. - [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public. - [ ] External UDP and TCP authoritative DNS checks pass. - [ ] Registration is disabled after the one account is created. - [ ] The ACME account can update only its generated TXT record. - [ ] The certificate covers both `ci.commoninternet.net` and its wildcard. - [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior version until the replacement passes health checks. - [ ] No credential or private key has been committed, logged, or written into the Nix store.