recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
7147d777eedef74cf0986814642e9d804ae3c88e
cc-ci/docs/acme-dns-renewal-plan.md
T

12 KiB
Raw Blame History

Plan: restricted ACME DNS renewal for cc-ci

Outcome

Replace the manually issued, sops-stored wildcard certificate with unattended DNS-01 renewal for these exact names:

ci.commoninternet.net
*.ci.commoninternet.net

The cc-ci host will run an authoritative acme-dns instance only for acme.commoninternet.net. It will never receive a Gandi credential or any credential that can edit the parent commoninternet.net zone.

The only enduring delegation from the parent zone is:

_acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.

That CNAME authorizes the generated acme-dns account to answer ACME TXT challenges for the ci wildcard, not to edit any parent-zone DNS record.

Project facts and constraints

  • The target is the production cc-ci-hetzner NixOS 26.05 host, not the orchestrator. Its public IPv4 is 91.98.47.73; it has no public IPv6.
  • The wildcard currently points at the public gateway, which TLS-passthroughs to cc-ci's Traefik. DNS authority for acme.commoninternet.net must point directly to 91.98.47.73; the gateway is not involved in DNS.
  • Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and 443 only; any Hetzner Cloud firewall must also be checked before deployment.
  • TLS terminates in the Docker Swarm Traefik service. It currently reads ssl_cert and ssl_key Swarm secrets populated from /var/lib/ci-certs/live/{fullchain.pem,privkey.pem} by runner/warm_reconcile.py. A normal host-service reload cannot install a renewed certificate.
  • The existing certificate is expired: its served validity ended 2026-08-24 18:18:52 UTC. Keep the current files as rollback material until the new production certificate and Traefik rotation have both been verified.
  • pkgs.acme-dns and pkgs.lego are available in the pinned nixpkgs. NixOS security.acme uses Lego and supports a DNS provider plus an environment file and post-renew hook. Confirm the pinned provider spelling with lego --help during implementation; Lego's current documented provider code is acmedns.

Security invariants

  1. Do not request, add, store, or use GANDI_API_KEY, a Gandi PAT, or any parent-zone update credential on cc-ci or the orchestrator.
  2. Bind the acme-dns HTTP API to 127.0.0.1 only. Its API may use plain HTTP because it is loopback-only; do not create a circular API TLS dependency.
  3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
  4. The generated acme-dns account data is a secret. Keep it as a root/acme-only persistent state file under /var/lib/acme/; never put it in Nix text, the Nix store, git, .env.public, or a log.
  5. After the account exists, set disable_registration = true. The existing account must still be able to call /update.
  6. Limit the account's update source with ACME_DNS_ALLOWLIST=127.0.0.1/32. This is defence in depth in addition to the loopback API binding.

Intended DNS design

Use an out-of-bailiwick nameserver name to avoid in-bailiwick glue ambiguity:

ns-acme.commoninternet.net.  A   91.98.47.73
acme.commoninternet.net.     NS  ns-acme.commoninternet.net.

acme-dns itself serves the delegated zone and returns its matching NS record:

acme.commoninternet.net.     NS  ns-acme.commoninternet.net.

This host is authoritative for acme.commoninternet.net and its generated children only. It is not authoritative for ci.commoninternet.net or for commoninternet.net.

Implementation phases

1. Preflight and safety checks

Before changing Nix configuration, record:

ssh cc-ci 'ss -lntup "( sport = :53 )"'
ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
ssh cc-ci 'nft list ruleset'
ssh cc-ci 'docker service ls'

Confirm that no service owns port 53, that the Traefik Swarm services are healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and 53/udp. Do not replace an existing DNS service.

Obtain the operator's ACME contact email before enabling security.acme.

2. Add a dedicated acme-dns Nix module

Create nix/modules/acme-dns.nix and import it from nix/hosts/cc-ci-hetzner/configuration.nix. The module should:

  • create a dedicated unprivileged acme-dns user and group;
  • run ${pkgs.acme-dns}/bin/acme-dns -c <public generated config> with a persistent working/state directory /var/lib/acme-dns;
  • grant only CAP_NET_BIND_SERVICE to bind DNS port 53;
  • use SQLite at /var/lib/acme-dns/acme-dns.db with mode 0600;
  • bind DNS to 91.98.47.73:53 with protocol = "both4";
  • set domain = "acme.commoninternet.net", nsname = "ns-acme.commoninternet.net", and a public hostmaster-style nsadmin value;
  • include the public NS record above in general.records;
  • bind [api] to 127.0.0.1:8080, set tls = "none", use a restrictive CORS list, and initially leave disable_registration = false;
  • use a hardened systemd unit: NoNewPrivileges, PrivateTmp, ProtectSystem = "strict", ProtectHome, PrivateDevices, and only the state directory as writable; and
  • open networking.firewall.allowedTCPPorts = [ 53 ] and allowedUDPPorts = [ 53 ] in the cc-ci Hetzner host configuration.

The configuration file is public data and may be generated by Nix. It must not contain account credentials.

Deploy this phase with the normal cc-ci deployment discipline: first nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner, verify SSH, Traefik, and the host remain healthy, then run the identical switch target. Verify local DNS on both transports:

dig @91.98.47.73 acme.commoninternet.net NS
dig +tcp @91.98.47.73 acme.commoninternet.net NS

3. Operator gate: delegate the narrow DNS zone

After the service is healthy, ask the operator to add exactly these records at Gandi (using its DNS UI, never a token on this host):

ns-acme.commoninternet.net.  A   91.98.47.73
acme.commoninternet.net.     NS  ns-acme.commoninternet.net.

If Gandi models delegation as a nameserver/glue form rather than ordinary zone records, use its equivalent UI flow. Do not proceed until public recursive DNS shows the delegation and direct queries work from an external network:

dig NS acme.commoninternet.net @1.1.1.1
dig TXT test.acme.commoninternet.net @91.98.47.73
dig +tcp TXT test.acme.commoninternet.net @91.98.47.73

4. Configure NixOS ACME in staging mode and obtain the account target

Extend the new module with one security.acme.certs entry for the base name ci.commoninternet.net:

{
  domain = "ci.commoninternet.net";
  extraDomainNames = [ "*.ci.commoninternet.net" ];
  dnsProvider = "acmedns"; # verify against the pinned Lego binary
  environmentFile = "/etc/acme-dns/lego.env";
  dnsResolver = "1.1.1.1:53";
}

/etc/acme-dns/lego.env contains only non-secret wiring:

ACME_DNS_API_BASE=http://127.0.0.1:8080
ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
ACME_DNS_ALLOWLIST=127.0.0.1/32

Lego registers and persists its per-domain acme-dns account in the storage path. The path is writable only by the ACME service user and is not Nix-managed content. Do not hand-create its JSON: let the pinned Lego provider establish the account format.

Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate unit manually after the NS delegation is confirmed. The first staging run is expected to create the account and may fail validation because the CNAME is not yet present. Read the storage file only with a root-only helper that prints the generated fulldomain and never its username or password.

5. Operator gate: permanent challenge CNAME

Ask the operator to create the exact target reported in phase 4:

_acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.

This is a permanent record. It must not be created, changed, or removed by an agent. Confirm the complete chain through a public recursive resolver before continuing:

dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73

6. Staging issuance, then production issuance

Run the NixOS ACME certificate unit against staging and verify all of the following:

  1. it updates only the generated acme-dns TXT target;
  2. public recursive DNS sees the CNAME and the TXT value;
  3. staging issues a certificate containing both requested names; and
  4. no Gandi variable, credential file, or API request appears in the unit.

Only then select the production Let's Encrypt directory and issue the real certificate. Keep the old sops certificate live during both attempts.

7. Make Traefik consume renewals safely

Do not use only reloadServices: Traefik receives Docker Swarm secrets and cannot see an updated host file. Add a root-only renewal handoff service, serialized with all other Traefik reconciliation, and call it from the ACME certificate's postRun hook.

The handoff must:

  1. atomically copy the new fullchain.pem and key from the NixOS ACME output into /var/lib/ci-certs/live, with the existing 0444/0400 modes;
  2. generate a new, content-derived non-secret Swarm secret version;
  3. insert new ssl_cert and ssl_key Swarm secrets, update the Traefik recipe environment to reference those versions, and reconcile/redeploy Traefik;
  4. health-check https://traefik.ci.commoninternet.net/api/version with SNI;
  5. retain the prior secret version until the new task is healthy, then remove it; and
  6. record a failure clearly without deleting the last-known-good certificate.

Implement this as a tested extension of runner/warm_reconcile.py (or a small, explicitly locked companion) rather than an ad-hoc shell command. The renewal path and the normal deploy-proxy path must share a lock so they cannot race over Swarm secret versions.

After production issuance and a successful Traefik rotation, remove the wildcard_cert and wildcard_key sops declarations from nix/modules/secrets.nix; otherwise later Nix activations would overwrite the renewed host files. Remove the obsolete encrypted values from the private cc-ci-secrets repository only after rollback is no longer needed.

8. Lock registration and prove unattended renewal

In a follow-up Nix change, set api.disable_registration = true, test that the existing account can still update its TXT record, and confirm /register is rejected. Then verify:

systemctl list-timers 'acme-*'
systemctl start acme-ci.commoninternet.net.service
journalctl -u acme-ci.commoninternet.net.service -b

Perform a controlled staging renewal after registration is disabled, observe the renewed Traefik secret version, and confirm the certificate served through the gateway has the expected names and a new validity window.

Final acceptance checklist

  • cc-ci and the orchestrator contain no Gandi API credential.
  • Gandi delegates only acme.commoninternet.net to cc-ci.
  • Only _acme-challenge.ci.commoninternet.net CNAMEs into that zone.
  • The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
  • External UDP and TCP authoritative DNS checks pass.
  • Registration is disabled after the one account is created.
  • The ACME account can update only its generated TXT record.
  • The certificate covers both ci.commoninternet.net and its wildcard.
  • A renewal rotates Traefik's Swarm secrets and preserves a working prior version until the replacement passes health checks.
  • No credential or private key has been committed, logged, or written into the Nix store.
Reference in New Issue View Git Blame Copy Permalink