cc-ci/docs/acme-dns-renewal-plan.md

# Plan: restricted ACME DNS renewal for cc-ci

## Outcome

Replace the manually issued, sops-stored wildcard certificate with unattended
DNS-01 renewal for these exact names:

```text
ci.commoninternet.net
*.ci.commoninternet.net
```

The cc-ci host will run an authoritative `acme-dns` instance only for
`acme.commoninternet.net`. It will never receive a Gandi credential or any
credential that can edit the parent `commoninternet.net` zone.

The only enduring delegation from the parent zone is:

```text
_acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.
```

That CNAME authorizes the generated acme-dns account to answer ACME TXT
challenges for the ci wildcard, not to edit any parent-zone DNS record.

## Project facts and constraints

- The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the
  orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6.
- The wildcard currently points at the public gateway, which TLS-passthroughs
  to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point
  directly to `91.98.47.73`; the gateway is not involved in DNS.
- Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and
  443 only; any Hetzner Cloud firewall must also be checked before deployment.
- TLS terminates in the Docker Swarm Traefik service. It currently reads
  `ssl_cert` and `ssl_key` **Swarm secrets** populated from
  `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by
  `runner/warm_reconcile.py`. A normal host-service reload cannot install a
  renewed certificate.
- The existing certificate is expired: its served validity ended
  `2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until
  the new production certificate and Traefik rotation have both been verified.
- `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS
  `security.acme` uses Lego and supports a DNS provider plus an environment
  file and post-renew hook. Confirm the pinned provider spelling with
  `lego --help` during implementation; Lego's current documented provider code
  is `acmedns`.

## Security invariants

1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any
   parent-zone update credential on cc-ci or the orchestrator.
2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP
   because it is loopback-only; do not create a circular API TLS dependency.
3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
4. The generated acme-dns account data is a secret. Keep it as a root/acme-only
   persistent state file under `/var/lib/acme/`; never put it in Nix text, the
   Nix store, git, `.env.public`, or a log.
5. After the account exists, set `disable_registration = true`. The existing
   account must still be able to call `/update`.
6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`.
   This is defence in depth in addition to the loopback API binding.

## Intended DNS design

Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue
ambiguity:

```text
ns-acme.commoninternet.net.  A   91.98.47.73
acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
```

`acme-dns` itself serves the delegated zone and returns its matching NS record:

```text
acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
```

This host is authoritative for `acme.commoninternet.net` and its generated
children only. It is not authoritative for `ci.commoninternet.net` or for
`commoninternet.net`.

## Implementation phases

### 1. Preflight and safety checks

Before changing Nix configuration, record:

```bash
ssh cc-ci 'ss -lntup "( sport = :53 )"'
ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
ssh cc-ci 'nft list ruleset'
ssh cc-ci 'docker service ls'
```

Confirm that no service owns port 53, that the Traefik Swarm services are
healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and
53/udp. Do not replace an existing DNS service.

Obtain the operator's ACME contact email before enabling `security.acme`.

### 2. Add a dedicated acme-dns Nix module

Create `nix/modules/acme-dns.nix` and import it from
`nix/hosts/cc-ci-hetzner/configuration.nix`. The module should:

- create a dedicated unprivileged `acme-dns` user and group;
- run `${pkgs.acme-dns}/bin/acme-dns -c <public generated config>` with a
  persistent working/state directory `/var/lib/acme-dns`;
- grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53;
- use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`;
- bind DNS to `91.98.47.73:53` with `protocol = "both4"`;
- set `domain = "acme.commoninternet.net"`,
  `nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style
  `nsadmin` value;
- include the public NS record above in `general.records`;
- bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive
  CORS list, and initially leave `disable_registration = false`;
- use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`,
  `ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the
  state directory as writable; and
- open `networking.firewall.allowedTCPPorts = [ 53 ]` and
  `allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration.

The configuration file is public data and may be generated by Nix. It must not
contain account credentials.

Deploy this phase with the normal cc-ci deployment discipline: first
`nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and
the host remain healthy, then run the identical `switch` target. Verify local
DNS on both transports:

```bash
dig @91.98.47.73 acme.commoninternet.net NS
dig +tcp @91.98.47.73 acme.commoninternet.net NS
```

### 3. Operator gate: delegate the narrow DNS zone

After the service is healthy, ask the operator to add exactly these records at
Gandi (using its DNS UI, never a token on this host):

```dns
ns-acme.commoninternet.net.  A   91.98.47.73
acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
```

If Gandi models delegation as a nameserver/glue form rather than ordinary zone
records, use its equivalent UI flow. Do not proceed until public recursive DNS
shows the delegation and direct queries work from an external network:

```bash
dig NS acme.commoninternet.net @1.1.1.1
dig TXT test.acme.commoninternet.net @91.98.47.73
dig +tcp TXT test.acme.commoninternet.net @91.98.47.73
```

### 4. Configure NixOS ACME in staging mode and obtain the account target

Extend the new module with one `security.acme.certs` entry for the base name
`ci.commoninternet.net`:

```nix
{
  domain = "ci.commoninternet.net";
  extraDomainNames = [ "*.ci.commoninternet.net" ];
  dnsProvider = "acmedns"; # verify against the pinned Lego binary
  environmentFile = "/etc/acme-dns/lego.env";
  dnsResolver = "1.1.1.1:53";
}
```

`/etc/acme-dns/lego.env` contains only non-secret wiring:

```text
ACME_DNS_API_BASE=http://127.0.0.1:8080
ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
ACME_DNS_ALLOWLIST=127.0.0.1/32
```

Lego registers and persists its per-domain acme-dns account in the storage
path. The path is writable only by the ACME service user and is not Nix-managed
content. Do not hand-create its JSON: let the pinned Lego provider establish
the account format.

Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate
unit manually after the NS delegation is confirmed. The first staging run is
expected to create the account and may fail validation because the CNAME is not
yet present. Read the storage file only with a root-only helper that prints the
generated **fulldomain** and never its username or password.

### 5. Operator gate: permanent challenge CNAME

Ask the operator to create the exact target reported in phase 4:

```dns
_acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.
```

This is a permanent record. It must not be created, changed, or removed by an
agent. Confirm the complete chain through a public recursive resolver before
continuing:

```bash
dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73
```

### 6. Staging issuance, then production issuance

Run the NixOS ACME certificate unit against staging and verify all of the
following:

1. it updates only the generated acme-dns TXT target;
2. public recursive DNS sees the CNAME and the TXT value;
3. staging issues a certificate containing both requested names; and
4. no Gandi variable, credential file, or API request appears in the unit.

Only then select the production Let's Encrypt directory and issue the real
certificate. Keep the old sops certificate live during both attempts.

### 7. Make Traefik consume renewals safely

Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and
cannot see an updated host file. Add a root-only renewal handoff service,
serialized with all other Traefik reconciliation, and call it from the ACME
certificate's `postRun` hook.

The handoff must:

1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output
   into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes;
2. generate a new, content-derived **non-secret** Swarm secret version;
3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe
   environment to reference those versions, and reconcile/redeploy Traefik;
4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI;
5. retain the prior secret version until the new task is healthy, then remove
   it; and
6. record a failure clearly without deleting the last-known-good certificate.

Implement this as a tested extension of `runner/warm_reconcile.py` (or a
small, explicitly locked companion) rather than an ad-hoc shell command. The
renewal path and the normal `deploy-proxy` path must share a lock so they cannot
race over Swarm secret versions.

After production issuance and a successful Traefik rotation, remove the
`wildcard_cert` and `wildcard_key` sops declarations from
`nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the
renewed host files. Remove the obsolete encrypted values from the private
`cc-ci-secrets` repository only after rollback is no longer needed.

### 8. Lock registration and prove unattended renewal

In a follow-up Nix change, set `api.disable_registration = true`, test that the
existing account can still update its TXT record, and confirm `/register` is
rejected. Then verify:

```bash
systemctl list-timers 'acme-*'
systemctl start acme-ci.commoninternet.net.service
journalctl -u acme-ci.commoninternet.net.service -b
```

Perform a controlled staging renewal after registration is disabled, observe
the renewed Traefik secret version, and confirm the certificate served through
the gateway has the expected names and a new validity window.

## Final acceptance checklist

- [ ] cc-ci and the orchestrator contain no Gandi API credential.
- [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci.
- [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone.
- [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
- [ ] External UDP and TCP authoritative DNS checks pass.
- [ ] Registration is disabled after the one account is created.
- [ ] The ACME account can update only its generated TXT record.
- [ ] The certificate covers both `ci.commoninternet.net` and its wildcard.
- [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior
      version until the replacement passes health checks.
- [ ] No credential or private key has been committed, logged, or written into
      the Nix store.