fix: roll out rotated traefik certificate secrets

Route deepseek-v4-flash through the opencode zen subscription endpoint
(opencode/deepseek-v4-flash) instead of tinfoil pay-per-use. The zen
subscription is already paid for; flash consumes the balance far more
slowly than glm-5.2 did (flash cost ~$0.0000002/tok vs glm's higher rate),
so a full weekly run is much less likely to exhaust the balance — and
the operator has reset it.

Completes the fleet-wide move: upgrader parent (LOOP_MODEL) + report
(REPORT_MODEL) in upgrader.env, and this PR sets the subagents. Supervisor
stays on opencode-go/glm-5.2.

Verified: zen endpoint serves deepseek-v4-flash (HTTP 200, 1.2s).

.env.public

@@ -0,0 +1,7 @@
+# Non-sensitive runtime configuration shared by the cc-ci orchestrator and agents.
+#
+# Keep credentials, tokens, and keys in /srv/cc-ci/.testenv.  The orchestrator
+# loads this file first via cc-ci-plan/load-env.sh.
+GITEA_USERNAME=autonomic-bot
+TINFOIL_MODEL=deepseek-v4-pro
+TINFOIL_BASE_URL=https://inference.tinfoil.sh/v1

AGENTS.md

@@ -36,3 +36,19 @@ Two kinds of tests live here — run them on **different** cadences:
 
 A red test is information. Never skip, delete, or relax a test to make a run green — fix the root
 cause or record it in `machine-docs/DEFERRED.md`. (This is a standing build guardrail.)
+
+## Ship work as PRs, merge them yourself, operator reviews retrospectively
+
+Work on this repo goes: **branch → PR → merge it yourself once verified → operator reviews
+retrospectively.** Do not commit straight to `main`, and do not wait for review before merging — the
+invocation is the authorization, and blocking would stall the CI this repo runs.
+
+The PR is therefore not a gate; it is how the work stays legible after the fact. Write the
+description to be read later: what changed, why, and the evidence it works (harness output, a
+verified run, a before/after number). A PR that says "fix test" has failed at its only job.
+
+The same policy covers `recipe-maintainers/cc-ci-orchestrator`. It does **NOT** cover recipe repos —
+any `coop-cloud/<recipe>` or its mirror is created and verified but **never agent-merged**, because
+those change what deploys on other people's infrastructure.
+
+Before editing a test, read `tests/STYLE.md`.

cc-ci-plan/upstream/gitea.md

@@ -0,0 +1,19 @@
+# gitea upstream sources
+
+## gitea/gitea
+- image: gitea/gitea
+- source: https://github.com/go-gitea/gitea
+- releases: https://github.com/go-gitea/gitea/releases
+- security: https://blog.gitea.com/
+
+## postgres
+- image: postgres
+- source: https://github.com/postgres/postgres
+- releases: https://www.postgresql.org/docs/release/
+- security: https://www.postgresql.org/support/security/
+
+## mariadb
+- image: mariadb
+- source: https://github.com/MariaDB/server
+- releases: https://mariadb.com/kb/en/release-notes/
+- security: https://mariadb.com/kb/en/security/

docs/acme-dns-renewal-plan.md

@@ -0,0 +1,283 @@
+# Plan: restricted ACME DNS renewal for cc-ci
+
+## Outcome
+
+Replace the manually issued, sops-stored wildcard certificate with unattended
+DNS-01 renewal for these exact names:
+
+```text
+ci.commoninternet.net
+*.ci.commoninternet.net
+```
+
+The cc-ci host will run an authoritative `acme-dns` instance only for
+`acme.commoninternet.net`. It will never receive a Gandi credential or any
+credential that can edit the parent `commoninternet.net` zone.
+
+The only enduring delegation from the parent zone is:
+
+```text
+_acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.
+```
+
+That CNAME authorizes the generated acme-dns account to answer ACME TXT
+challenges for the ci wildcard, not to edit any parent-zone DNS record.
+
+## Project facts and constraints
+
+- The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the
+  orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6.
+- The wildcard currently points at the public gateway, which TLS-passthroughs
+  to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point
+  directly to `91.98.47.73`; the gateway is not involved in DNS.
+- Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and
+  443 only; any Hetzner Cloud firewall must also be checked before deployment.
+- TLS terminates in the Docker Swarm Traefik service. It currently reads
+  `ssl_cert` and `ssl_key` **Swarm secrets** populated from
+  `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by
+  `runner/warm_reconcile.py`. A normal host-service reload cannot install a
+  renewed certificate.
+- The existing certificate is expired: its served validity ended
+  `2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until
+  the new production certificate and Traefik rotation have both been verified.
+- `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS
+  `security.acme` uses Lego and supports a DNS provider plus an environment
+  file and post-renew hook. Confirm the pinned provider spelling with
+  `lego --help` during implementation; Lego's current documented provider code
+  is `acmedns`.
+
+## Security invariants
+
+1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any
+   parent-zone update credential on cc-ci or the orchestrator.
+2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP
+   because it is loopback-only; do not create a circular API TLS dependency.
+3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
+4. The generated acme-dns account data is a secret. Keep it as a root/acme-only
+   persistent state file under `/var/lib/acme/`; never put it in Nix text, the
+   Nix store, git, `.env.public`, or a log.
+5. After the account exists, set `disable_registration = true`. The existing
+   account must still be able to call `/update`.
+6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`.
+   This is defence in depth in addition to the loopback API binding.
+
+## Intended DNS design
+
+Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue
+ambiguity:
+
+```text
+ns-acme.commoninternet.net.  A   91.98.47.73
+acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
+```
+
+`acme-dns` itself serves the delegated zone and returns its matching NS record:
+
+```text
+acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
+```
+
+This host is authoritative for `acme.commoninternet.net` and its generated
+children only. It is not authoritative for `ci.commoninternet.net` or for
+`commoninternet.net`.
+
+## Implementation phases
+
+### 1. Preflight and safety checks
+
+Before changing Nix configuration, record:
+
+```bash
+ssh cc-ci 'ss -lntup "( sport = :53 )"'
+ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
+ssh cc-ci 'nft list ruleset'
+ssh cc-ci 'docker service ls'
+```
+
+Confirm that no service owns port 53, that the Traefik Swarm services are
+healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and
+53/udp. Do not replace an existing DNS service.
+
+Obtain the operator's ACME contact email before enabling `security.acme`.
+
+### 2. Add a dedicated acme-dns Nix module
+
+Create `nix/modules/acme-dns.nix` and import it from
+`nix/hosts/cc-ci-hetzner/configuration.nix`. The module should:
+
+- create a dedicated unprivileged `acme-dns` user and group;
+- run `${pkgs.acme-dns}/bin/acme-dns -c <public generated config>` with a
+  persistent working/state directory `/var/lib/acme-dns`;
+- grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53;
+- use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`;
+- bind DNS to `91.98.47.73:53` with `protocol = "both4"`;
+- set `domain = "acme.commoninternet.net"`,
+  `nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style
+  `nsadmin` value;
+- include the public NS record above in `general.records`;
+- bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive
+  CORS list, and initially leave `disable_registration = false`;
+- use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`,
+  `ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the
+  state directory as writable; and
+- open `networking.firewall.allowedTCPPorts = [ 53 ]` and
+  `allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration.
+
+The configuration file is public data and may be generated by Nix. It must not
+contain account credentials.
+
+Deploy this phase with the normal cc-ci deployment discipline: first
+`nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and
+the host remain healthy, then run the identical `switch` target. Verify local
+DNS on both transports:
+
+```bash
+dig @91.98.47.73 acme.commoninternet.net NS
+dig +tcp @91.98.47.73 acme.commoninternet.net NS
+```
+
+### 3. Operator gate: delegate the narrow DNS zone
+
+After the service is healthy, ask the operator to add exactly these records at
+Gandi (using its DNS UI, never a token on this host):
+
+```dns
+ns-acme.commoninternet.net.  A   91.98.47.73
+acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
+```
+
+If Gandi models delegation as a nameserver/glue form rather than ordinary zone
+records, use its equivalent UI flow. Do not proceed until public recursive DNS
+shows the delegation and direct queries work from an external network:
+
+```bash
+dig NS acme.commoninternet.net @1.1.1.1
+dig TXT test.acme.commoninternet.net @91.98.47.73
+dig +tcp TXT test.acme.commoninternet.net @91.98.47.73
+```
+
+### 4. Configure NixOS ACME in staging mode and obtain the account target
+
+Extend the new module with one `security.acme.certs` entry for the base name
+`ci.commoninternet.net`:
+
+```nix
+{
+  domain = "ci.commoninternet.net";
+  extraDomainNames = [ "*.ci.commoninternet.net" ];
+  dnsProvider = "acmedns"; # verify against the pinned Lego binary
+  environmentFile = "/etc/acme-dns/lego.env";
+  dnsResolver = "1.1.1.1:53";
+}
+```
+
+`/etc/acme-dns/lego.env` contains only non-secret wiring:
+
+```text
+ACME_DNS_API_BASE=http://127.0.0.1:8080
+ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
+ACME_DNS_ALLOWLIST=127.0.0.1/32
+```
+
+Lego registers and persists its per-domain acme-dns account in the storage
+path. The path is writable only by the ACME service user and is not Nix-managed
+content. Do not hand-create its JSON: let the pinned Lego provider establish
+the account format.
+
+Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate
+unit manually after the NS delegation is confirmed. The first staging run is
+expected to create the account and may fail validation because the CNAME is not
+yet present. Read the storage file only with a root-only helper that prints the
+generated **fulldomain** and never its username or password.
+
+### 5. Operator gate: permanent challenge CNAME
+
+Ask the operator to create the exact target reported in phase 4:
+
+```dns
+_acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.
+```
+
+This is a permanent record. It must not be created, changed, or removed by an
+agent. Confirm the complete chain through a public recursive resolver before
+continuing:
+
+```bash
+dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
+dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
+dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73
+```
+
+### 6. Staging issuance, then production issuance
+
+Run the NixOS ACME certificate unit against staging and verify all of the
+following:
+
+1. it updates only the generated acme-dns TXT target;
+2. public recursive DNS sees the CNAME and the TXT value;
+3. staging issues a certificate containing both requested names; and
+4. no Gandi variable, credential file, or API request appears in the unit.
+
+Only then select the production Let's Encrypt directory and issue the real
+certificate. Keep the old sops certificate live during both attempts.
+
+### 7. Make Traefik consume renewals safely
+
+Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and
+cannot see an updated host file. Add a root-only renewal handoff service,
+serialized with all other Traefik reconciliation, and call it from the ACME
+certificate's `postRun` hook.
+
+The handoff must:
+
+1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output
+   into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes;
+2. generate a new, content-derived **non-secret** Swarm secret version;
+3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe
+   environment to reference those versions, and reconcile/redeploy Traefik;
+4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI;
+5. retain the prior secret version until the new task is healthy, then remove
+   it; and
+6. record a failure clearly without deleting the last-known-good certificate.
+
+Implement this as a tested extension of `runner/warm_reconcile.py` (or a
+small, explicitly locked companion) rather than an ad-hoc shell command. The
+renewal path and the normal `deploy-proxy` path must share a lock so they cannot
+race over Swarm secret versions.
+
+After production issuance and a successful Traefik rotation, remove the
+`wildcard_cert` and `wildcard_key` sops declarations from
+`nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the
+renewed host files. Remove the obsolete encrypted values from the private
+`cc-ci-secrets` repository only after rollback is no longer needed.
+
+### 8. Lock registration and prove unattended renewal
+
+In a follow-up Nix change, set `api.disable_registration = true`, test that the
+existing account can still update its TXT record, and confirm `/register` is
+rejected. Then verify:
+
+```bash
+systemctl list-timers 'acme-*'
+systemctl start acme-ci.commoninternet.net.service
+journalctl -u acme-ci.commoninternet.net.service -b
+```
+
+Perform a controlled staging renewal after registration is disabled, observe
+the renewed Traefik secret version, and confirm the certificate served through
+the gateway has the expected names and a new validity window.
+
+## Final acceptance checklist
+
+- [ ] cc-ci and the orchestrator contain no Gandi API credential.
+- [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci.
+- [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone.
+- [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
+- [ ] External UDP and TCP authoritative DNS checks pass.
+- [ ] Registration is disabled after the one account is created.
+- [ ] The ACME account can update only its generated TXT record.
+- [ ] The certificate covers both `ci.commoninternet.net` and its wildcard.
+- [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior
+      version until the replacement passes health checks.
+- [ ] No credential or private key has been committed, logged, or written into
+      the Nix store.

docs/public-environment.md

@@ -0,0 +1,15 @@
+# Public runtime environment
+
+`.env.public` contains non-sensitive configuration that the cc-ci orchestrator
+and its agent sessions need at runtime. It is intentionally tracked so it can
+be inspected and reproduced with the rest of the CI configuration.
+
+Load it together with the local secret file by sourcing
+`/srv/cc-ci/cc-ci-plan/load-env.sh`. The helper reads `.env.public` first and
+then `/srv/cc-ci/.testenv`; credentials, tokens, and keys belong only in the
+latter file.
+
+Do not put a value in `.env.public` merely because it is convenient. If it
+would grant access or require rotation, it is a secret and belongs in
+`.testenv`. Public service endpoints, model names, and account identifiers may
+be tracked here.

nix/hosts/cc-ci-hetzner/configuration.nix

@@ -14,6 +14,7 @@
     ./networking.nix
     ../../modules/packages.nix
     ../../modules/secrets.nix
+    ../../modules/acme-dns.nix
     ../../modules/swarm.nix
     ../../modules/docker-prune.nix
     ../../modules/abra.nix

nix/modules/acme-dns.nix

@@ -0,0 +1,155 @@
+# Restricted DNS-01 certificate issuance for ci.commoninternet.net.
+#
+# This host is authoritative only for acme.commoninternet.net. Gandi continues
+# to own commoninternet.net; it delegates this narrow zone and one permanent
+# _acme-challenge CNAME manually. No Gandi credential is present here.
+{ pkgs, ... }:
+let
+  acmeDnsConfig = pkgs.writeText "cc-ci-acme-dns.conf" ''
+    [general]
+    listen = "91.98.47.73:53"
+    protocol = "both4"
+    domain = "acme.commoninternet.net"
+    nsname = "ns-acme.commoninternet.net"
+    nsadmin = "hostmaster.commoninternet.net"
+    records = [
+      "acme.commoninternet.net. NS ns-acme.commoninternet.net.",
+      "ns-acme.commoninternet.net. A 91.98.47.73",
+    ]
+    debug = false
+
+    [database]
+    # acme-dns 2.x registers the embedded driver under `sqlite` (not the
+    # legacy `sqlite3` identifier).
+    engine = "sqlite"
+    connection = "/var/lib/acme-dns/acme-dns.db"
+
+    [api]
+    ip = "127.0.0.1"
+    port = "8080"
+    tls = "none"
+    # The one Lego account was bootstrapped before this configuration was
+    # hardened. Updates authenticated by that account remain available.
+    disable_registration = true
+    corsorigins = []
+
+    [logconfig]
+    loglevel = "info"
+    logtype = "stdout"
+    logformat = "json"
+  '';
+
+  # These are wiring values only. The acme-dns account JSON is generated by
+  # Lego below /var/lib/acme and never enters Nix, git, or /etc.
+  legoEnvironment = pkgs.writeText "cc-ci-acme-dns-lego.env" ''
+    ACME_DNS_API_BASE=http://127.0.0.1:8080
+    ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
+    ACME_DNS_ALLOWLIST=127.0.0.1/32
+  '';
+in
+{
+  users.groups.acme-dns = { };
+  users.users.acme-dns = {
+    isSystemUser = true;
+    group = "acme-dns";
+    home = "/var/lib/acme-dns";
+  };
+
+  environment.etc."acme-dns/lego.env".source = legoEnvironment;
+
+  # The staging order has completed successfully. This marker permits the
+  # production ACME post-run hook to hand a renewed certificate to Traefik.
+  systemd.tmpfiles.rules = [
+    "f /var/lib/ci-certs/acme-production-enabled 0600 root root -"
+  ];
+
+  networking.firewall = {
+    allowedTCPPorts = [ 53 ];
+    allowedUDPPorts = [ 53 ];
+  };
+
+  systemd.services.acme-dns = {
+    description = "Restricted authoritative DNS for cc-ci ACME DNS-01";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      User = "acme-dns";
+      Group = "acme-dns";
+      StateDirectory = "acme-dns";
+      StateDirectoryMode = "0700";
+      WorkingDirectory = "/var/lib/acme-dns";
+      ExecStart = "${pkgs.acme-dns}/bin/acme-dns -c ${acmeDnsConfig}";
+      Restart = "on-failure";
+      RestartSec = "5s";
+      AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
+      CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
+      NoNewPrivileges = true;
+      PrivateTmp = true;
+      PrivateDevices = true;
+      ProtectHome = true;
+      ProtectSystem = "strict";
+      ReadWritePaths = [ "/var/lib/acme-dns" ];
+      RestrictAddressFamilies = [ "AF_INET" "AF_UNIX" ];
+    };
+  };
+
+  # Traefik consumes its wildcard as immutable Swarm secrets, so a renewed
+  # host certificate must be copied and reconciled rather than merely reloaded.
+  # This service is started only by the production-mode ACME postRun hook.
+  systemd.services.cc-ci-acme-traefik-handoff = {
+    description = "Install renewed cc-ci wildcard into Traefik Swarm secrets";
+    after = [ "docker.service" "deploy-proxy.service" ];
+    requires = [ "docker.service" ];
+    path = [ pkgs.coreutils pkgs.docker pkgs.systemd pkgs.gnugrep ];
+    serviceConfig = {
+      Type = "oneshot";
+      UMask = "0077";
+    };
+    script = ''
+      src=/var/lib/acme/ci.commoninternet.net
+      dst=/var/lib/ci-certs/live
+      test -s "$src/fullchain.pem"
+      test -s "$src/key.pem"
+      install -d -m 0700 "$dst"
+      install -m 0444 "$src/fullchain.pem" "$dst/fullchain.pem.new"
+      install -m 0400 "$src/key.pem" "$dst/privkey.pem.new"
+      mv -f "$dst/fullchain.pem.new" "$dst/fullchain.pem"
+      mv -f "$dst/privkey.pem.new" "$dst/privkey.pem"
+
+      # deploy-proxy performs the health-gated Swarm rollout. Its reconciler
+      # derives a fresh version from the public certificate chain and inserts
+      # the matching ssl_cert/ssl_key secrets before deploying Traefik.
+      systemctl restart deploy-proxy.service
+
+      # A successful rollout no longer references old wildcard versions. Best
+      # effort removal retains any secret Docker still reports as in use.
+      keep="v$(sha256sum "$dst/fullchain.pem" | cut -c1-16)"
+      docker secret ls --format '{{.Name}}' | \
+        grep -E '^traefik_ci_commoninternet_net_ssl_(cert|key)_v' | \
+        grep -v -E "_(ssl_cert|ssl_key)_$keep\$" | \
+        while IFS= read -r stale; do docker secret rm "$stale" || true; done
+    '';
+  };
+
+  security.acme = {
+    acceptTerms = true;
+    certs."ci.commoninternet.net" = {
+      domain = "ci.commoninternet.net";
+      extraDomainNames = [ "*.ci.commoninternet.net" ];
+      # Staging issuance succeeded using the permanent, narrowly delegated
+      # CNAME. Production uses the same restricted acme-dns account.
+      dnsProvider = "acmedns";
+      environmentFile = "/etc/acme-dns/lego.env";
+      dnsResolver = "1.1.1.1:53";
+      server = "https://acme-v02.api.letsencrypt.org/directory";
+      postRun = ''
+        # The production marker is deployed only after staging proves the
+        # permanent CNAME and restricted acme-dns account work end to end.
+        if [ -e /var/lib/ci-certs/acme-production-enabled ]; then
+          ${pkgs.systemd}/bin/systemctl --no-block start cc-ci-acme-traefik-handoff.service
+        fi
+      '';
+    };
+  };
+}

opencode.json

@@ -2,7 +2,7 @@
   "$schema": "https://opencode.ai/config.json",
   "agent": {
     "general": {
-      "model": "opencode/deepseek-v4-pro"
+      "model": "opencode/deepseek-v4-flash"
     }
   }
 }

runner/warm_reconcile.py

@@ -24,6 +24,7 @@ Run as root on cc-ci (direct docker/volume access). CLI: `warm_reconcile.py <app
 
 from __future__ import annotations
 
+import hashlib
 import json
 import os
 import re
@@ -37,12 +38,51 @@ from harness import abra, lifecycle, warmsnap  # noqa: E402
 # --------------------------------------------------------------------------- specs
 
 
-def _traefik_setup(recipe: str, domain: str, version: str) -> None:
+CERT_DIR = "/var/lib/ci-certs/live"
+
+
+def wildcard_secret_version(cert_dir: str = CERT_DIR) -> str:
+    """Stable Swarm-secret version for the public certificate chain.
+
+    The certificate chain is public material, so its digest is safe to use as a
+    version label. The key is deliberately never read or hashed for logging.
+    """
+    chain = os.path.join(cert_dir, "fullchain.pem")
+    if not os.path.isfile(chain):
+        raise RuntimeError(f"FATAL: wildcard certificate missing at {chain}")
+    with open(chain, "rb") as certificate:
+        digest = hashlib.sha256(certificate.read()).hexdigest()
+    return "v" + digest[:16]
+
+
+def _traefik_requires_certificate_rollout(domain: str, secret_version: str) -> bool:
+    """Whether Traefik's active service still references an older cert version."""
+    stack = lifecycle._stack_name(domain)  # noqa: SLF001
+    service = f"{stack}_app"
+    result = _run(
+        [
+            "docker",
+            "service",
+            "inspect",
+            service,
+            "--format",
+            "{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{.SecretName}} {{end}}",
+        ],
+        timeout=30,
+    )
+    expected = {
+        f"{stack}_ssl_cert_{secret_version}",
+        f"{stack}_ssl_key_{secret_version}",
+    }
+    return not expected.issubset(set(result.stdout.split()))
+
+
+def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
     """Per-app config for the traefik reverse-proxy reconcile — preserves EXACTLY what the prior
     proxy.nix bash reconcile did (wildcard/file-provider mode serving the pre-issued cert as
     ssl_cert/ssl_key swarm secrets; NO ACME). Uses the proven abra.env_set (newline-safe, unlike the
     bash set_env that bit keycloak)."""
-    cert_dir = "/var/lib/ci-certs/live"
+    cert_dir = CERT_DIR
     if not (
         os.path.isfile(f"{cert_dir}/fullchain.pem") and os.path.isfile(f"{cert_dir}/privkey.pem")
     ):
@@ -56,14 +96,15 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
     abra.env_set(domain, "DOMAIN", domain)
     abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
     abra.env_set(domain, "WILDCARDS_ENABLED", "1")
-    abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", "v1")
-    abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", "v1")
+    secret_version = wildcard_secret_version(cert_dir)
+    abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", secret_version)
+    abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", secret_version)
     abra.env_set(domain, "COMPOSE_FILE", '"compose.yml:compose.wildcard.yml"')
     stack = lifecycle._stack_name(domain)  # noqa: SLF001
     have = set(lifecycle._docker_names("secret", stack))  # noqa: SLF001
 
     def _has(name):
-        return any(s.endswith(f"_{name}_v1") for s in have)
+        return any(s.endswith(f"_{name}_{secret_version}") for s in have)
 
     if not _has("ssl_cert"):
         _run(
@@ -74,7 +115,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
                 "insert",
                 domain,
                 "ssl_cert",
-                "v1",
+                secret_version,
                 f"{cert_dir}/fullchain.pem",
                 "-f",
                 "-n",
@@ -91,7 +132,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
                 "insert",
                 domain,
                 "ssl_key",
-                "v1",
+                secret_version,
                 f"{cert_dir}/privkey.pem",
                 "-f",
                 "-n",
@@ -99,6 +140,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> None:
             timeout=120,
             check=True,
         )
+    return _traefik_requires_certificate_rollout(domain, secret_version)
 
 
 SPECS: dict[str, dict] = {
@@ -457,8 +499,9 @@ def reconcile(app: str) -> str:
     # Per-app config/secrets: a spec may provide its own `setup` (traefik's cert/file-provider wiring);
     # otherwise the default keycloak-shaped path (app new + DOMAIN/LETS_ENCRYPT + generate secrets).
     setup = spec.get("setup")
+    setup_needs_rollout = False
     if setup:
-        setup(recipe, domain, latest)
+        setup_needs_rollout = bool(setup(recipe, domain, latest))
     else:
         ensure_app_config(recipe, domain, latest)
         ensure_secrets(domain)
@@ -476,6 +519,20 @@ def reconcile(app: str) -> str:
         write_last_good(recipe, target)
         return f"deployed-fresh:{target}"
 
+    # A certificate rotation changes Traefik's immutable Swarm secrets but
+    # must not be held hostage by an unrelated recipe-major upgrade policy.
+    # Redeploy the current recipe version so its compose spec references the
+    # just-created cert/key secret pair, then apply the usual health gate.
+    if setup_needs_rollout:
+        if not current:
+            raise RuntimeError(f"{app} has services but no current version")
+        print(f"[{app}] certificate changed → redeploy {current}", flush=True)
+        deploy_version(recipe, domain, current, dt)
+        if not wait_healthy(spec):
+            raise RuntimeError(f"{app} certificate rollout {current} did not become healthy")
+        write_last_good(recipe, current)
+        return f"certificate-rolled-out:{current}"
+
     # Deployed & already on latest → converge to a no-op (commit last-good if healthy).
     if current == latest:
         if wait_healthy(spec, timeout=60):

tests/STYLE.md

@@ -0,0 +1,116 @@
+# cc-ci test style guide
+
+Rules for writing and changing tests under `tests/`. Read this before any test edit — in particular
+before a `/recipe-upgrade <recipe> --with-tests` or `/ci-test-review` fix, where the temptation is to
+make a red run green rather than to make the test right.
+
+The tests are the **independent gate** on recipe upgrades. Their value is entirely in being hard to
+fool, so every rule below exists to keep them (a) honest and (b) alive across upgrades.
+
+---
+
+## 1. Set up state through the application, not its database
+
+**Order of preference for any fixture that must create state:**
+
+1. **The app's public HTTP API.**
+2. **The app's official CLI or release console** (`docker exec … <app-cli>`).
+3. **Writing rows into its database — last resort only**, and only with a comment saying which of the
+   above were tried and why they did not work.
+
+Direct SQL couples the test to the app's *internal schema*, which upgrades are free to change. The
+app's own interface is the thing it promises to keep working.
+
+> **Why this rule exists.** `tests/plausible/custom/test_event_tracking.py` used to register its test
+> site with `INSERT INTO sites (...)`. That was sufficient for plausible v2. In v3 a site must belong
+> to a **team**, and the app silently discards events for a teamless site — `POST /api/event` still
+> returns **202** and the row is still in postgres, so the only visible symptom was that nothing ever
+> reached ClickHouse. It read as a mysterious ingestion stall and held the recipe RED for six weeks.
+>
+> The fix was not to also INSERT a team row. It was to stop writing rows: the fixture now calls
+> `Plausible.Sites.create/2` through the app's release console, and the app provisions whatever its
+> data model currently needs. The same expression works unchanged on v2 (which has no `teams` table
+> at all) **and** v3 — not because the test handles both, but because it stopped depending on the
+> schema.
+
+When the ideal interface is unavailable, say so in the code. plausible's HTTP provisioning API
+(`POST /api/v1/sites`) is gated behind a paid plan and answers `:upgrade_required` on CE, so the test
+drops to option 2 and records that in a comment.
+
+## 2. Gate on version rather than writing dual-path fixtures
+
+If a behaviour genuinely only exists from version X, **gate the test on the version** instead of
+branching inside it:
+
+```python
+pytest.mark.skipif(app_version < (3,), reason="teams were introduced in v3")
+```
+
+Do **not** write a fixture that carefully supports both schemas. Version-portable code is harder to
+read, harder to trust, and quietly rots once nobody runs the old path.
+
+Corollary: **old tests can simply be deleted** once the fleet has moved past that version. The older
+version is only ever exercised through the *upgrade* tier (deploy base → upgrade → assert), so tests
+that only make sense for a superseded version are dead weight, not coverage.
+
+Prefer §1 first: an app-level fixture often makes the version difference disappear, and then no gate
+is needed at all.
+
+## 3. Never weaken an assertion to turn a run green
+
+There is a hard line between these two, and only the second is allowed as a way out of a red run:
+
+* **Weakening** — relaxing *what* is asserted: dropping a field check, accepting a wider status set,
+  asserting a 202 ack instead of the stored result, deleting the read-back.
+* **Correcting the fixture or the wait** — fixing *how* the test sets up or how long it allows, with
+  the assertion untouched.
+
+If a test can only pass by asserting less, it has found a real regression. Report it; do not edit it.
+
+## 4. Assert real state, not acknowledgements
+
+An HTTP 202 means "accepted", not "done". Read the effect back out of the system that owns it — the
+row in the analytics store, the file on disk, the record in the API — and assert on the values you
+sent. plausible's ingestion returns 202 for events it goes on to discard entirely; a test that
+stopped at the ack would have been permanently, silently green.
+
+## 5. Derive waits from the recipe's declared readiness, not a guess
+
+A per-recipe `recipe_meta.py` already declares `DEPLOY_TIMEOUT` / `HTTP_TIMEOUT` because someone
+measured that app's boot profile. A custom test that hard-codes a shorter window contradicts it and
+will flake or fail on a slower version.
+
+Remember the **tier order**: `custom` runs after `backup`/`restore`, which disrupts the datastore and
+restarts the app. A window sized for a warm app is not sized for that. plausible's health check
+allowed 60s; v3 boots through `sleep 10` → `createdb` → `migrate` → cache warmers first.
+
+## 6. Diagnose from the app's own telemetry before touching a test
+
+Before concluding a test is stale, find the app's account of what happened. It is usually definitive
+and it stops you fixing the wrong thing. plausible records dropped events in ClickHouse's
+`ingest_counters`: `dropped_not_found` with 0 rows before the fix, `buffered` with rows after — that
+single counter identified the root cause after the HTTP status had suggested everything was fine.
+
+Prove the diagnosis both ways where you can: same input, broken state → symptom; corrected state →
+no symptom.
+
+## 7. Fixtures must be idempotent
+
+A fixture may run against a warm canonical, a restored volume, or a re-run. Creating state must be
+safe to repeat — look the object up first and reuse it, rather than assuming a clean database.
+
+## 8. Keep test identities obviously synthetic
+
+Use `ccci-`-prefixed names and `.example` / `.invalid` domains for anything a test creates, so state
+it leaves behind is instantly attributable and can never be confused with real data.
+
+---
+
+## Changing a test: the checklist
+
+1. Reproduce the failure and get the **app's own** explanation (§6).
+2. Classify: recipe bug, or stale test? Only a stale test justifies a test edit.
+3. Fix the **fixture, wait, or setup** — never the assertion (§3).
+4. Prefer the app's interface over its database (§1); gate on version rather than branching (§2).
+5. Verify green against the recipe PR head with the changed test, plus a regression sample.
+6. Say in the commit and PR **what evidence** proves the diagnosis, not just what changed.

tests/plausible/custom/test_event_tracking.py

@@ -14,7 +14,11 @@ Both assert real app state (the event reached the analytics store), not just the
 
 plausible only ingests events for *known* sites — the in-memory `sites_cache` gates ingestion and
 drops events for unregistered domains (empirically confirmed: an event for an unregistered domain
-never appears in events_v2). So each test first registers a site row in the metadata postgres, then
+never appears in events_v2). Sites are therefore provisioned through plausible's OWN creation path
+rather than by writing rows — under v3 a site must belong to a TEAM, and a teamless site is dropped as
+`dropped_not_found` while the POST still acks 202, which reads as a silent ingestion stall. Letting the
+app create the site sidesteps that entirely, and works unchanged on v2. So each test first provisions
+the site, then
 POSTs repeatedly while polling ClickHouse: the sites_cache must refresh to admit the new site and the
 event write-buffer must flush to ClickHouse, so the first landing is not instantaneous. Re-POSTing the
 same event is safe — we assert the row count is >= 1.
@@ -40,6 +44,10 @@ _UA = (
     "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
 )
 
+# Identity the harness provisions sites under. Ephemeral per-run deploy, never a real account.
+_HARNESS_EMAIL = "cc-ci@ci.invalid"
+_HARNESS_PW = "ccci-harness-passphrase-2026"
+
 
 def _ch(domain: str, sql: str) -> str:
     """Run a ClickHouse query against the `plausible_events_db` service; return stdout (stripped)."""
@@ -50,18 +58,61 @@ def _ch(domain: str, sql: str) -> str:
     ).strip()
 
 
+# plausible's own provisioning path. Creating a site through the app (rather than INSERTing rows)
+# means the app applies whatever its current data model requires — which is what makes this work
+# unchanged across the v2→v3 jump, where sites gained a mandatory owning TEAM. Verified on cc-ci
+# against BOTH v2.0.0 (no `teams` table at all) and v3.2.1: identical expression, site usable, events
+# ingested. See tests/STYLE.md.
+#
+# The HTTP provisioning API (`POST /api/v1/sites`) would be the first choice, but it is gated behind
+# a paid plan — on CE it answers `:upgrade_required` — so the app's release console is the closest
+# public interface available here.
+_PROVISION_SITE_EXS = """
+pw = "__PW__"
+email = "__EMAIL__"
+user =
+  case Plausible.Auth.find_user_by(email: email) do
+    nil ->
+      {:ok, u} =
+        Plausible.Auth.User.new(%{name: "cc-ci", email: email, password: pw, password_confirmation: pw})
+        |> Plausible.Repo.insert()
+      u
+    u -> u
+  end
+site = "__SITE__"
+result =
+  case Plausible.Sites.get_by_domain(site) do
+    nil -> Plausible.Sites.create(user, %{"domain" => site, "timezone" => "UTC"})
+    s -> {:ok, s}
+  end
+case result do
+  {:ok, _} -> IO.puts("CCCI_SITE_OK " <> site)
+  other -> IO.puts("CCCI_SITE_ERR " <> inspect(other))
+end
+"""
+
+
 def _register_site(domain: str, site: str) -> None:
-    """Insert a site row into the metadata postgres (`db` service) so plausible will ingest events for
-    it. Idempotent (ON CONFLICT DO NOTHING)."""
-    sql = (
-        "INSERT INTO sites (domain, timezone, inserted_at, updated_at, native_stats_start_at) "
-        f"VALUES ('{site}','UTC', now(), now(), now()) ON CONFLICT (domain) DO NOTHING; "
-        f"SELECT domain FROM sites WHERE domain = '{site}';"
+    """Provision `site` via plausible's own site-creation path, so it is a site the app will ingest for.
+
+    Idempotent: an existing domain is reused rather than re-created.
+
+    Do NOT reach into postgres to do this. A `sites` INSERT was enough under v2, but v3 requires the
+    site to belong to a TEAM and silently discards events for a teamless site — `POST /api/event`
+    still acks 202 and the row still exists, so the only symptom is that nothing reaches ClickHouse
+    (ClickHouse's own `ingest_counters` records it as `dropped_not_found`). That is what put this
+    recipe RED on build 1224. Going through the app removes the whole class of problem: it provisions
+    the team itself.
+    """
+    exs = (
+        _PROVISION_SITE_EXS.replace("__PW__", _HARNESS_PW)
+        .replace("__EMAIL__", _HARNESS_EMAIL)
+        .replace("__SITE__", site)
+    )
+    out = lifecycle.exec_in_app(domain, ["/app/bin/plausible", "rpc", exs], service="app")
+    assert f"CCCI_SITE_OK {site}" in out, (
+        f"could not provision site {site!r} via the app: {out.strip()[-400:]}"
     )
-    out = lifecycle.exec_in_app(
-        domain, ["psql", "-q", "-U", "plausible", "-d", "plausible", "-tAc", sql], service="db"
-    ).strip()
-    assert out == site, f"site {site!r} not registered in postgres (got {out!r})"
 
 
 def _post_event(base_domain: str, site: str, name: str, pathname: str) -> int:
@@ -93,9 +144,9 @@ def _ingest_and_count(
     last_status = None
     while True:
         last_status = _post_event(base_domain, site, name, pathname)
-        assert (
-            last_status == 202
-        ), f"POST /api/event for {name!r} → HTTP {last_status} (expected 202)"
+        assert last_status == 202, (
+            f"POST /api/event for {name!r} → HTTP {last_status} (expected 202)"
+        )
         time.sleep(interval)
         raw = _ch(base_domain, count_sql)
         count = int(raw) if raw.isdigit() else 0
@@ -143,6 +194,6 @@ def test_custom_event_roundtrip(live_app):
         live_app,
         f"SELECT name FROM events_v2 WHERE pathname = '{pathname}' LIMIT 1",
     )
-    assert (
-        stored_name == event_name
-    ), f"custom event stored as {stored_name!r}, expected {event_name!r}"
+    assert stored_name == event_name, (
+        f"custom event stored as {stored_name!r}, expected {event_name!r}"
+    )

tests/plausible/custom/test_health_check.py

@@ -17,6 +17,12 @@ def test_plausible_root_serves(live_app):
     62-char SECRET_KEY_BASE, see recipe_meta.EXTRA_ENV); the dedicated
     /api/health endpoint is.
     """
+    # The custom tier runs AFTER the backup/restore tier, which disrupts postgres under the app and
+    # restarts it. v3 (community-edition) then boots through `sleep 10` + `db createdb` + `db migrate`
+    # + cache warmers before /api/health flips to 200, which does not fit in 60s — that is what put
+    # this recipe RED on build 1224 while install/upgrade/backup/restore all passed. The assertion is
+    # unchanged (still a hard 200 from the real readiness endpoint); only the wait matches the boot
+    # profile the recipe already declares via recipe_meta.HTTP_TIMEOUT (1200).
     url = f"https://{live_app}/api/health"
-    status, _ = harness_http.retry_http_get(url, expect_status=(200,), max_wait=60, interval=3)
+    status, _ = harness_http.retry_http_get(url, expect_status=(200,), max_wait=300, interval=5)
     assert status == 200, f"GET {url} HTTP {status}"

tests/unit/test_acme_dns.py

@@ -0,0 +1,21 @@
+"""Unit coverage for the public wildcard-secret version label."""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parents[2] / "runner"))
+import warm_reconcile as wr  # noqa: E402
+
+
+def test_wildcard_secret_version_is_stable_and_does_not_need_key(tmp_path):
+    (tmp_path / "fullchain.pem").write_text("public certificate chain\n")
+    assert wr.wildcard_secret_version(str(tmp_path)) == wr.wildcard_secret_version(str(tmp_path))
+    assert wr.wildcard_secret_version(str(tmp_path)).startswith("v")
+
+
+def test_wildcard_secret_version_changes_with_certificate_chain(tmp_path):
+    chain = tmp_path / "fullchain.pem"
+    chain.write_text("first public certificate chain\n")
+    first = wr.wildcard_secret_version(str(tmp_path))
+    chain.write_text("replacement public certificate chain\n")
+    assert wr.wildcard_secret_version(str(tmp_path)) != first

tests/unit/test_warm_reconcile.py

@@ -9,6 +9,7 @@ from __future__ import annotations
 
 import os
 import sys
+from types import SimpleNamespace
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
@@ -115,6 +116,20 @@ def test_traefik_spec_is_stateless_with_setup():
     assert "setup" not in wr.SPECS["keycloak"]
 
 
+def test_traefik_certificate_rollout_detects_active_secret_version(monkeypatch):
+    stack = "traefik_ci_commoninternet_net"
+    monkeypatch.setattr(wr.lifecycle, "_stack_name", lambda _domain: stack)
+    monkeypatch.setattr(
+        wr,
+        "_run",
+        lambda *_args, **_kwargs: SimpleNamespace(
+            stdout=f"{stack}_ssl_cert_vnew {stack}_ssl_key_vnew"
+        ),
+    )
+    assert not wr._traefik_requires_certificate_rollout("traefik.ci.commoninternet.net", "vnew")
+    assert wr._traefik_requires_certificate_rollout("traefik.ci.commoninternet.net", "vold")
+
+
 def test_manual_migration_markers():
     assert wr.notes_flag_manual_migration("This release requires a MANUAL MIGRATION of the DB.")
     assert wr.notes_flag_manual_migration("Breaking change: action required before upgrade.")