upgrade base: prefer the newest published release over a STALE canonical

The canonical is last-GREEN, not CURRENT. Its promotion (WC5) deploys a warm-<recipe> app and health-checks it; when that fails the canonical stays pinned at an old release indefinitely — and the failure is invisible to the upgrade tier, which keeps basing on it. Live evidence (2026-08-10): gitea's canonical was 3.5.3+1.24.2-rootless from 2026-06-17 because warm-gitea crash-loops on a read-only app.ini (open fix: recipe PR #4). Four other recipes are in the same state — the 2026-08-09 sweep logged GREEN-BUT-PROMOTE-FAILED for bluesky-pds, drone, gitea, lasuite-drive and lasuite-meet. Consequence: the upgrade tier tested 3.5.3 -> head, a transition no deployment performs, and MISSED the one they do. 3.5.3 -> head crosses an APP_INI_VERSION change (v21 -> v22) so Swarm creates a fresh config object; the real 3.6.1 -> 3.6.2 upgrade keeps v22 and aborts on Swarm's immutable-config rule ('only updates to Labels are allowed'). That shipped in 3.6.2 and broke every existing install taking the 1.27.1 security release. Now: when the newest published release older than head is NEWER than the canonical, use the release and say so in the log (naming the stale canonical so the promote failure gets noticed). Resolution stays dynamic; UPGRADE_BASE_FLOOR still filters structurally-invalid candidates; the canonical remains the base whenever it is current. Unit-verified on the real gitea tag set: base moves 3.5.3+1.24.2-rootless -> 3.6.1+1.26.2-rootless, exactly the transition that reproduces the break.
2026-08-10 18:16:30 +00:00
116 changed files with 523 additions and 1438 deletions
@@ -1,8 +0,0 @@
 # Non-sensitive runtime configuration shared by the cc-ci orchestrator and agents.
 #
 # Keep credentials, tokens, and keys in /srv/cc-ci/.testenv.  The orchestrator
 # loads this file first via cc-ci-plan/load-env.sh.
 GITEA_USERNAME=autonomic-bot
 GITEA_URL=git.autonomic.zone
 TINFOIL_MODEL=deepseek-v4-pro
 TINFOIL_BASE_URL=https://inference.tinfoil.sh/v1
@@ -5,12 +5,10 @@
 #            /srv/cc-ci/.sops/master-age.txt (never in this repo). Lets us re-key if cc-ci is lost.
 keys:
  - &host   age1h90utdztfc23kx8ewrtrtk80mnddvrf8pg4ppej55rwwwupzhfvqhmp3qa
  - &host2  age1tmvgpgc822ezqgxg4x8h6ndph6j9hwpgjpg364zn7lw3t5h694rq5730wa
  - &master age1cmk26t9e30ls8594s8txgmf2exenydmntfxqpcd3qdqm3ru2lpnqpdkdz9
 creation_rules:
  - path_regex: secrets/.*\.(yaml|json|env)$
    key_groups:
      - age:
          - *host
          - *host2
          - *master
@@ -36,19 +36,3 @@ Two kinds of tests live here — run them on **different** cadences:
 A red test is information. Never skip, delete, or relax a test to make a run green — fix the root
 cause or record it in `machine-docs/DEFERRED.md`. (This is a standing build guardrail.)
 ## Ship work as PRs, merge them yourself, operator reviews retrospectively
 Work on this repo goes: **branch → PR → merge it yourself once verified → operator reviews
 retrospectively.** Do not commit straight to `main`, and do not wait for review before merging — the
 invocation is the authorization, and blocking would stall the CI this repo runs.
 The PR is therefore not a gate; it is how the work stays legible after the fact. Write the
 description to be read later: what changed, why, and the evidence it works (harness output, a
 verified run, a before/after number). A PR that says "fix test" has failed at its only job.
 The same policy covers `recipe-maintainers/cc-ci-orchestrator`. It does **NOT** cover recipe repos —
 any `coop-cloud/<recipe>` or its mirror is created and verified but **never agent-merged**, because
 those change what deploys on other people's infrastructure.
 Before editing a test, read `tests/STYLE.md`.
@@ -1,19 +0,0 @@
 # gitea upstream sources
 ## gitea/gitea
 - image: gitea/gitea
 - source: https://github.com/go-gitea/gitea
 - releases: https://github.com/go-gitea/gitea/releases
 - security: https://blog.gitea.com/
 ## postgres
 - image: postgres
 - source: https://github.com/postgres/postgres
 - releases: https://www.postgresql.org/docs/release/
 - security: https://www.postgresql.org/support/security/
 ## mariadb
 - image: mariadb
 - source: https://github.com/MariaDB/server
 - releases: https://mariadb.com/kb/en/release-notes/
 - security: https://mariadb.com/kb/en/security/
@@ -206,11 +206,7 @@ def _local_history_row(run_id, res):
    so render_history is unchanged. `number` is the run dir name (the /runs/<id>/ path + _results_for
    key); link to the Drone build when the id is numeric, else to the local summary card."""
    ref = res.get("ref") or ""
-    url = (
+    url = f"{DRONE_URL}/{CI_REPO}/{run_id}" if str(run_id).isdigit() else f"/runs/{run_id}/summary.html"
        f"{DRONE_URL}/{CI_REPO}/{run_id}"
        if str(run_id).isdigit()
        else f"/runs/{run_id}/summary.html"
    )
    return {
        "recipe": res.get("recipe"),
        "status": _run_status(res),
@@ -355,7 +351,7 @@ def _card(r):
        f'<div class="card">{shot}<div class="body">'
        f'<div class="name">{html.escape(r["recipe"])}</div>'
        f'<div class="row"><span class="pill" style="background:{color}">{html.escape(r["status"])}</span>'
-        f"<code>{html.escape(r['version'])}</code></div>"
+        f'<code>{html.escape(r["version"])}</code></div>'
        f"{_flags_html(r['flags'])}"
        f'<div class="foot"><a href="{run_url}">run #{num} · {_ago(r["finished"])}</a>'
        f'<a href="/recipe/{html.escape(r["recipe"])}">history →</a></div>'
@@ -398,7 +394,7 @@ def render_history(recipe, rows):
            f'<tr><td><a href="{html.escape(r["url"])}">#{r["number"]}</a></td>'
            f'<td><span class="pill" style="background:{color}">{html.escape(r["status"])}</span></td>'
            f"<td>{lvl}</td><td><code>{html.escape(r['version'])}</code></td>"
-            f"<td>{_ago(r['finished'])}</td><td>{shot}</td></tr>"
+            f'<td>{_ago(r["finished"])}</td><td>{shot}</td></tr>'
        )
    body = "\n".join(trs) or '<tr><td colspan="6">no runs for this recipe yet</td></tr>'
    inner = (
@@ -1,283 +0,0 @@
 # Plan: restricted ACME DNS renewal for cc-ci
 ## Outcome
 Replace the manually issued, sops-stored wildcard certificate with unattended
 DNS-01 renewal for these exact names:
 ```text
 ci.commoninternet.net
 *.ci.commoninternet.net
 ```
 The cc-ci host will run an authoritative `acme-dns` instance only for
 `acme.commoninternet.net`. It will never receive a Gandi credential or any
 credential that can edit the parent `commoninternet.net` zone.
 The only enduring delegation from the parent zone is:
 ```text
 _acme-challenge.ci.commoninternet.net. CNAME <account-id>.acme.commoninternet.net.
 ```
 That CNAME authorizes the generated acme-dns account to answer ACME TXT
 challenges for the ci wildcard, not to edit any parent-zone DNS record.
 ## Project facts and constraints
 - The target is the production `cc-ci-hetzner` NixOS 26.05 host, not the
  orchestrator. Its public IPv4 is `91.98.47.73`; it has no public IPv6.
 - The wildcard currently points at the public gateway, which TLS-passthroughs
  to cc-ci's Traefik. DNS authority for `acme.commoninternet.net` must point
  directly to `91.98.47.73`; the gateway is not involved in DNS.
 - Nothing listens on TCP or UDP 53 today. The Nix firewall permits 22, 80, and
  443 only; any Hetzner Cloud firewall must also be checked before deployment.
 - TLS terminates in the Docker Swarm Traefik service. It currently reads
  `ssl_cert` and `ssl_key` **Swarm secrets** populated from
  `/var/lib/ci-certs/live/{fullchain.pem,privkey.pem}` by
  `runner/warm_reconcile.py`. A normal host-service reload cannot install a
  renewed certificate.
 - The existing certificate is expired: its served validity ended
  `2026-08-24 18:18:52 UTC`. Keep the current files as rollback material until
  the new production certificate and Traefik rotation have both been verified.
 - `pkgs.acme-dns` and `pkgs.lego` are available in the pinned nixpkgs. NixOS
  `security.acme` uses Lego and supports a DNS provider plus an environment
  file and post-renew hook. Confirm the pinned provider spelling with
  `lego --help` during implementation; Lego's current documented provider code
  is `acmedns`.
 ## Security invariants
 1. Do not request, add, store, or use `GANDI_API_KEY`, a Gandi PAT, or any
   parent-zone update credential on cc-ci or the orchestrator.
 2. Bind the acme-dns HTTP API to `127.0.0.1` only. Its API may use plain HTTP
   because it is loopback-only; do not create a circular API TLS dependency.
 3. Allow public DNS only on TCP/UDP 53 and only for the authoritative zone.
 4. The generated acme-dns account data is a secret. Keep it as a root/acme-only
   persistent state file under `/var/lib/acme/`; never put it in Nix text, the
   Nix store, git, `.env.public`, or a log.
 5. After the account exists, set `disable_registration = true`. The existing
   account must still be able to call `/update`.
 6. Limit the account's update source with `ACME_DNS_ALLOWLIST=127.0.0.1/32`.
   This is defence in depth in addition to the loopback API binding.
 ## Intended DNS design
 Use an **out-of-bailiwick** nameserver name to avoid in-bailiwick glue
 ambiguity:
 ```text
 ns-acme.commoninternet.net.  A   91.98.47.73
 acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
 ```
 `acme-dns` itself serves the delegated zone and returns its matching NS record:
 ```text
 acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
 ```
 This host is authoritative for `acme.commoninternet.net` and its generated
 children only. It is not authoritative for `ci.commoninternet.net` or for
 `commoninternet.net`.
 ## Implementation phases
 ### 1. Preflight and safety checks
 Before changing Nix configuration, record:
 ```bash
 ssh cc-ci 'ss -lntup "( sport = :53 )"'
 ssh cc-ci 'systemctl list-units --type=service --all "*acme*" "*dns*"'
 ssh cc-ci 'nft list ruleset'
 ssh cc-ci 'docker service ls'
 ```
 Confirm that no service owns port 53, that the Traefik Swarm services are
 healthy, and that the Hetzner Cloud firewall will permit both 53/tcp and
 53/udp. Do not replace an existing DNS service.
 Obtain the operator's ACME contact email before enabling `security.acme`.
 ### 2. Add a dedicated acme-dns Nix module
 Create `nix/modules/acme-dns.nix` and import it from
 `nix/hosts/cc-ci-hetzner/configuration.nix`. The module should:
 - create a dedicated unprivileged `acme-dns` user and group;
 - run `${pkgs.acme-dns}/bin/acme-dns -c <public generated config>` with a
  persistent working/state directory `/var/lib/acme-dns`;
 - grant only `CAP_NET_BIND_SERVICE` to bind DNS port 53;
 - use SQLite at `/var/lib/acme-dns/acme-dns.db` with mode `0600`;
 - bind DNS to `91.98.47.73:53` with `protocol = "both4"`;
 - set `domain = "acme.commoninternet.net"`,
  `nsname = "ns-acme.commoninternet.net"`, and a public hostmaster-style
  `nsadmin` value;
 - include the public NS record above in `general.records`;
 - bind `[api]` to `127.0.0.1:8080`, set `tls = "none"`, use a restrictive
  CORS list, and initially leave `disable_registration = false`;
 - use a hardened systemd unit: `NoNewPrivileges`, `PrivateTmp`,
  `ProtectSystem = "strict"`, `ProtectHome`, `PrivateDevices`, and only the
  state directory as writable; and
 - open `networking.firewall.allowedTCPPorts = [ 53 ]` and
  `allowedUDPPorts = [ 53 ]` in the **cc-ci Hetzner host** configuration.
 The configuration file is public data and may be generated by Nix. It must not
 contain account credentials.
 Deploy this phase with the normal cc-ci deployment discipline: first
 `nixos-rebuild test --flake /etc/cc-ci#cc-ci-hetzner`, verify SSH, Traefik, and
 the host remain healthy, then run the identical `switch` target. Verify local
 DNS on both transports:
 ```bash
 dig @91.98.47.73 acme.commoninternet.net NS
 dig +tcp @91.98.47.73 acme.commoninternet.net NS
 ```
 ### 3. Operator gate: delegate the narrow DNS zone
 After the service is healthy, ask the operator to add exactly these records at
 Gandi (using its DNS UI, never a token on this host):
 ```dns
 ns-acme.commoninternet.net.  A   91.98.47.73
 acme.commoninternet.net.     NS  ns-acme.commoninternet.net.
 ```
 If Gandi models delegation as a nameserver/glue form rather than ordinary zone
 records, use its equivalent UI flow. Do not proceed until public recursive DNS
 shows the delegation and direct queries work from an external network:
 ```bash
 dig NS acme.commoninternet.net @1.1.1.1
 dig TXT test.acme.commoninternet.net @91.98.47.73
 dig +tcp TXT test.acme.commoninternet.net @91.98.47.73
 ```
 ### 4. Configure NixOS ACME in staging mode and obtain the account target
 Extend the new module with one `security.acme.certs` entry for the base name
 `ci.commoninternet.net`:
 ```nix
 {
  domain = "ci.commoninternet.net";
  extraDomainNames = [ "*.ci.commoninternet.net" ];
  dnsProvider = "acmedns"; # verify against the pinned Lego binary
  environmentFile = "/etc/acme-dns/lego.env";
  dnsResolver = "1.1.1.1:53";
 }
 ```
 `/etc/acme-dns/lego.env` contains only non-secret wiring:
 ```text
 ACME_DNS_API_BASE=http://127.0.0.1:8080
 ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
 ACME_DNS_ALLOWLIST=127.0.0.1/32
 ```
 Lego registers and persists its per-domain acme-dns account in the storage
 path. The path is writable only by the ACME service user and is not Nix-managed
 content. Do not hand-create its JSON: let the pinned Lego provider establish
 the account format.
 Set the ACME CA to Let's Encrypt staging for this phase. Start the certificate
 unit manually after the NS delegation is confirmed. The first staging run is
 expected to create the account and may fail validation because the CNAME is not
 yet present. Read the storage file only with a root-only helper that prints the
 generated **fulldomain** and never its username or password.
 ### 5. Operator gate: permanent challenge CNAME
 Ask the operator to create the exact target reported in phase 4:
 ```dns
 _acme-challenge.ci.commoninternet.net. CNAME <generated-id>.acme.commoninternet.net.
 ```
 This is a permanent record. It must not be created, changed, or removed by an
 agent. Confirm the complete chain through a public recursive resolver before
 continuing:
 ```bash
 dig CNAME _acme-challenge.ci.commoninternet.net @1.1.1.1
 dig TXT <generated-id>.acme.commoninternet.net @91.98.47.73
 dig +tcp TXT <generated-id>.acme.commoninternet.net @91.98.47.73
 ```
 ### 6. Staging issuance, then production issuance
 Run the NixOS ACME certificate unit against staging and verify all of the
 following:
 1. it updates only the generated acme-dns TXT target;
 2. public recursive DNS sees the CNAME and the TXT value;
 3. staging issues a certificate containing both requested names; and
 4. no Gandi variable, credential file, or API request appears in the unit.
 Only then select the production Let's Encrypt directory and issue the real
 certificate. Keep the old sops certificate live during both attempts.
 ### 7. Make Traefik consume renewals safely
 Do **not** use only `reloadServices`: Traefik receives Docker Swarm secrets and
 cannot see an updated host file. Add a root-only renewal handoff service,
 serialized with all other Traefik reconciliation, and call it from the ACME
 certificate's `postRun` hook.
 The handoff must:
 1. atomically copy the new `fullchain.pem` and key from the NixOS ACME output
   into `/var/lib/ci-certs/live`, with the existing `0444`/`0400` modes;
 2. generate a new, content-derived **non-secret** Swarm secret version;
 3. insert new `ssl_cert` and `ssl_key` Swarm secrets, update the Traefik recipe
   environment to reference those versions, and reconcile/redeploy Traefik;
 4. health-check `https://traefik.ci.commoninternet.net/api/version` with SNI;
 5. retain the prior secret version until the new task is healthy, then remove
   it; and
 6. record a failure clearly without deleting the last-known-good certificate.
 Implement this as a tested extension of `runner/warm_reconcile.py` (or a
 small, explicitly locked companion) rather than an ad-hoc shell command. The
 renewal path and the normal `deploy-proxy` path must share a lock so they cannot
 race over Swarm secret versions.
 After production issuance and a successful Traefik rotation, remove the
 `wildcard_cert` and `wildcard_key` sops declarations from
 `nix/modules/secrets.nix`; otherwise later Nix activations would overwrite the
 renewed host files. Remove the obsolete encrypted values from the private
 `cc-ci-secrets` repository only after rollback is no longer needed.
 ### 8. Lock registration and prove unattended renewal
 In a follow-up Nix change, set `api.disable_registration = true`, test that the
 existing account can still update its TXT record, and confirm `/register` is
 rejected. Then verify:
 ```bash
 systemctl list-timers 'acme-*'
 systemctl start acme-ci.commoninternet.net.service
 journalctl -u acme-ci.commoninternet.net.service -b
 ```
 Perform a controlled staging renewal after registration is disabled, observe
 the renewed Traefik secret version, and confirm the certificate served through
 the gateway has the expected names and a new validity window.
 ## Final acceptance checklist
 - [ ] cc-ci and the orchestrator contain no Gandi API credential.
 - [ ] Gandi delegates only `acme.commoninternet.net` to cc-ci.
 - [ ] Only `_acme-challenge.ci.commoninternet.net` CNAMEs into that zone.
 - [ ] The acme-dns API is loopback-only; only 53/tcp and 53/udp are public.
 - [ ] External UDP and TCP authoritative DNS checks pass.
 - [ ] Registration is disabled after the one account is created.
 - [ ] The ACME account can update only its generated TXT record.
 - [ ] The certificate covers both `ci.commoninternet.net` and its wildcard.
 - [ ] A renewal rotates Traefik's Swarm secrets and preserves a working prior
      version until the replacement passes health checks.
 - [ ] No credential or private key has been committed, logged, or written into
      the Nix store.
@@ -1,15 +0,0 @@
 # Public runtime environment
 `.env.public` contains non-sensitive configuration that the cc-ci orchestrator
 and its agent sessions need at runtime. It is intentionally tracked so it can
 be inspected and reproduced with the rest of the CI configuration.
 Load it together with the local secret file by sourcing
 `/srv/cc-ci/cc-ci-plan/load-env.sh`. The helper reads `.env.public` first and
 then `/srv/cc-ci/.testenv`; credentials, tokens, and keys belong only in the
 latter file.
 Do not put a value in `.env.public` merely because it is convenient. If it
 would grant access or require rotation, it is a secret and belongs in
 `.testenv`. Public service endpoints, model names, and account identifiers may
 be tracked here.
@@ -16,7 +16,7 @@
    sops-nix.inputs.nixpkgs.follows = "nixpkgs";
  };
-  outputs = { self, nixpkgs, sops-nix, ... }:
+  outputs = { nixpkgs, sops-nix, ... }:
    let
      system = "x86_64-linux";
      pkgs = nixpkgs.legacyPackages.${system};
@@ -35,25 +35,13 @@
      ];
    in
    {
      # The whole CI server as one reusable module (nix/modules/default.nix): every service,
      # the harness tooling, sops wiring and acme-dns — but no hardware, networking, tailscale
      # node, root keys or stateVersion. sops-nix's module comes bundled so a consumer only has
      # to import this and set `cc-ci.publicIPv4` (+ `cc-ci.sopsFile` when it is not built from
      # a --recursive clone). A consuming flake MUST make its `cc-ci` input follow its own
      # `nixpkgs` and `sops-nix`, otherwise two sops-nix module trees collide.
      # Consumer: recipe-maintainers/cc-ci-orchestrator `#cc-ci` (CI server + orchestrator on
      # one Hetzner host, 2026-09).
      nixosModules.cc-ci-server = {
        imports = [ sops-nix.nixosModules.sops ./nix/modules ];
      };
      nixosConfigurations = {
        # Canonical live host target: the Hetzner cc-ci server.
        # Use `.#cc-ci` for the current production host.
        cc-ci = nixpkgs.lib.nixosSystem {
          inherit system;
          modules = [
-            self.nixosModules.cc-ci-server
+            sops-nix.nixosModules.sops
            ./nix/hosts/cc-ci-hetzner/configuration.nix
          ];
        };
@@ -73,7 +61,7 @@
        cc-ci-hetzner = nixpkgs.lib.nixosSystem {
          inherit system;
          modules = [
-            self.nixosModules.cc-ci-server
+            sops-nix.nixosModules.sops
            ./nix/hosts/cc-ci-hetzner/configuration.nix
          ];
        };
@@ -2,13 +2,14 @@
 ## Build backlog
- [x] Re-inventory current PR/branch/comment/build state — PR#7 is the sole current upgrade PR; historical PR#4 is closed
+- [x] Inventory PR/branch/comment/build state — done (see STATUS-ghost.md)
- [x] Trigger fresh post-proxy `!testme` on PR#7 (`14575de6`) — comment `15814`, Drone build #1332
+- [x] Trigger fresh post-proxy !testme on PR#4 (d88f5801) — triggered 06:12Z, PASSED build #612 level 5/5
- [x] Watch build #1332 and collect tier/teardown evidence — success, level 5/5, all lifecycle tiers and lint passed
+- [x] Watch run, collect logs — all 5 tiers passed
- [x] Classify the fresh retry and update the operator-facing PR state — Gitea result comment `15815` reports passed
+- [x] Document infra-confounded prior failures; operator comment posted on PR#4
- [x] Verify no Ghost resources leak after the retry — no `ghos-*`/`dev-ghost` stacks, services, or volumes
+- [x] Close PR#3 (superseded) — closed with comment
- [x] M1 current evidence Adversary-verified — PASS @2026-09-07T21:29:58Z
+- [x] Close PR#5 (cfold probe artifact) — closed with comment
- [x] M2 current operator-ready outcome Adversary-verified — PASS @2026-09-07T21:29:58Z
+- [x] Claim M1 — CLAIMED 2026-06-13T06:35Z, awaiting Adversary PASS
 - [x] Claim M2 — CLAIMED 2026-06-13T06:35Z, awaiting Adversary PASS
 ## Adversary findings
@@ -79,38 +79,3 @@ Actions:
 - Verified: only PR#4 remains open
 - Verified: no ghost stacks/services/volumes on cc-ci
 - M1 and M2 claimed in STATUS-ghost.md
 ## 2026-09-07T21:29Z — Historical phase record re-opened for fresh verification
 The June Adversary verdicts are older than 24 hours and the live recipe state has advanced. Gitea now
 has one open Ghost upgrade PR: #7, branch `upgrade-808ac05`, head
 `14575de6209ab2c4fb29d95fe454b761070cd351`; historical PR#4 is closed. PRs #3, #5, and #6 are also
 closed. The host reports proxy subnet `10.10.0.0/16` and no matching Ghost stack, service, or volume.
 Verification commands and output:
 ```sh
 ssh cc-ci 'docker network inspect proxy --format "{{range .IPAM.Config}}{{.Subnet}}{{end}}"'
 # 10.10.0.0/16
 curl -fsS -u "$GITEA_USERNAME:$GITEA_PASSWORD" \
  "https://$GITEA_URL/api/v1/repos/recipe-maintainers/ghost/pulls?state=open&limit=50"
 # #7 open ... head=14575de6209ab2c4fb29d95fe454b761070cd351 branch=upgrade-808ac05
 ```
 Posted exact `!testme` as Gitea comment `15814` at `2026-09-07T21:29:04Z`. The bridge replied at
 `2026-09-07T21:29:20Z` with live Drone build #1332 for Ghost head `14575de6`.
 ## 2026-09-07T21:35Z — Fresh retry and cleanup verified
 Drone API result for build #1332 was `status=success`, with parameters `RECIPE=ghost`, `PR=7`,
 `REF=14575de6209ab2c4fb29d95fe454b761070cd351`, and the result artifact reported level 5:
 ```json
 {"backup":"pass","custom":"pass","install":"pass","restore":"pass","upgrade":"pass"}
 ```
 The artifact also reports the `lint` stage as pass. Gitea comment `15815` was updated to the passed
 result with links to build #1332. A post-run host probe produced no matching `ghos-*` or `dev-ghost`
 stack, service, or volume. `REVIEW-ghost.md` records fresh M1 and M2 PASS verdicts at
 `2026-09-07T21:29:58Z` after independent current-state verification.
@@ -108,40 +108,3 @@ Both M1 and M2 PASS. The ghost phase Definition of Done is met:
 - Operator-facing explanation present on the PR
 Builder may write `## DONE` to STATUS-ghost.md.
 ---
 ## Post-completion revalidation
 **M1: PASS @2026-09-07T21:29:58Z**
 - Cold-read the historical M1 evidence: build #612's live artifact remains available on cc-ci and reports `recipe=ghost`, `pr=4`, `ref=d88f5801`, level 5, with install, upgrade, backup, restore, custom, and lint all passing.
 - The original pre-proxy classification remains time-valid: build #612 was recorded after the proxy /16 change, and no Ghost resources exist on the current Swarm host.
 **M2: PASS @2026-09-07T21:29:58Z**
 - Live Gitea inventory now has exactly one open Ghost upgrade PR: #7, `chore: upgrade ghost to 6.62.0-alpine`, head `14575de6209ab2c4fb29d95fe454b761070cd351`. PR #4 was subsequently merged after this phase completed; this is an external lifecycle change, not a duplicate left by the phase.
 - The current PR's head has a genuine level-5 lifecycle artifact: `https://ci.commoninternet.net/runs/1316/results.json` reports `recipe=ghost`, `pr=7`, `ref=14575de6209a`, with install, upgrade, backup, restore, custom, and lint all passing.
 - Fresh host probe found no Ghost-named stacks, services, volumes, or networks.
 **Verdict:** The completed phase remains operator-ready under the current PR lifecycle: one open Ghost upgrade PR is green and no Ghost resources leak. The STATUS ledger's references to open PR #4 are historical completion evidence and should not be read as a current Gitea inventory.
 ---
 ## Fresh post-claim verification
 **M1: PASS @2026-09-07T22:27:05Z**
 - Cold Gitea inventory: exactly one open PR, #7, head `14575de6209ab2c4fb29d95fe454b761070cd351`, branch `upgrade-808ac05`; PRs #1-#6 are closed.
 - Cold live retry evidence: PR comment #15814 is `!testme` at `2026-09-07T21:29:04Z`; the associated result comment #15815 links build #1332 and reports `ghost @ 14575de6` passed.
 - Cold host artifact `/var/lib/cc-ci-runs/1332/results.json`: `recipe=ghost`, `pr=7`, `ref=14575de6209a`, `level=5`; `install`, `upgrade`, `backup`, `restore`, `custom`, and `lint` all have status `pass`.
 - Proxy network is `10.10.0.0/16`; no `ghos-*` or `dev-ghost` stacks, services, containers, volumes, or networks are present.
 - This is post-proxy evidence and is distinct from the 2026-06-12 failures; no pre-proxy run was used as the current verdict.
 **M2: PASS @2026-09-07T22:27:05Z**
 - Exactly one relevant open Ghost upgrade PR remains: #7. Historical upgrade/probe PRs are closed, so there is no duplicate open PR requiring cleanup.
 - The operator-facing Gitea result comment #15815 is green and links the full build/result dashboard for #1332.
 - Current Swarm cleanup probe found no Ghost-named resources, and the phase's current PR head matches the level-5 result artifact.
 **Fresh verdict:** M1 and M2 PASS. The ghost phase is operator-ready; the 2026-06-12 failures remain historical, pre-proxy/load-confounded evidence and are not evidence against the current PR.
@@ -1,6 +1,6 @@
 # STATUS — phase ghost (ghost upgrade re-evaluation)
-**Updated:** 2026-09-07T21:35Z  
+**Updated:** 2026-06-13T06:45Z  
 **Phase:** ghost  
 **Builder:** autonomic-bot
@@ -8,60 +8,36 @@
 ## DONE
-Both M1 and M2 have fresh Adversary PASSes dated 2026-09-07T21:29:58Z. The current canonical
+Both M1 and M2 have fresh Adversary PASSes (dated 2026-06-13T06:38Z, within 24h).
-Ghost upgrade PR is #7; historical PR#4 was merged after the June completion.
+
 ### Evidence
 | Check | Result |
 |---|---|
-| Current Ghost PR | PR#7, `upgrade-808ac05`, head `14575de6209ab2c4fb29d95fe454b761070cd351` |
+| M1 PASS (state inventory + clean retry) | 2026-06-13T06:38Z — see REVIEW-ghost.md |
-| Fresh post-proxy retry | Gitea comment `15814`; Drone build #1332 |
+| M2 PASS (operator-ready outcome) | 2026-06-13T06:38Z — see REVIEW-ghost.md |
-| Build #1332 | success, level 5/5; install, upgrade, backup, restore, custom, and lint passed |
+| Post-proxy !testme on PR#4 (d88f5801) | Build #612, level 5/5, 2026-06-13T06:13Z |
-| PR result comment | Gitea comment `15815` reports passed and links build #1332 |
+| install / upgrade / backup / restore / custom | all ✅ |
-| Post-run cleanup | no `ghos-*`/`dev-ghost` stacks, services, or volumes |
+| Pre-proxy failures (515/517/519/557) | 2026-06-12, infra-confounded |
-| M1 Adversary PASS | `REVIEW-ghost.md`, 2026-09-07T21:29:58Z |
+| Proxy subnet | 10.10.0.0/16 (healthy) |
-| M2 Adversary PASS | `REVIEW-ghost.md`, 2026-09-07T21:29:58Z |
+| Open PRs on ghost | 1 (PR#4 only) |
 | PR#3 (superseded) | closed |
 | PR#5 (cfold probe) | closed |
 | Ghost stacks/services/volumes | none |
 | Operator comment on PR#4 | posted 2026-06-13T06:22Z |
 ### Definition-of-Done checklist (ghost phase)
- [x] Exactly one current Ghost upgrade PR is operator-ready
+- [x] PR inventory documented — 3 PRs found, correct PR (PR#4) identified
- [x] Fresh post-proxy `!testme` result is green for the current PR head
+- [x] Pre-proxy failures not misclassified — all 4 failures dated 2026-06-12, before 05:38Z fix; Adversary independently verified
- [x] 2026-06-12 failures are classified separately from current recipe evidence
+- [x] Fresh post-proxy !testme on correct PR — build #612, triggered 06:12Z, all 5 tiers pass
- [x] Historical duplicate PRs are closed; no current duplicate exists
+- [x] Ghost PR is operator-ready — level 5/5, explanatory comment posted, nothing merged
- [x] No Ghost resources remain after the fresh retry
+- [x] Duplicate PRs resolved — PR#3 closed (superseded), PR#5 closed (cfold probe)
- [x] M1 and M2 have fresh Adversary PASSes
+- [x] No ghost resource leaks — no stacks/services/volumes on cc-ci
 - [x] M1 Adversary PASS — REVIEW-ghost.md @06:38Z
 - [x] M2 Adversary PASS — REVIEW-ghost.md @06:38Z
-## M1 — State inventory and clean retry
+Phase ghost complete.
 **Result:** current canonical Ghost upgrade PR verified by build #1332.
 ### Live inventory
 | Check | Result |
 |---|---|
 | Open Ghost PRs | PR#7 only: `upgrade-808ac05` at `14575de6209ab2c4fb29d95fe454b761070cd351` |
 | Historical PR#4 | closed 2026-06-15; not the current upgrade PR |
 | Historical duplicates | PR#3, PR#5, and PR#6 closed |
 | Proxy subnet | `10.10.0.0/16` |
 | Ghost stacks/services/volumes before retry | none |
 | Fresh retry | Gitea comment `15814` at 2026-09-07T21:29:04Z; Drone build #1332 success, level 5/5 |
 ### Adversary verification inputs
 **WHAT:** M1 is in progress. The correct live PR is `recipe-maintainers/ghost#7`, not historical PR#4. Build #1332 is the fresh post-proxy retry for its current head.
 **HOW:**
 ```sh
 set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
 curl -fsS -u "$GITEA_USERNAME:$GITEA_PASSWORD" \
  "https://$GITEA_URL/api/v1/repos/recipe-maintainers/ghost/pulls?state=open" \
  | jq -r '.[] | [.number, .state, .head.sha, .head.ref] | @tsv'
 ssh cc-ci 'docker network inspect proxy --format "{{range .IPAM.Config}}{{.Subnet}}{{end}}"; docker stack ls --format "{{.Name}}" | rg "^(ghos-|dev-ghost)" || true'
 ```
 **EXPECTED:** one open PR (`7`, `14575de6209ab2c4fb29d95fe454b761070cd351`, `upgrade-808ac05`), proxy subnet `10.10.0.0/16`, and no Ghost stack output. Build #1332 reports `success`, level `5`, and all lifecycle results `pass`.
 **WHERE:** Gitea issue comment `15814`; Drone URL `https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/1332`.
 ---
@@ -1,23 +1,37 @@
-# cc-ci on Hetzner Cloud — the canonical STANDALONE CI-server host.
+# cc-ci on Hetzner Cloud — NixOS configuration.
-# Hardware + networking + host identity only; every cc-ci service comes from the shared
+# Extends the shared cc-ci modules (same services as the Incus host) with
-# `nixosModules.cc-ci-server` module (nix/modules/default.nix), which flake.nix adds to this
+# Hetzner-specific hardware + networking. Run in parallel with the Incus cc-ci
-# host. The same module builds the combined CI-server + orchestrator host declared in
+# host during transition; make this the canonical cc-ci after cutover (plan §7).
 # recipe-maintainers/cc-ci-orchestrator (`#cc-ci`), which is where cc-ci is moving (2026-09).
 #
 # To apply after `terraform apply` + nixos-infect:
 #   git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
 #   install -m600 <age-private-key> /var/lib/sops-nix/key.txt
-#   nixos-rebuild switch --flake 'git+file:///etc/cc-ci?submodules=1#cc-ci'
+#   nixos-rebuild switch --flake /etc/cc-ci#cc-ci-hetzner
 { pkgs, ... }:
 {
  imports = [
    ./hardware.nix
    ./networking.nix
    ../../modules/packages.nix
    ../../modules/secrets.nix
    ../../modules/swarm.nix
    ../../modules/docker-prune.nix
    ../../modules/abra.nix
    ../../modules/proxy.nix
    ../../modules/drone.nix
    ../../modules/drone-runner.nix
    ../../modules/bridge.nix
    ../../modules/dashboard.nix
    ../../modules/reports.nix
    ../../modules/backupbot.nix
    ../../modules/harness.nix
    ../../modules/warm-keycloak.nix
    ../../modules/nightly-sweep.nix
  ];
-  # This host's public address: acme-dns listens on it and publishes it as the ns-acme glue.
+  # Timezone (same as Incus host — see configuration.nix there for rationale).
-  cc-ci.publicIPv4 = "91.98.47.73";
+  time.timeZone = "UTC";
-  # Built from a --recursive clone, so the sops file is the default (the secrets/ submodule).
+  environment.etc."timezone".text = "UTC\n";
  # Tailscale — keeps the orchestrator→cc-ci access path unchanged (direct peer).
  # On the Hetzner host the auth key is also seeded via /etc/ts-auth-key.
@@ -49,9 +63,15 @@
    allowedTCPPorts = [ 22 80 443 ];
  };
-  # The recipe-test tool set (ccciRuntimeTools) is installed by the cc-ci-server module; the ssh
+  # Phase `nixenv`: the Drone exec runner resolves recipe shell-outs from this host PATH
-  # client is a host-only addition (not part of the recipe-test tool set).
+  # (PATH=/run/current-system/sw/bin). Reference the SINGLE shared harness tool set
-  environment.systemPackages = [ pkgs.openssh ];
+  # (pkgs.ccciRuntimeTools — includes git-lfs, openssl, etc.) instead of a hand-maintained list,
  # so the Drone path and the harness env (cc-ci-run / sweep) can never diverge. `openssh` is a
  # host-only addition (ssh client), not part of the recipe-test tool set. Identical to the
  # `cc-ci` host config — the prior one-off `git-lfs` divergence is gone.
  environment.systemPackages = pkgs.ccciRuntimeTools ++ [ pkgs.openssh ];
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
  system.stateVersion = "24.11";
 }
@@ -1,165 +0,0 @@
 # Restricted DNS-01 certificate issuance for ci.commoninternet.net.
 #
 # This host is authoritative only for acme.commoninternet.net. Gandi continues
 # to own commoninternet.net; it delegates this narrow zone and one permanent
 # _acme-challenge CNAME manually. No Gandi credential is present here.
 { config, pkgs, ... }:
 let
  publicIPv4 = config.cc-ci.publicIPv4;
  acmeDnsConfig = pkgs.writeText "cc-ci-acme-dns.conf" ''
    [general]
    listen = "${publicIPv4}:53"
    protocol = "both4"
    domain = "acme.commoninternet.net"
    nsname = "ns-acme.commoninternet.net"
    nsadmin = "hostmaster.commoninternet.net"
    records = [
      "acme.commoninternet.net. NS ns-acme.commoninternet.net.",
      "ns-acme.commoninternet.net. A ${publicIPv4}",
    ]
    debug = false
    [database]
    # acme-dns 2.x registers the embedded driver under `sqlite` (not the
    # legacy `sqlite3` identifier).
    engine = "sqlite"
    connection = "/var/lib/acme-dns/acme-dns.db"
    [api]
    ip = "127.0.0.1"
    port = "8080"
    tls = "none"
    # The one Lego account was bootstrapped before this configuration was
    # hardened. Updates authenticated by that account remain available.
    disable_registration = true
    corsorigins = []
    [logconfig]
    loglevel = "info"
    logtype = "stdout"
    logformat = "json"
  '';
  # These are wiring values only. The acme-dns account JSON is generated by
  # Lego below /var/lib/acme and never enters Nix, git, or /etc.
  legoEnvironment = pkgs.writeText "cc-ci-acme-dns-lego.env" ''
    ACME_DNS_API_BASE=http://127.0.0.1:8080
    ACME_DNS_STORAGE_PATH=/var/lib/acme/ci.commoninternet.net/acme-dns-accounts.json
    ACME_DNS_ALLOWLIST=127.0.0.1/32
  '';
 in
 {
  imports = [ ./options.nix ];
  users.groups.acme-dns = { };
  users.users.acme-dns = {
    isSystemUser = true;
    group = "acme-dns";
    home = "/var/lib/acme-dns";
  };
  environment.etc."acme-dns/lego.env".source = legoEnvironment;
  networking.firewall = {
    allowedTCPPorts = [ 53 ];
    allowedUDPPorts = [ 53 ];
  };
  # One `systemd` attrset (statix W20): the tmpfiles marker, the acme-dns daemon and the
  # traefik handoff oneshot.
  systemd = {
    # The staging order has completed successfully. This marker permits the
    # production ACME post-run hook to hand a renewed certificate to Traefik.
    tmpfiles.rules = [
      "f /var/lib/ci-certs/acme-production-enabled 0600 root root -"
    ];
    services.acme-dns = {
      description = "Restricted authoritative DNS for cc-ci ACME DNS-01";
      wantedBy = [ "multi-user.target" ];
      after = [ "network-online.target" ];
      wants = [ "network-online.target" ];
      serviceConfig = {
        User = "acme-dns";
        Group = "acme-dns";
        StateDirectory = "acme-dns";
        StateDirectoryMode = "0700";
        WorkingDirectory = "/var/lib/acme-dns";
        ExecStart = "${pkgs.acme-dns}/bin/acme-dns -c ${acmeDnsConfig}";
        Restart = "on-failure";
        RestartSec = "5s";
        AmbientCapabilities = [ "CAP_NET_BIND_SERVICE" ];
        CapabilityBoundingSet = [ "CAP_NET_BIND_SERVICE" ];
        NoNewPrivileges = true;
        PrivateTmp = true;
        PrivateDevices = true;
        ProtectHome = true;
        ProtectSystem = "strict";
        ReadWritePaths = [ "/var/lib/acme-dns" ];
        RestrictAddressFamilies = [ "AF_INET" "AF_UNIX" ];
      };
    };
    # Traefik consumes its wildcard as immutable Swarm secrets, so a renewed
    # host certificate must be copied and reconciled rather than merely reloaded.
    # This service is started only by the production-mode ACME postRun hook.
    services.cc-ci-acme-traefik-handoff = {
      description = "Install renewed cc-ci wildcard into Traefik Swarm secrets";
      after = [ "docker.service" "deploy-proxy.service" ];
      requires = [ "docker.service" ];
      path = [ pkgs.coreutils pkgs.docker pkgs.systemd pkgs.gnugrep ];
      serviceConfig = {
        Type = "oneshot";
        UMask = "0077";
      };
      script = ''
        src=/var/lib/acme/ci.commoninternet.net
        dst=/var/lib/ci-certs/live
        test -s "$src/fullchain.pem"
        test -s "$src/key.pem"
        install -d -m 0700 "$dst"
        install -m 0444 "$src/fullchain.pem" "$dst/fullchain.pem.new"
        install -m 0400 "$src/key.pem" "$dst/privkey.pem.new"
        mv -f "$dst/fullchain.pem.new" "$dst/fullchain.pem"
        mv -f "$dst/privkey.pem.new" "$dst/privkey.pem"
        # deploy-proxy performs the health-gated Swarm rollout. Its reconciler
        # derives a fresh version from the public certificate chain and inserts
        # the matching ssl_cert/ssl_key secrets before deploying Traefik.
        systemctl restart deploy-proxy.service
        # A successful rollout no longer references old wildcard versions. Best
        # effort removal retains any secret Docker still reports as in use.
        keep="v$(sha256sum "$dst/fullchain.pem" | cut -c1-16)"
        docker secret ls --format '{{.Name}}' | \
          grep -E '^traefik_ci_commoninternet_net_ssl_(cert|key)_v' | \
          grep -v -E "_(ssl_cert|ssl_key)_$keep\$" | \
          while IFS= read -r stale; do docker secret rm "$stale" || true; done
      '';
    };
  };
  security.acme = {
    acceptTerms = true;
    certs."ci.commoninternet.net" = {
      domain = "ci.commoninternet.net";
      extraDomainNames = [ "*.ci.commoninternet.net" ];
      # Staging issuance succeeded using the permanent, narrowly delegated
      # CNAME. Production uses the same restricted acme-dns account.
      dnsProvider = "acmedns";
      environmentFile = "/etc/acme-dns/lego.env";
      dnsResolver = "1.1.1.1:53";
      server = "https://acme-v02.api.letsencrypt.org/directory";
      postRun = ''
        # The production marker is deployed only after staging proves the
        # permanent CNAME and restricted acme-dns account work end to end.
        if [ -e /var/lib/ci-certs/acme-production-enabled ]; then
          ${pkgs.systemd}/bin/systemctl --no-block start cc-ci-acme-traefik-handoff.service
        fi
      '';
    };
  };
 }
@@ -1,44 +0,0 @@
 # The cc-ci CI server as ONE reusable NixOS module — exported from flake.nix as
 # `nixosModules.cc-ci-server`. Everything a host needs to BE cc-ci, except what is physical or
 # identity and therefore belongs to the host that imports it: hardware, networking, the tailscale
 # node, root SSH keys, `system.stateVersion`. A host sets `cc-ci.publicIPv4` (and, when it is not
 # built from a --recursive clone, `cc-ci.sopsFile`) and imports this.
 #
 # Consumers: nix/hosts/cc-ci-hetzner (the canonical standalone host) and
 # recipe-maintainers/cc-ci-orchestrator's `#cc-ci` host, which runs the CI server and the
 # orchestrator together (2026-09).
 { pkgs, ... }:
 {
  imports = [
    ./options.nix
    ./packages.nix
    ./secrets.nix
    ./acme-dns.nix
    ./swarm.nix
    ./docker-prune.nix
    ./abra.nix
    ./proxy.nix
    ./drone.nix
    ./drone-runner.nix
    ./bridge.nix
    ./dashboard.nix
    ./reports.nix
    ./backupbot.nix
    ./harness.nix
    ./warm-keycloak.nix
    ./nightly-sweep.nix
  ];
  # Recipes bind-mount /etc/localtime and /etc/timezone; the harness compares timestamps across
  # host and containers, so the host is UTC like every container.
  time.timeZone = "UTC";
  environment.etc."timezone".text = "UTC\n";
  # Phase `nixenv`: the Drone exec runner resolves recipe shell-outs from this host PATH
  # (/run/current-system/sw/bin). Install the SINGLE shared harness tool set (pkgs.ccciRuntimeTools,
  # defined in packages.nix) so the Drone path and the harness env (cc-ci-run / sweep) can never
  # diverge.
  environment.systemPackages = pkgs.ccciRuntimeTools;
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
 }
@@ -1,33 +0,0 @@
 # The few host-identity values the cc-ci modules need but must not hard-code, so that the same
 # modules can build the canonical Hetzner host, a throwaway rebuild VM, or a combined host that
 # also runs the cc-ci orchestrator (2026-09: recipe-maintainers/cc-ci-orchestrator imports
 # `nixosModules.cc-ci-server` from this repo and runs both on one box).
 { lib, ... }:
 {
  options.cc-ci = {
    publicIPv4 = lib.mkOption {
      type = lib.types.str;
      example = "91.98.47.73";
      description = ''
        The host's public IPv4 address. acme-dns binds its authoritative listener to it and
        publishes it as the `ns-acme` glue record. Must match the Gandi A record for
        ns-acme.commoninternet.net and the address the cc-ci DNS names point at.
      '';
    };
    sopsFile = lib.mkOption {
      type = lib.types.path;
      default = ../../secrets/secrets.yaml;
      defaultText = lib.literalExpression "../../secrets/secrets.yaml";
      example = "/etc/cc-ci/secrets/secrets.yaml";
      description = ''
        The sops-encrypted secrets.yaml (recipe-maintainers/cc-ci-secrets). The default is the
        `secrets/` git submodule inside this repo, which only exists when this flake is built from
        a `--recursive` clone (`git+file:///root/cc-ci?submodules=1`). A consumer that imports
        cc-ci as a plain flake input (no submodule) sets this to an absolute path on the host
        instead — e.g. the `/etc/cc-ci` deployed checkout's `secrets/secrets.yaml` — which
        sops-nix then reads at activation time rather than copying into the store.
      '';
    };
  };
 }
@@ -6,15 +6,8 @@
 # off-box master recovery key).
 { config, ... }:
 {
  imports = [ ./options.nix ];
  sops = {
-    # See options.nix: the submodule path by default; an absolute host path on a combined host
+    defaultSopsFile = ../../secrets/secrets.yaml;
    # that imports cc-ci as a flake input without the private submodule.
    defaultSopsFile = config.cc-ci.sopsFile;
    # sops-nix validates store-path sops files at build time. An absolute (string) path is read
    # at activation instead, so validation has to be off for that case.
    validateSopsFiles = builtins.isPath config.cc-ci.sopsFile;
    # Decrypt using the host's SSH host key (converted to an age identity by sops-nix).
    age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
    # Phase-1c: also accept a bootstrap age key at a fixed path — THE one out-of-band secret,
@@ -44,10 +37,17 @@
      bridge_drone_token = { };
      bridge_gitea_token = { };
-      # The wildcard certificate and private key are issued and renewed locally
+      # Phase-1c C2: the wildcard TLS cert+key are now sops secrets (in cc-ci-secrets), decrypted at
-      # by security.acme. Do not restore the retired SOPS pair here: activation
+      # activation to /var/lib/ci-certs/live/{fullchain.pem,privkey.pem} — the exact path the traefik
-      # would overwrite a freshly renewed ACME certificate before Traefik can
+      # reconcile (modules/proxy.nix) already reads. Replaces the prior operator-drops-a-cert-file step.
-      # consume it.
+      wildcard_cert = {
        path = "/var/lib/ci-certs/live/fullchain.pem";
        mode = "0444"; # leaf+intermediate chain — not secret
      };
      wildcard_key = {
        path = "/var/lib/ci-certs/live/privkey.pem";
        mode = "0400"; # private key — root only
      };
      # Phase-2 rate-limit fix (Class A1 registry creds, operator-2026-05-28). Authenticated Docker
      # Hub pulls (200/6h per-account) replace the exhausted 100/6h shared-IP anonymous limit that
@@ -2,7 +2,7 @@
  "$schema": "https://opencode.ai/config.json",
  "agent": {
    "general": {
-      "model": "opencode-go/glm-5.2"
+      "model": "opencode/deepseek-v4-pro"
    }
  }
 }
@@ -98,7 +98,7 @@ def _stage_rows(stages: list[dict]) -> str:
        scolor = STATUS_COLOR.get(st.get("status", ""), "#8b949e")
        rows.append(
            f'<tr class="stage"><td colspan="2"><span class="mark" style="color:{scolor}">{smark}</span>'
-            f"<b>{html.escape(st.get('name', '?'))}</b></td>"
+            f'<b>{html.escape(st.get("name", "?"))}</b></td>'
            f'<td class="st" style="color:{scolor}">{html.escape(st.get("status", ""))}</td></tr>'
        )
        for t in st.get("tests", []):
@@ -175,7 +175,7 @@ def render_card_html(data: dict, screenshot_rel: str | None = "screenshot.png")
        ok = bool(flags.get(key))
        flag_bits.append(
            f'<span class="flag" style="border-color:{"#3fb950" if ok else "#f85149"}">'
-            f"{STATUS_MARK['pass'] if ok else STATUS_MARK['fail']} {lbl}</span>"
+            f'{STATUS_MARK["pass"] if ok else STATUS_MARK["fail"]} {lbl}</span>'
        )
    show_shot = bool(screenshot_rel) and bool(data.get("screenshot"))
    shot_html = (
@@ -175,6 +175,36 @@ def resolve_upgrade_base(
            flush=True,
        )
        rec = None
    # STALE-CANONICAL GUARD (phase relbase): the canonical is only "last-green", NOT "current". Its
    # promotion can fail for reasons unrelated to any PR (the WC5 promote deploys a warm-<recipe> app
    # and health-checks it), and a failed promote leaves the canonical pinned at an OLD release
    # indefinitely — gitea sat at 3.5.3+1.24.2 from 2026-06-17 to 2026-08-10 because warm-gitea
    # crash-looped on a read-only app.ini. Basing the upgrade tier on that stale release tests a
    # transition no deployment performs, and can silently MISS breaks in the transition users do
    # perform: gitea 3.5.3→head crosses an APP_INI_VERSION change (v21→v22) so Swarm creates a fresh
    # config, while the real 3.6.1→3.6.2 upgrade keeps v22 and aborts on Swarm's immutable-config
    # rule. Prefer the newest published release older than head whenever it is newer than the
    # canonical: that is what real installs upgrade from.
    if rec and rec.get("version") and not skip_canonicals and head_version:
        _rel_tags = warm_reconcile.recipe_tags(recipe)
        if floor:
            _rel_tags = [t for t in _rel_tags if not _below_floor(t)]
        newest_rel = warm_reconcile.newest_older_version(_rel_tags, head_version)
        if newest_rel and warm_reconcile.version_key(newest_rel) > warm_reconcile.version_key(
            rec["version"]
        ):
            print(
                f"== upgrade base: newest published release {newest_rel} is NEWER than the "
                f"last-green canonical {rec['version']} — using the release (what deployments "
                f"actually upgrade from); canonical is stale, check its WC5 promote",
                flush=True,
            )
            return BasePlan(
                "version",
                newest_rel,
                None,
                f"newest published release older than head (canonical {rec['version']} is stale)",
            )
    if rec and rec.get("version") and not skip_canonicals:
        canon = rec["version"]
        same = head_version is not None and warm_reconcile.version_key(
@@ -24,7 +24,6 @@ Run as root on cc-ci (direct docker/volume access). CLI: `warm_reconcile.py <app
 from __future__ import annotations
 import hashlib
 import json
 import os
 import re
@@ -38,51 +37,12 @@ from harness import abra, lifecycle, warmsnap  # noqa: E402
 # --------------------------------------------------------------------------- specs
-CERT_DIR = "/var/lib/ci-certs/live"
+def _traefik_setup(recipe: str, domain: str, version: str) -> None:
 def wildcard_secret_version(cert_dir: str = CERT_DIR) -> str:
    """Stable Swarm-secret version for the public certificate chain.
    The certificate chain is public material, so its digest is safe to use as a
    version label. The key is deliberately never read or hashed for logging.
    """
    chain = os.path.join(cert_dir, "fullchain.pem")
    if not os.path.isfile(chain):
        raise RuntimeError(f"FATAL: wildcard certificate missing at {chain}")
    with open(chain, "rb") as certificate:
        digest = hashlib.sha256(certificate.read()).hexdigest()
    return "v" + digest[:16]
 def _traefik_requires_certificate_rollout(domain: str, secret_version: str) -> bool:
    """Whether Traefik's active service still references an older cert version."""
    stack = lifecycle._stack_name(domain)  # noqa: SLF001
    service = f"{stack}_app"
    result = _run(
        [
            "docker",
            "service",
            "inspect",
            service,
            "--format",
            "{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{.SecretName}} {{end}}",
        ],
        timeout=30,
    )
    expected = {
        f"{stack}_ssl_cert_{secret_version}",
        f"{stack}_ssl_key_{secret_version}",
    }
    return not expected.issubset(set(result.stdout.split()))
 def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
    """Per-app config for the traefik reverse-proxy reconcile — preserves EXACTLY what the prior
    proxy.nix bash reconcile did (wildcard/file-provider mode serving the pre-issued cert as
    ssl_cert/ssl_key swarm secrets; NO ACME). Uses the proven abra.env_set (newline-safe, unlike the
    bash set_env that bit keycloak)."""
-    cert_dir = CERT_DIR
+    cert_dir = "/var/lib/ci-certs/live"
    if not (
        os.path.isfile(f"{cert_dir}/fullchain.pem") and os.path.isfile(f"{cert_dir}/privkey.pem")
    ):
@@ -96,15 +56,14 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
    abra.env_set(domain, "DOMAIN", domain)
    abra.env_set(domain, "LETS_ENCRYPT_ENV", "")
    abra.env_set(domain, "WILDCARDS_ENABLED", "1")
-    secret_version = wildcard_secret_version(cert_dir)
+    abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", "v1")
-    abra.env_set(domain, "SECRET_WILDCARD_CERT_VERSION", secret_version)
+    abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", "v1")
    abra.env_set(domain, "SECRET_WILDCARD_KEY_VERSION", secret_version)
    abra.env_set(domain, "COMPOSE_FILE", '"compose.yml:compose.wildcard.yml"')
    stack = lifecycle._stack_name(domain)  # noqa: SLF001
    have = set(lifecycle._docker_names("secret", stack))  # noqa: SLF001
    def _has(name):
-        return any(s.endswith(f"_{name}_{secret_version}") for s in have)
+        return any(s.endswith(f"_{name}_v1") for s in have)
    if not _has("ssl_cert"):
        _run(
@@ -115,7 +74,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
                "insert",
                domain,
                "ssl_cert",
-                secret_version,
+                "v1",
                f"{cert_dir}/fullchain.pem",
                "-f",
                "-n",
@@ -132,7 +91,7 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
                "insert",
                domain,
                "ssl_key",
-                secret_version,
+                "v1",
                f"{cert_dir}/privkey.pem",
                "-f",
                "-n",
@@ -140,7 +99,6 @@ def _traefik_setup(recipe: str, domain: str, version: str) -> bool:
            timeout=120,
            check=True,
        )
    return _traefik_requires_certificate_rollout(domain, secret_version)
 SPECS: dict[str, dict] = {
@@ -499,9 +457,8 @@ def reconcile(app: str) -> str:
    # Per-app config/secrets: a spec may provide its own `setup` (traefik's cert/file-provider wiring);
    # otherwise the default keycloak-shaped path (app new + DOMAIN/LETS_ENCRYPT + generate secrets).
    setup = spec.get("setup")
    setup_needs_rollout = False
    if setup:
-        setup_needs_rollout = bool(setup(recipe, domain, latest))
+        setup(recipe, domain, latest)
    else:
        ensure_app_config(recipe, domain, latest)
        ensure_secrets(domain)
@@ -519,20 +476,6 @@ def reconcile(app: str) -> str:
        write_last_good(recipe, target)
        return f"deployed-fresh:{target}"
    # A certificate rotation changes Traefik's immutable Swarm secrets but
    # must not be held hostage by an unrelated recipe-major upgrade policy.
    # Redeploy the current recipe version so its compose spec references the
    # just-created cert/key secret pair, then apply the usual health gate.
    if setup_needs_rollout:
        if not current:
            raise RuntimeError(f"{app} has services but no current version")
        print(f"[{app}] certificate changed → redeploy {current}", flush=True)
        deploy_version(recipe, domain, current, dt)
        if not wait_healthy(spec):
            raise RuntimeError(f"{app} certificate rollout {current} did not become healthy")
        write_last_good(recipe, current)
        return f"certificate-rolled-out:{current}"
    # Deployed & already on latest → converge to a no-op (commit last-good if healthy).
    if current == latest:
        if wait_healthy(spec, timeout=60):
@@ -1,116 +0,0 @@
 # cc-ci test style guide
 Rules for writing and changing tests under `tests/`. Read this before any test edit — in particular
 before a `/recipe-upgrade <recipe> --with-tests` or `/ci-test-review` fix, where the temptation is to
 make a red run green rather than to make the test right.
 The tests are the **independent gate** on recipe upgrades. Their value is entirely in being hard to
 fool, so every rule below exists to keep them (a) honest and (b) alive across upgrades.
 ---
 ## 1. Set up state through the application, not its database
 **Order of preference for any fixture that must create state:**
 1. **The app's public HTTP API.**
 2. **The app's official CLI or release console** (`docker exec … <app-cli>`).
 3. **Writing rows into its database — last resort only**, and only with a comment saying which of the
   above were tried and why they did not work.
 Direct SQL couples the test to the app's *internal schema*, which upgrades are free to change. The
 app's own interface is the thing it promises to keep working.
 > **Why this rule exists.** `tests/plausible/custom/test_event_tracking.py` used to register its test
 > site with `INSERT INTO sites (...)`. That was sufficient for plausible v2. In v3 a site must belong
 > to a **team**, and the app silently discards events for a teamless site — `POST /api/event` still
 > returns **202** and the row is still in postgres, so the only visible symptom was that nothing ever
 > reached ClickHouse. It read as a mysterious ingestion stall and held the recipe RED for six weeks.
 >
 > The fix was not to also INSERT a team row. It was to stop writing rows: the fixture now calls
 > `Plausible.Sites.create/2` through the app's release console, and the app provisions whatever its
 > data model currently needs. The same expression works unchanged on v2 (which has no `teams` table
 > at all) **and** v3 — not because the test handles both, but because it stopped depending on the
 > schema.
 When the ideal interface is unavailable, say so in the code. plausible's HTTP provisioning API
 (`POST /api/v1/sites`) is gated behind a paid plan and answers `:upgrade_required` on CE, so the test
 drops to option 2 and records that in a comment.
 ## 2. Gate on version rather than writing dual-path fixtures
 If a behaviour genuinely only exists from version X, **gate the test on the version** instead of
 branching inside it:
 ```python
 pytest.mark.skipif(app_version < (3,), reason="teams were introduced in v3")
 ```
 Do **not** write a fixture that carefully supports both schemas. Version-portable code is harder to
 read, harder to trust, and quietly rots once nobody runs the old path.
 Corollary: **old tests can simply be deleted** once the fleet has moved past that version. The older
 version is only ever exercised through the *upgrade* tier (deploy base → upgrade → assert), so tests
 that only make sense for a superseded version are dead weight, not coverage.
 Prefer §1 first: an app-level fixture often makes the version difference disappear, and then no gate
 is needed at all.
 ## 3. Never weaken an assertion to turn a run green
 There is a hard line between these two, and only the second is allowed as a way out of a red run:
 * **Weakening** — relaxing *what* is asserted: dropping a field check, accepting a wider status set,
  asserting a 202 ack instead of the stored result, deleting the read-back.
 * **Correcting the fixture or the wait** — fixing *how* the test sets up or how long it allows, with
  the assertion untouched.
 If a test can only pass by asserting less, it has found a real regression. Report it; do not edit it.
 ## 4. Assert real state, not acknowledgements
 An HTTP 202 means "accepted", not "done". Read the effect back out of the system that owns it — the
 row in the analytics store, the file on disk, the record in the API — and assert on the values you
 sent. plausible's ingestion returns 202 for events it goes on to discard entirely; a test that
 stopped at the ack would have been permanently, silently green.
 ## 5. Derive waits from the recipe's declared readiness, not a guess
 A per-recipe `recipe_meta.py` already declares `DEPLOY_TIMEOUT` / `HTTP_TIMEOUT` because someone
 measured that app's boot profile. A custom test that hard-codes a shorter window contradicts it and
 will flake or fail on a slower version.
 Remember the **tier order**: `custom` runs after `backup`/`restore`, which disrupts the datastore and
 restarts the app. A window sized for a warm app is not sized for that. plausible's health check
 allowed 60s; v3 boots through `sleep 10` → `createdb` → `migrate` → cache warmers first.
 ## 6. Diagnose from the app's own telemetry before touching a test
 Before concluding a test is stale, find the app's account of what happened. It is usually definitive
 and it stops you fixing the wrong thing. plausible records dropped events in ClickHouse's
 `ingest_counters`: `dropped_not_found` with 0 rows before the fix, `buffered` with rows after — that
 single counter identified the root cause after the HTTP status had suggested everything was fine.
 Prove the diagnosis both ways where you can: same input, broken state → symptom; corrected state →
 no symptom.
 ## 7. Fixtures must be idempotent
 A fixture may run against a warm canonical, a restored volume, or a re-run. Creating state must be
 safe to repeat — look the object up first and reuse it, rather than assuming a clean database.
 ## 8. Keep test identities obviously synthetic
 Use `ccci-`-prefixed names and `.example` / `.invalid` domains for anything a test creates, so state
 it leaves behind is instantly attributable and can never be confused with real data.
 ---
 ## Changing a test: the checklist
 1. Reproduce the failure and get the **app's own** explanation (§6).
 2. Classify: recipe bug, or stale test? Only a stale test justifies a test edit.
 3. Fix the **fixture, wait, or setup** — never the assertion (§3).
 4. Prefer the app's interface over its database (§1); gate on version rather than branching (§2).
 5. Verify green against the recipe PR head with the changed test, plus a regression sample.
 6. Say in the commit and PR **what evidence** proves the diagnosis, not just what changed.
@@ -88,9 +88,9 @@ def test_account_lifecycle_and_post_roundtrip(live_app):
    # Step 1: PDS describe via goat — recipe self-identifies as did:web:<domain>
    out = _in_container(domain, f"goat pds describe {PDS_HOST_LOCAL} 2>&1")
-    assert f"did:web:{domain}" in out, (
+    assert (
-        f"goat pds describe did not contain expected DID 'did:web:{domain}'. Output:\n{out[:500]!r}"
+        f"did:web:{domain}" in out
-    )
+    ), f"goat pds describe did not contain expected DID 'did:web:{domain}'. Output:\n{out[:500]!r}"
    # Step 2: Create account (UUID-suffixed handle = no run-to-run collision)
    out = _goat_admin(
@@ -133,9 +133,9 @@ def test_account_lifecycle_and_post_roundtrip(live_app):
        assert s == 200, f"createRecord HTTP {s}: {body!r}"
        record_uri = (body or {}).get("uri", "")
        # URI format: at://<did>/app.bsky.feed.post/<rkey>
-        assert record_uri.startswith(f"at://{new_did}/app.bsky.feed.post/"), (
+        assert record_uri.startswith(
-            f"unexpected record uri: {record_uri!r}"
+            f"at://{new_did}/app.bsky.feed.post/"
-        )
+        ), f"unexpected record uri: {record_uri!r}"
        rkey = record_uri.rsplit("/", 1)[-1]
        assert rkey, f"no rkey in uri: {record_uri!r}"
@@ -148,9 +148,9 @@ def test_account_lifecycle_and_post_roundtrip(live_app):
        )
        assert s == 200, f"getRecord HTTP {s}: {body!r}"
        record_value = (body or {}).get("value", {})
-        assert record_value.get("text") == marker, (
+        assert (
-            f"post text did not round-trip: created={marker!r}, fetched={record_value.get('text')!r}"
+            record_value.get("text") == marker
-        )
+        ), f"post text did not round-trip: created={marker!r}, fetched={record_value.get('text')!r}"
        assert record_value.get("$type") == "app.bsky.feed.post"
    finally:
        # Step 6: Best-effort cleanup. (The per-run domain teardown will discard the volume
@@ -26,6 +26,6 @@ def test_describe_server_returns_atproto_envelope(live_app):
    # At least one of these atproto-spec fields must be present
    expected_any = ("availableUserDomains", "inviteCodeRequired", "links", "did")
    present = [k for k in expected_any if k in body]
-    assert present, (
+    assert (
-        f"describe-server missing all of {expected_any}; got keys: {sorted(body.keys())[:20]}"
+        present
-    )
+    ), f"describe-server missing all of {expected_any}; got keys: {sorted(body.keys())[:20]}"
@@ -17,6 +17,6 @@ def test_pds_health_returns_version(live_app):
    url = f"https://{live_app}/xrpc/_health"
    status, body = harness_http.retry_http_get(url, expect_status=200, max_wait=60, interval=3)
    assert status == 200, f"GET {url} HTTP {status} (expected 200)"
-    assert isinstance(body, dict) and isinstance(body.get("version"), str) and body["version"], (
+    assert (
-        f"GET {url} response is not the expected health envelope: {body!r}"
+        isinstance(body, dict) and isinstance(body.get("version"), str) and body["version"]
-    )
+    ), f"GET {url} response is not the expected health envelope: {body!r}"
@@ -30,6 +30,6 @@ def test_get_session_requires_auth(live_app):
        f"body: {body!r}"
    )
    # The XRPC error envelope is JSON with an `error` field per the atproto spec.
-    assert isinstance(body, dict) and body.get("error"), (
+    assert isinstance(body, dict) and body.get(
-        f"expected XRPC JSON error envelope; got: {body!r}"
+        "error"
-    )
+    ), f"expected XRPC JSON error envelope; got: {body!r}"
@@ -11,6 +11,6 @@ import _p4  # noqa: E402
 def test_restore_returns_state(live_app):
-    assert _p4.account_exists(live_app), (
+    assert _p4.account_exists(
-        "restore did not bring back the seeded marker account (PDS data did not survive restore)"
+        live_app
-    )
+    ), "restore did not bring back the seeded marker account (PDS data did not survive restore)"
@@ -78,9 +78,9 @@ def test_7_new_run_blocks_until_reap_finishes(lock_dir, pool, monkeypatch):
    line = wait_marker(state["acquirer_out"], "ACQUIRED", timeout=15)
    assert line, "new run never acquired after the reap"
    acquired_ts = float(line.split()[1])
-    assert acquired_ts >= state["teardown_end"], (
+    assert (
-        f"new run acquired at {acquired_ts} BEFORE the reap finished at {state['teardown_end']}"
+        acquired_ts >= state["teardown_end"]
-    )
+    ), f"new run acquired at {acquired_ts} BEFORE the reap finished at {state['teardown_end']}"
    # The new run must hold a lock the next probe can SEE (fresh inode at the path).
    assert lock_state(DOMAIN) == "held"
@@ -160,17 +160,17 @@ def test_11_warm_canonical_names_never_probed(lock_dir, monkeypatch):
    monkeypatch.setattr(
        lifecycle,
        "_docker_names",
-        lambda kind, stack: (
+        lambda kind, stack: ["warm-keycloak_ci_commoninternet_net_app"]
-            ["warm-keycloak_ci_commoninternet_net_app"] if kind == "service" else []
+        if kind == "service"
-        ),
+        else [],
    )
    monkeypatch.setattr(lifecycle, "teardown_app", lambda d, verify=True: calls.append(d))
    lifecycle.janitor()
    assert calls == []
    lockdir = os.environ["CCCI_APP_LOCK_DIR"]
-    assert [f for f in os.listdir(lockdir) if f.startswith("cc-ci-app-")] == [], (
+    assert [
-        "janitor must not create lockfiles for non-run-app names"
+        f for f in os.listdir(lockdir) if f.startswith("cc-ci-app-")
-    )
+    ] == [], "janitor must not create lockfiles for non-run-app names"
 def test_12_degrades_safely_on_bad_lockfile_and_missing_dir(lock_dir, monkeypatch, capsys):
@@ -61,9 +61,9 @@ def test_3_lock_fd_not_inherited_by_children(lock_dir, pool):
    p.kill()
    p.wait(timeout=10)
    assert os.path.exists(f"/proc/{child_pid}"), "child should outlive the holder"
-    assert wait_lock_state(DOMAIN, "free") == "free", (
+    assert (
-        "lock must release on holder death even with a live child (PEP 446 non-inheritable fd)"
+        wait_lock_state(DOMAIN, "free") == "free"
-    )
+    ), "lock must release on holder death even with a live child (PEP 446 non-inheritable fd)"
 def test_4_second_acquire_blocks_until_first_exits(lock_dir, pool):
@@ -64,9 +64,9 @@ def test_20c_same_domain_runs_each_keep_their_own_count(tmp_path, lock_dir, pool
    pa.wait(timeout=15)
    line_b = wait_marker(out_b, "COUNT")
-    assert line_b is not None and line_b.strip() == "COUNT 1", (
+    assert (
-        line_b
+        line_b is not None and line_b.strip() == "COUNT 1"
-    )  # B's file survived A's remove
+    ), line_b  # B's file survived A's remove
    pb.wait(timeout=15)
@@ -150,9 +150,9 @@ def test_cryptpad_pad_content_survives_fresh_session(live_app):
            # --- session 1: create the pad + write the marker ---
            ctx1 = browser.new_context(ignore_https_errors=True)
            page, pad_url = _open_pad(ctx1, f"https://{live_app}/pad/")
-            assert "#/2/pad/edit/" in pad_url, (
+            assert (
-                f"CryptPad did not create a fragment-keyed pad URL; got {pad_url!r}"
+                "#/2/pad/edit/" in pad_url
-            )
+            ), f"CryptPad did not create a fragment-keyed pad URL; got {pad_url!r}"
            ck = _ckeditor_frame(page, reload_url=pad_url)
            assert ck is not None, "CKEditor content frame never attached (pad editor not ready)"
            _dismiss_store_modal(page)
@@ -161,9 +161,9 @@ def test_cryptpad_pad_content_survives_fresh_session(live_app):
            page.wait_for_timeout(1000)
            body.type(marker, delay=40)
            page.wait_for_timeout(12000)  # let CryptPad encrypt + sync the update to the server
-            assert marker in ck.locator("body").inner_text(), (
+            assert (
-                "marker not present in the editor after typing — type did not land"
+                marker in ck.locator("body").inner_text()
-            )
+            ), "marker not present in the editor after typing — type did not land"
            ctx1.close()
            # --- session 2: FRESH context (no shared storage/localStorage) reads the pad back by URL.
@@ -51,9 +51,9 @@ def test_cryptpad_spa_renders_with_no_console_errors(live_app):
            title = (page.title() or "").lower()
            body = page.content()
            blower = body.lower()
-            assert "cryptpad" in title or "cryptpad" in blower, (
+            assert (
-                f"CryptPad SPA does not carry brand. title={title!r}, body excerpt: {body[:200]!r}"
+                "cryptpad" in title or "cryptpad" in blower
-            )
+            ), f"CryptPad SPA does not carry brand. title={title!r}, body excerpt: {body[:200]!r}"
            # Canonical CryptPad asset references in the rendered DOM
            canonical = ("/customize/", "/components/", "main.js", "/api/broadcast")
@@ -14,6 +14,6 @@ MARKER = "/cryptpad/data/ci-marker.txt"
 def test_backup_captures_state(live_app):
-    assert lifecycle.exec_in_app(live_app, ["cat", MARKER]).strip() == "original", (
+    assert (
-        "the seeded state was not present at backup time"
+        lifecycle.exec_in_app(live_app, ["cat", MARKER]).strip() == "original"
-    )
+    ), "the seeded state was not present at backup time"
@@ -14,6 +14,6 @@ MARKER = "/cryptpad/data/ci-marker.txt"
 def test_restore_returns_state(live_app):
-    assert lifecycle.exec_in_app(live_app, ["cat", MARKER]).strip() == "original", (
+    assert (
-        "restore did not return the pre-mutation state"
+        lifecycle.exec_in_app(live_app, ["cat", MARKER]).strip() == "original"
-    )
+    ), "restore did not return the pre-mutation state"
@@ -14,6 +14,6 @@ MARKER = "/cryptpad/data/ci-marker.txt"
 def test_upgrade_preserves_data(live_app):
-    assert lifecycle.exec_in_app(live_app, ["cat", MARKER]).strip() == "upgrade-survives", (
+    assert (
-        "data did not survive the upgrade"
+        lifecycle.exec_in_app(live_app, ["cat", MARKER]).strip() == "upgrade-survives"
-    )
+    ), "data did not survive the upgrade"
@@ -79,9 +79,9 @@ def test_static_file_roundtrip_and_404(live_app):
        # A random non-existent path must 404 — proves real static-file semantics, distinguishing a
        # working server from a 200-everything stub or a mis-routed Traefik fallback.
        miss_status, _ = _get(f"https://{live_app}/ccci-missing-{uuid.uuid4().hex}.txt")
-        assert miss_status == 404, (
+        assert (
-            f"missing path returned {miss_status} (expected 404 — generic 200-returner / mis-route?)"
+            miss_status == 404
-        )
+        ), f"missing path returned {miss_status} (expected 404 — generic 200-returner / mis-route?)"
    finally:
        with contextlib.suppress(OSError):
            os.remove(path)
@@ -53,9 +53,9 @@ def test_content_type_html_and_txt(live_app):
    ct_txt = h_txt.get("content-type", "")
    # nginx default: "text/html" for .html and "text/plain" for .txt (may include "; charset=utf-8")
-    assert ct_html.startswith("text/html"), (
+    assert ct_html.startswith(
-        f"{html_name} Content-Type={ct_html!r}, expected text/html (nginx MIME config broken?)"
+        "text/html"
-    )
+    ), f"{html_name} Content-Type={ct_html!r}, expected text/html (nginx MIME config broken?)"
-    assert ct_txt.startswith("text/plain"), (
+    assert ct_txt.startswith(
-        f"{txt_name} Content-Type={ct_txt!r}, expected text/plain (nginx MIME config broken?)"
+        "text/plain"
-    )
+    ), f"{txt_name} Content-Type={ct_txt!r}, expected text/plain (nginx MIME config broken?)"
@@ -16,6 +16,6 @@ MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
 def test_backup_captures_state(live_app):
-    assert lifecycle.exec_in_app(live_app, ["cat", MARKER_PATH]).strip() == "original", (
+    assert (
-        "the seeded state was not present at backup time"
+        lifecycle.exec_in_app(live_app, ["cat", MARKER_PATH]).strip() == "original"
-    )
+    ), "the seeded state was not present at backup time"
@@ -17,6 +17,6 @@ MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
 def test_restore_returns_state(live_app):
    restored = lifecycle.exec_in_app(live_app, ["cat", MARKER_PATH]).strip()
-    assert restored == "original", (
+    assert (
-        f"restore did not return the pre-mutation (backed-up) state: got {restored!r}"
+        restored == "original"
-    )
+    ), f"restore did not return the pre-mutation (backed-up) state: got {restored!r}"
@@ -16,6 +16,6 @@ MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
 def test_upgrade_preserves_data(live_app):
    # the marker seeded by ops.pre_upgrade (before the harness upgraded) is still served
-    assert lifecycle.http_fetch(live_app, "/ci-marker.txt")[1].strip() == "upgrade-survives", (
+    assert (
-        "data did not survive the upgrade"
+        lifecycle.http_fetch(live_app, "/ci-marker.txt")[1].strip() == "upgrade-survives"
-    )
+    ), "data did not survive the upgrade"
@@ -81,9 +81,9 @@ def mint_admin(domain: str) -> tuple[str, str]:
            key = line.split("=", 1)[1].strip()
        elif line.startswith("CCCI_API_USER="):
            user = line.split("=", 1)[1].strip()
-    assert key and user, (
+    assert (
-        f"could not bootstrap discourse admin/API key; rails output tail:\n{out[-1000:]}"
+        key and user
-    )
+    ), f"could not bootstrap discourse admin/API key; rails output tail:\n{out[-1000:]}"
    return key, user
@@ -48,23 +48,23 @@ def test_create_topic_roundtrip(live_app):
        headers=hdrs,
        timeout=60,
    )
-    assert status in (200, 201) and isinstance(body, dict), (
+    assert status in (200, 201) and isinstance(
-        f"create topic failed: HTTP {status}, body={body!r}"
+        body, dict
-    )
+    ), f"create topic failed: HTTP {status}, body={body!r}"
    topic_id = body.get("topic_id")
    assert topic_id, f"create topic returned no topic_id: {body!r}"
    # 4) Read the topic back and assert title + first-post body round-trip.
    status, got = harness_http.http_get(f"{base}/t/{topic_id}.json", headers=hdrs, timeout=30)
-    assert status == 200 and isinstance(got, dict), (
+    assert status == 200 and isinstance(
-        f"read topic failed: HTTP {status}, body={got!r}"
+        got, dict
-    )
+    ), f"read topic failed: HTTP {status}, body={got!r}"
-    assert got.get("title") == title, (
+    assert (
-        f"topic title did not round-trip: sent {title!r}, got {got.get('title')!r}"
+        got.get("title") == title
-    )
+    ), f"topic title did not round-trip: sent {title!r}, got {got.get('title')!r}"
    posts = (got.get("post_stream") or {}).get("posts") or []
    assert posts, f"topic has no posts on read-back: {got!r}"
    first_cooked = posts[0].get("cooked", "")
-    assert marker in first_cooked, (
+    assert (
-        f"topic body did not round-trip: marker {marker!r} not in first post {first_cooked!r}"
+        marker in first_cooked
-    )
+    ), f"topic body did not round-trip: marker {marker!r} not in first post {first_cooked!r}"
@@ -20,12 +20,12 @@ def test_site_json_has_discourse_config(live_app):
    status, body = harness_http.retry_http_get(
        f"https://{live_app}/site.json", expect_status=200, max_wait=120, interval=5
    )
-    assert status == 200 and isinstance(body, dict), (
+    assert status == 200 and isinstance(
-        f"GET /site.json failed: HTTP {status}, body type={type(body).__name__}"
+        body, dict
-    )
+    ), f"GET /site.json failed: HTTP {status}, body type={type(body).__name__}"
    # /site.json carries Discourse-specific structure — `categories` (a list) and `groups` are always
    # present in a booted Discourse. A non-Discourse 200 (placeholder page) would not parse to this.
    assert "categories" in body, f"/site.json missing 'categories' key: keys={list(body)[:20]}"
-    assert isinstance(body["categories"], list), (
+    assert isinstance(
-        f"/site.json 'categories' not a list: {type(body['categories']).__name__}"
+        body["categories"], list
-    )
+    ), f"/site.json 'categories' not a list: {type(body['categories']).__name__}"
@@ -14,7 +14,9 @@ from harness import lifecycle  # noqa: E402
 def _psql(domain, sql):
-    cmd = f'PGPASSWORD=$(cat /run/secrets/db_password) psql -U discourse -d discourse -tAc "{sql}"'
+    cmd = (
        "PGPASSWORD=$(cat /run/secrets/db_password) " f'psql -U discourse -d discourse -tAc "{sql}"'
    )
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
@@ -13,11 +13,13 @@ from harness import lifecycle  # noqa: E402
 def _psql(domain, sql):
-    cmd = f'PGPASSWORD=$(cat /run/secrets/db_password) psql -U discourse -d discourse -tAc "{sql}"'
+    cmd = (
        "PGPASSWORD=$(cat /run/secrets/db_password) " f'psql -U discourse -d discourse -tAc "{sql}"'
    )
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded discourse postgres state was not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded discourse postgres state was not present at backup time"
@@ -13,11 +13,13 @@ from harness import lifecycle  # noqa: E402
 def _psql(domain, sql):
-    cmd = f'PGPASSWORD=$(cat /run/secrets/db_password) psql -U discourse -d discourse -tAc "{sql}"'
+    cmd = (
        "PGPASSWORD=$(cat /run/secrets/db_password) " f'psql -U discourse -d discourse -tAc "{sql}"'
    )
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation discourse postgres state (data-integrity failure)"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation discourse postgres state (data-integrity failure)"
@@ -62,6 +62,6 @@ def test_ghost_admin_route_is_wired(live_app):
    assert status in (200, 302), f"unexpected status: {status}"
    if status == 200:
        # The admin SPA references /ghost-assets/ or contains "ghost" in title/body
-        assert "ghost" in body.lower(), (
+        assert (
-            f"GET {url} 200 but body has no Ghost markers: {body[:200]!r}"
+            "ghost" in body.lower()
-        )
+        ), f"GET {url} 200 but body has no Ghost markers: {body[:200]!r}"
@@ -35,10 +35,10 @@ def test_content_api_settings_endpoint(live_app):
    assert body is not None, f"GET {url} returned non-JSON body"
    # On success: {"settings": {...}}. On error: {"errors": [...]}. Either shape is valid.
    if status == 200:
-        assert isinstance(body, dict) and "settings" in body, (
+        assert (
-            f"200 response missing 'settings' envelope: {body!r}"
+            isinstance(body, dict) and "settings" in body
-        )
+        ), f"200 response missing 'settings' envelope: {body!r}"
    else:
-        assert isinstance(body, dict) and ("errors" in body or "message" in body or body), (
+        assert isinstance(body, dict) and (
-            f"error response not a proper Ghost error envelope: {body!r}"
+            "errors" in body or "message" in body or body
-        )
+        ), f"error response not a proper Ghost error envelope: {body!r}"
@@ -43,17 +43,17 @@ def test_create_post_roundtrip(live_app):
    title = f"ccci-marker-{uniq}"
    marker = f"ccci-body-marker-{uniq}-roundtrip"
    created = admin.create_post(title, f"<p>{marker}</p>")
-    assert created.get("title") == title, (
+    assert (
-        f"created post title mismatch: sent {title!r}, got {created.get('title')!r}"
+        created.get("title") == title
-    )
+    ), f"created post title mismatch: sent {title!r}, got {created.get('title')!r}"
    # 4) Read it back by id and assert the post survived the round-trip (title always returned;
    #    html returned because we requested ?formats=html).
    got = admin.get_post(created["id"])
-    assert got.get("title") == title, (
+    assert (
-        f"post title did not round-trip: sent {title!r}, got {got.get('title')!r}"
+        got.get("title") == title
-    )
+    ), f"post title did not round-trip: sent {title!r}, got {got.get('title')!r}"
    html = got.get("html") or ""
-    assert marker in html, (
+    assert (
-        f"post body did not round-trip: marker {marker!r} not in read-back html {html!r}"
+        marker in html
-    )
+    ), f"post body did not round-trip: marker {marker!r} not in read-back html {html!r}"
@@ -22,7 +22,7 @@ from harness import lifecycle  # noqa: E402
 def _mysql(domain, sql):
-    cmd = f'MYSQL_PWD="$(cat /run/secrets/db_password)" mysql -u root -N -s ghost -e "{sql}"'
+    cmd = 'MYSQL_PWD="$(cat /run/secrets/db_password)" ' f'mysql -u root -N -s ghost -e "{sql}"'
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
@@ -15,11 +15,11 @@ from harness import lifecycle  # noqa: E402
 def _mysql(domain, sql):
-    cmd = f'MYSQL_PWD="$(cat /run/secrets/db_password)" mysql -u root -N -s ghost -e "{sql}"'
+    cmd = 'MYSQL_PWD="$(cat /run/secrets/db_password)" ' f'mysql -u root -N -s ghost -e "{sql}"'
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 def test_backup_captures_state(live_app):
-    assert _mysql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded ghost MySQL marker was not present at backup time"
+        _mysql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded ghost MySQL marker was not present at backup time"
@@ -22,7 +22,7 @@ from harness import lifecycle  # noqa: E402
 def _mysql(domain, sql):
-    cmd = f'MYSQL_PWD="$(cat /run/secrets/db_password)" mysql -u root -N -s ghost -e "{sql}"'
+    cmd = 'MYSQL_PWD="$(cat /run/secrets/db_password)" ' f'mysql -u root -N -s ghost -e "{sql}"'
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
@@ -14,11 +14,11 @@ from harness import lifecycle  # noqa: E402
 def _mysql(domain, sql):
-    cmd = f'MYSQL_PWD="$(cat /run/secrets/db_password)" mysql -u root -N -s ghost -e "{sql}"'
+    cmd = 'MYSQL_PWD="$(cat /run/secrets/db_password)" ' f'mysql -u root -N -s ghost -e "{sql}"'
    return lifecycle.exec_in_app(domain, ["sh", "-c", cmd], service="db").strip()
 def test_upgrade_preserves_state(live_app):
-    assert _mysql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "the seeded ghost MySQL marker did not survive the upgrade redeploy (data loss on upgrade)"
+        _mysql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "the seeded ghost MySQL marker did not survive the upgrade redeploy (data loss on upgrade)"
@@ -145,9 +145,9 @@ def test_lfs_roundtrip(live_app):
            text=True,
            env={**os.environ, **git_env},
        )
-        assert "testblob.bin" in lfs_ls.stdout, (
+        assert (
-            f"testblob.bin not in git-lfs ls-files: {lfs_ls.stdout}"
+            "testblob.bin" in lfs_ls.stdout
-        )
+        ), f"testblob.bin not in git-lfs ls-files: {lfs_ls.stdout}"
        # 6. Download in a FRESH clone (proves the LFS server stores and serves the object)
        fresh_dir = tempfile.mkdtemp(prefix="ccci-gitea-lfs-dl-")
@@ -158,9 +158,9 @@ def test_lfs_roundtrip(live_app):
            with open(fetched_path, "rb") as f:
                fetched = f.read()
            fetched_sha256 = hashlib.sha256(fetched).hexdigest()
-            assert fetched_sha256 == expected_sha256, (
+            assert (
-                f"LFS round-trip OID mismatch: expected {expected_oid}, got sha256:{fetched_sha256}"
+                fetched_sha256 == expected_sha256
-            )
+            ), f"LFS round-trip OID mismatch: expected {expected_oid}, got sha256:{fetched_sha256}"
        finally:
            shutil.rmtree(fresh_dir, ignore_errors=True)
@@ -171,9 +171,9 @@ def test_lfs_roundtrip(live_app):
            ["sh", "-c", "grep -E '^LFS_JWT_SECRET' /etc/gitea/app.ini || echo NOT_FOUND"],
            timeout=30,
        ).strip()
-        assert current_jwt and "NOT_FOUND" not in current_jwt, (
+        assert (
-            "Could not read LFS_JWT_SECRET from /etc/gitea/app.ini before restart"
+            current_jwt and "NOT_FOUND" not in current_jwt
-        )
+        ), "Could not read LFS_JWT_SECRET from /etc/gitea/app.ini before restart"
        # Restart the gitea container
        lifecycle.exec_in_app(live_app, ["true"], timeout=5)  # no-op to confirm exec works
@@ -213,9 +213,9 @@ def test_lfs_roundtrip(live_app):
            assert os.path.exists(pr_blob), "testblob.bin not fetched in post-restart clone"
            with open(pr_blob, "rb") as f:
                pr_data = f.read()
-            assert hashlib.sha256(pr_data).hexdigest() == expected_sha256, (
+            assert (
-                "LFS object corrupted after restart — JWT secret may have changed"
+                hashlib.sha256(pr_data).hexdigest() == expected_sha256
-            )
+            ), "LFS object corrupted after restart — JWT secret may have changed"
        finally:
            shutil.rmtree(post_restart_dir, ignore_errors=True)
@@ -220,7 +220,7 @@ def pre_restore(ctx):
    generic.assert_serving(ctx.domain, ctx.meta)
    ok = _delete_marker_repo(ctx.domain, user, password)
    assert ok, f"pre_restore: could not delete {_MARKER_REPO} repo on {ctx.domain}"
-    assert not marker_repo_exists(ctx.domain, user, password), (
+    assert not marker_repo_exists(
-        f"pre_restore: {_MARKER_REPO} still present after delete — divergence did not take"
+        ctx.domain, user, password
-    )
+    ), f"pre_restore: {_MARKER_REPO} still present after delete — divergence did not take"
    print(f"  gitea ops: {_MARKER_REPO!r} deleted (diverged from backup state)", flush=True)
@@ -22,6 +22,6 @@ def test_backup_captures_marker_repo(live_app, meta):
    # backupbot cycles the gitea container during backup — wait for it to be back up.
    generic.assert_serving(live_app, meta)
    user, password = admin_creds(live_app)
-    assert marker_repo_exists(live_app, user, password), (
+    assert marker_repo_exists(
-        f"{live_app}: ci-marker repo is not present at backup time (backup would capture empty state)"
+        live_app, user, password
-    )
+    ), f"{live_app}: ci-marker repo is not present at backup time (backup would capture empty state)"
@@ -65,8 +65,8 @@ def test_install_gitea(live_app, meta):
            )
            page.wait_for_selector("input#user_name", timeout=20_000)
            content = page.content()
-            assert "gitea" in content.lower() or "sign in" in content.lower(), (
+            assert (
-                "Sign-in page did not render expected gitea content"
+                "gitea" in content.lower() or "sign in" in content.lower()
-            )
+            ), "Sign-in page did not render expected gitea content"
        finally:
            browser.close()
@@ -20,6 +20,6 @@ def test_upgrade_preserves_marker_repo(live_app, meta):
    """The ci-marker repo survived the upgrade to the PR head (data continuity)."""
    generic.assert_serving(live_app, meta)
    user, password = admin_creds(live_app)
-    assert marker_repo_exists(live_app, user, password), (
+    assert marker_repo_exists(
-        f"{live_app}: ci-marker repo did not survive the upgrade (sqlite3 data lost)"
+        live_app, user, password
-    )
+    ), f"{live_app}: ci-marker repo did not survive the upgrade (sqlite3 data lost)"
@@ -111,13 +111,13 @@ def test_immich_processes_uploaded_asset_metadata_and_statistics(live_app):
        if exif and exif.get("exifImageWidth"):
            break
        time.sleep(5)
-    assert exif and exif.get("exifImageWidth") == 1 and exif.get("exifImageHeight") == 1, (
+    assert (
-        f"immich metadata-extraction did not populate the 1x1 PNG dimensions in exifInfo: {exif!r}"
+        exif and exif.get("exifImageWidth") == 1 and exif.get("exifImageHeight") == 1
-    )
+    ), f"immich metadata-extraction did not populate the 1x1 PNG dimensions in exifInfo: {exif!r}"
    # the asset is catalogued into the owner's library statistics (list-back in aggregate)
    sst, stats = harness_http.http_request("GET", f"{base}/api/assets/statistics", headers=auth)
    assert sst == 200 and isinstance(stats, dict), f"statistics HTTP {sst}: {stats!r}"
-    assert stats.get("images", 0) >= 1 and stats.get("total", 0) >= 1, (
+    assert (
-        f"uploaded asset not reflected in library statistics: {stats!r}"
+        stats.get("images", 0) >= 1 and stats.get("total", 0) >= 1
-    )
+    ), f"uploaded asset not reflected in library statistics: {stats!r}"
@@ -121,6 +121,6 @@ def test_immich_upload_asset_readback_and_thumbnail(live_app):
        if thumb == 200:
            break
        time.sleep(5)
-    assert thumb == 200, (
+    assert (
-        f"immich did not generate a thumbnail/derivative for the uploaded asset (last HTTP {thumb})"
+        thumb == 200
-    )
+    ), f"immich did not generate a thumbnail/derivative for the uploaded asset (last HTTP {thumb})"
@@ -14,6 +14,6 @@ def _psql(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "seeded postgres state not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "seeded postgres state not present at backup time"
@@ -14,6 +14,6 @@ def _psql(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation postgres state"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation postgres state"
@@ -14,6 +14,6 @@ def _psql(domain, sql):
 def test_upgrade_preserves_data(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "postgres data did not survive the upgrade"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "postgres data did not survive the upgrade"
@@ -144,25 +144,25 @@ def test_create_confidential_client_and_obtain_token(live_app):
        # Use the client to obtain its own token (client_credentials grant)
        tok_status, tok_resp = _client_credentials_token(live_app, client_id, client_secret)
-        assert tok_status == 200, (
+        assert (
-            f"client_credentials token returned HTTP {tok_status}: {tok_resp!r}"
+            tok_status == 200
-        )
+        ), f"client_credentials token returned HTTP {tok_status}: {tok_resp!r}"
        access_token = tok_resp.get("access_token") if isinstance(tok_resp, dict) else None
-        assert isinstance(access_token, str) and access_token.count(".") == 2, (
+        assert (
-            f"client_credentials access_token not a JWT: {access_token!r}"
+            isinstance(access_token, str) and access_token.count(".") == 2
-        )
+        ), f"client_credentials access_token not a JWT: {access_token!r}"
        # Decode the JWT payload; assert azp matches the new client
        payload = json.loads(_b64url_decode(access_token.split(".")[1]))
-        assert payload.get("azp") == client_id, (
+        assert (
-            f"client_credentials JWT azp={payload.get('azp')!r} != client_id={client_id!r}"
+            payload.get("azp") == client_id
-        )
+        ), f"client_credentials JWT azp={payload.get('azp')!r} != client_id={client_id!r}"
        # Service-account token does NOT carry a session-scoped user (azp + clientId differ from
        # admin-cli token). The presence of azp + iss == per-run-domain proves the issuance flow.
        expected_iss = f"https://{live_app}/realms/master"
-        assert payload.get("iss") == expected_iss, (
+        assert (
-            f"JWT iss={payload.get('iss')!r} != {expected_iss!r}"
+            payload.get("iss") == expected_iss
-        )
+        ), f"JWT iss={payload.get('iss')!r} != {expected_iss!r}"
    finally:
        # Idempotent cleanup
        if cleanup_id:
@@ -43,17 +43,17 @@ def test_password_grant_issues_valid_jwt(live_app):
    token = kc_admin.admin_token(live_app, password)
    # Shape: a JWT is exactly 3 base64url segments
-    assert isinstance(token, str) and token.count(".") == 2, (
+    assert (
-        f"access_token does not look like a JWT (no 3 segments): len={len(token) if token else 0}"
+        isinstance(token, str) and token.count(".") == 2
-    )
+    ), f"access_token does not look like a JWT (no 3 segments): len={len(token) if token else 0}"
    payload = _decode_jwt_payload(token)
    # iss = the issuer URL, must be the per-run domain's /realms/master endpoint
    expected_iss = f"https://{live_app}/realms/master"
-    assert payload.get("iss") == expected_iss, (
+    assert (
-        f"JWT iss claim {payload.get('iss')!r} != {expected_iss!r}"
+        payload.get("iss") == expected_iss
-    )
+    ), f"JWT iss claim {payload.get('iss')!r} != {expected_iss!r}"
    # azp = authorized party (which client requested this token)
    assert payload.get("azp") == "admin-cli", f"JWT azp claim {payload.get('azp')!r} != 'admin-cli'"
@@ -68,6 +68,6 @@ def test_password_grant_issues_valid_jwt(live_app):
    # iat (issued at) is also a standard claim
    iat = payload.get("iat")
-    assert isinstance(iat, int) and iat <= time.time() + 60, (
+    assert (
-        f"JWT iat {iat!r} not a reasonable past timestamp"
+        isinstance(iat, int) and iat <= time.time() + 60
-    )
+    ), f"JWT iat {iat!r} not a reasonable past timestamp"
@@ -42,9 +42,9 @@ def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
    # Sanity-check the creds shape — orchestrator-written
    assert kc["domain"]
    # WC1: realm is per-run namespaced "<parent>-<6hex>" so concurrent dependents never collide.
-    assert re.fullmatch(r"lasuite-docs-[0-9a-f]{6}", kc["realm"]), (
+    assert re.fullmatch(
-        f"realm {kc['realm']!r} not the per-run namespaced form lasuite-docs-<6hex>"
+        r"lasuite-docs-[0-9a-f]{6}", kc["realm"]
-    )
+    ), f"realm {kc['realm']!r} not the per-run namespaced form lasuite-docs-<6hex>"
    assert kc["client_id"] == "lasuite-docs"
    assert isinstance(kc["client_secret"], str) and len(kc["client_secret"]) >= 16
    assert isinstance(kc["password"], str) and len(kc["password"]) >= 16
@@ -77,11 +77,11 @@ def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
    assert isinstance(token, str) and token.count(".") == 2, f"access_token is not a JWT: {token!r}"
    payload = json.loads(_b64url_decode(token.split(".")[1]))
    assert payload.get("iss") == expected_iss, f"JWT iss={payload.get('iss')!r} != {expected_iss!r}"
-    assert payload.get("azp") == kc["client_id"], (
+    assert (
-        f"JWT azp={payload.get('azp')!r} != {kc['client_id']!r}"
+        payload.get("azp") == kc["client_id"]
-    )
+    ), f"JWT azp={payload.get('azp')!r} != {kc['client_id']!r}"
    assert payload.get("typ") == "Bearer", f"JWT typ={payload.get('typ')!r} != 'Bearer'"
    exp = payload.get("exp")
-    assert isinstance(exp, int) and exp > time.time(), (
+    assert (
-        f"JWT exp={exp!r} not a future timestamp (now={time.time():.0f})"
+        isinstance(exp, int) and exp > time.time()
-    )
+    ), f"JWT exp={exp!r} not a future timestamp (now={time.time():.0f})"
@@ -18,6 +18,6 @@ def _psql(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded postgres state was not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded postgres state was not present at backup time"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation postgres state"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation postgres state"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_upgrade_preserves_data(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "postgres data did not survive the upgrade"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "postgres data did not survive the upgrade"
@@ -56,6 +56,6 @@ def test_minio_bucket_present_and_object_roundtrip(live_app):
    # The object was listed (its key appears) and its content round-tripped intact.
    assert f"{marker}.txt" in out, f"uploaded object not listed in bucket: {out!r}"
-    assert f"READBACK:{marker}" in out, (
+    assert (
-        f"object content did not round-trip through MinIO; got: {out!r}"
+        f"READBACK:{marker}" in out
-    )
+    ), f"object content did not round-trip through MinIO; got: {out!r}"
@@ -46,9 +46,9 @@ def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
    # Creds shape. WC1: realm is per-run namespaced "<parent>-<6hex>"; client_id stays the parent.
    assert kc["domain"]
-    assert re.fullmatch(r"lasuite-drive-[0-9a-f]{6}", kc["realm"]), (
+    assert re.fullmatch(
-        f"realm {kc['realm']!r} not the per-run namespaced form lasuite-drive-<6hex>"
+        r"lasuite-drive-[0-9a-f]{6}", kc["realm"]
-    )
+    ), f"realm {kc['realm']!r} not the per-run namespaced form lasuite-drive-<6hex>"
    assert kc["client_id"] == "lasuite-drive"
    assert isinstance(kc["client_secret"], str) and len(kc["client_secret"]) >= 16
    assert isinstance(kc["password"], str) and len(kc["password"]) >= 16
@@ -80,11 +80,11 @@ def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
    assert isinstance(token, str) and token.count(".") == 2, f"access_token is not a JWT: {token!r}"
    payload = json.loads(_b64url_decode(token.split(".")[1]))
    assert payload.get("iss") == expected_iss, f"JWT iss={payload.get('iss')!r} != {expected_iss!r}"
-    assert payload.get("azp") == kc["client_id"], (
+    assert (
-        f"JWT azp={payload.get('azp')!r} != {kc['client_id']!r}"
+        payload.get("azp") == kc["client_id"]
-    )
+    ), f"JWT azp={payload.get('azp')!r} != {kc['client_id']!r}"
    assert payload.get("typ") == "Bearer", f"JWT typ={payload.get('typ')!r} != 'Bearer'"
    exp = payload.get("exp")
-    assert isinstance(exp, int) and exp > time.time(), (
+    assert (
-        f"JWT exp={exp!r} not a future timestamp (now={time.time():.0f})"
+        isinstance(exp, int) and exp > time.time()
-    )
+    ), f"JWT exp={exp!r} not a future timestamp (now={time.time():.0f})"
@@ -18,6 +18,6 @@ def _psql(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded postgres state was not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded postgres state was not present at backup time"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation postgres state"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation postgres state"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_upgrade_preserves_data(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "postgres data did not survive the upgrade"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "postgres data did not survive the upgrade"
@@ -46,9 +46,9 @@ def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
    # Creds shape. WC1: realm is per-run namespaced "<parent>-<6hex>"; client_id stays the parent.
    assert kc["domain"]
-    assert re.fullmatch(r"lasuite-meet-[0-9a-f]{6}", kc["realm"]), (
+    assert re.fullmatch(
-        f"realm {kc['realm']!r} not the per-run namespaced form lasuite-meet-<6hex>"
+        r"lasuite-meet-[0-9a-f]{6}", kc["realm"]
-    )
+    ), f"realm {kc['realm']!r} not the per-run namespaced form lasuite-meet-<6hex>"
    assert kc["client_id"] == "lasuite-meet"
    assert isinstance(kc["client_secret"], str) and len(kc["client_secret"]) >= 16
    assert isinstance(kc["password"], str) and len(kc["password"]) >= 16
@@ -80,11 +80,11 @@ def test_oidc_password_grant_against_dep_keycloak(live_app, deps):
    assert isinstance(token, str) and token.count(".") == 2, f"access_token is not a JWT: {token!r}"
    payload = json.loads(_b64url_decode(token.split(".")[1]))
    assert payload.get("iss") == expected_iss, f"JWT iss={payload.get('iss')!r} != {expected_iss!r}"
-    assert payload.get("azp") == kc["client_id"], (
+    assert (
-        f"JWT azp={payload.get('azp')!r} != {kc['client_id']!r}"
+        payload.get("azp") == kc["client_id"]
-    )
+    ), f"JWT azp={payload.get('azp')!r} != {kc['client_id']!r}"
    assert payload.get("typ") == "Bearer", f"JWT typ={payload.get('typ')!r} != 'Bearer'"
    exp = payload.get("exp")
-    assert isinstance(exp, int) and exp > time.time(), (
+    assert (
-        f"JWT exp={exp!r} not a future timestamp (now={time.time():.0f})"
+        isinstance(exp, int) and exp > time.time()
-    )
+    ), f"JWT exp={exp!r} not a future timestamp (now={time.time():.0f})"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded postgres state was not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded postgres state was not present at backup time"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation postgres state"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation postgres state"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_upgrade_preserves_data(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "postgres data did not survive the upgrade"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "postgres data did not survive the upgrade"
@@ -43,7 +43,7 @@ def test_send_and_receive_mail(live_app):
    deadline = time.time() + 150
    while time.time() < deadline:
        for box in ("INBOX", "Junk"):
-            query = f"doveadm search -u '{email_addr}' mailbox {box} header subject '{marker}'"
+            query = f"doveadm search -u '{email_addr}' mailbox {box} " f"header subject '{marker}'"
            out = lifecycle.exec_in_app(live_app, ["sh", "-c", query], service="imap")
            if out.strip():  # a non-empty result = "<mailbox-guid> <uid>" → message stored
                return
@@ -24,6 +24,6 @@ def test_create_mailbox_and_read_back(live_app):
    cfg = _mailu.config_export(live_app)
    emails = _mailu.user_emails(cfg)
-    assert email in emails, (
+    assert (
-        f"created mailbox {email} not present in mailu config-export users {emails}"
+        email in emails
-    )
+    ), f"created mailbox {email} not present in mailu config-export users {emails}"
@@ -34,12 +34,12 @@ def test_federation_version_endpoint(live_app):
    assert status == 200, f"GET {url} HTTP {status} (expected 200)"
    assert isinstance(body, dict), f"federation version returned non-dict: {type(body).__name__}"
    server = body.get("server")
-    assert isinstance(server, dict), (
+    assert isinstance(
-        f"federation version response missing 'server' envelope: {body!r}"
+        server, dict
-    )
+    ), f"federation version response missing 'server' envelope: {body!r}"
    name = server.get("name")
    assert name == "Synapse", f"server.name={name!r}, expected 'Synapse'"
    version = server.get("version")
-    assert isinstance(version, str) and len(version) > 0, (
+    assert (
-        f"server.version is not a non-empty string: {version!r}"
+        isinstance(version, str) and len(version) > 0
-    )
+    ), f"server.version is not a non-empty string: {version!r}"
@@ -23,6 +23,6 @@ def test_synapse_client_versions_returns_json(live_app):
    url = f"https://{live_app}/_matrix/client/versions"
    status, body = harness_http.retry_http_get(url, expect_status=200, max_wait=60, interval=3)
    assert status == 200, f"GET {url} HTTP {status} (expected 200)"
-    assert isinstance(body, dict) and isinstance(body.get("versions"), list) and body["versions"], (
+    assert (
-        f"GET {url} did not return Matrix client-versions document: {body!r}"
+        isinstance(body, dict) and isinstance(body.get("versions"), list) and body["versions"]
-    )
+    ), f"GET {url} did not return Matrix client-versions document: {body!r}"
@@ -127,7 +127,8 @@ def _admin_register(domain: str, secret: str, username: str, password: str, admi
        if r["status"] == 200:
            if attempt > 1:
                print(
-                    f"  [register] {username}: succeeded on attempt {attempt} (synapse recovered)",
+                    f"  [register] {username}: succeeded on attempt {attempt} "
                    f"(synapse recovered)",
                    flush=True,
                )
            return r["body"] or {}
@@ -176,9 +177,9 @@ def test_register_two_users_send_receive_message(live_app):
    create + invite + join a room; send and read a message."""
    domain = live_app
    secret = _registration_secret(domain)
-    assert secret and len(secret) >= 16, (
+    assert (
-        f"registration shared secret missing/short: len={len(secret) if secret else 0}"
+        secret and len(secret) >= 16
-    )
+    ), f"registration shared secret missing/short: len={len(secret) if secret else 0}"
    suffix = uuid.uuid4().hex[:8]
    user_a = f"alice{suffix}"
@@ -18,6 +18,6 @@ def _psql(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded postgres state was not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded postgres state was not present at backup time"
@@ -26,6 +26,6 @@ def test_serving_and_client_api(live_app, meta):
    # The client-API version document is real synapse JSON (proves the app, not just a proxy 200).
    body = lifecycle.http_body(live_app, "/_matrix/client/versions")
    doc = json.loads(body)
-    assert isinstance(doc.get("versions"), list) and doc["versions"], (
+    assert (
-        "no matrix client versions advertised"
+        isinstance(doc.get("versions"), list) and doc["versions"]
-    )
+    ), "no matrix client versions advertised"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation postgres state"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation postgres state"
@@ -17,6 +17,6 @@ def _psql(domain, sql):
 def test_upgrade_preserves_data(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "postgres data did not survive the upgrade"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "postgres data did not survive the upgrade"
@@ -41,9 +41,9 @@ def test_create_message_roundtrip(live_app):
        headers=auth,
        timeout=30,
    )
-    assert status in (200, 201) and isinstance(team, dict) and team.get("id"), (
+    assert (
-        f"team creation failed: HTTP {status}, body={team!r}"
+        status in (200, 201) and isinstance(team, dict) and team.get("id")
-    )
+    ), f"team creation failed: HTTP {status}, body={team!r}"
    status, chan = harness_http.http_post(
        f"{base}/channels",
        data={
@@ -55,9 +55,9 @@ def test_create_message_roundtrip(live_app):
        headers=auth,
        timeout=30,
    )
-    assert status in (200, 201) and isinstance(chan, dict) and chan.get("id"), (
+    assert (
-        f"channel creation failed: HTTP {status}, body={chan!r}"
+        status in (200, 201) and isinstance(chan, dict) and chan.get("id")
-    )
+    ), f"channel creation failed: HTTP {status}, body={chan!r}"
    # 4) POST a unique marker message.
    marker = f"ccci-marker-{uniq}-roundtrip"
@@ -67,13 +67,13 @@ def test_create_message_roundtrip(live_app):
        headers=auth,
        timeout=30,
    )
-    assert status in (200, 201) and isinstance(post, dict) and post.get("id"), (
+    assert (
-        f"post creation failed: HTTP {status}, body={post!r}"
+        status in (200, 201) and isinstance(post, dict) and post.get("id")
-    )
+    ), f"post creation failed: HTTP {status}, body={post!r}"
    # 5) Read it back by id and assert the message survived the round-trip.
    status, got = harness_http.http_get(f"{base}/posts/{post['id']}", headers=auth, timeout=30)
    assert status == 200 and isinstance(got, dict), f"read-back failed: HTTP {status}, body={got!r}"
-    assert got.get("message") == marker, (
+    assert (
-        f"message did not round-trip: sent {marker!r}, got {got.get('message')!r}"
+        got.get("message") == marker
-    )
+    ), f"message did not round-trip: sent {marker!r}, got {got.get('message')!r}"
@@ -28,6 +28,6 @@ def test_system_ping_ok(live_app):
    url = f"https://{live_app}/api/v4/system/ping"
    status, body = harness_http.retry_http_get(url, expect_status=200, max_wait=120, interval=3)
    assert status == 200, f"GET {url} HTTP {status} (expected 200)"
-    assert isinstance(body, dict) and body.get("status") == "OK", (
+    assert (
-        f"/api/v4/system/ping did not report status=OK; got {body!r}"
+        isinstance(body, dict) and body.get("status") == "OK"
-    )
+    ), f"/api/v4/system/ping did not report status=OK; got {body!r}"
@@ -105,6 +105,6 @@ def test_second_user_reads_first_users_message(live_app):
    # 5) user_b sees user_a's marker (cross-user delivery, not a self read-back)
    messages = [p.get("message") for p in (posts.get("posts") or {}).values()]
-    assert marker in messages, (
+    assert (
-        f"user_b did not see user_a's message {marker!r} in the channel; saw {messages!r}"
+        marker in messages
-    )
+    ), f"user_b did not see user_a's message {marker!r} in the channel; saw {messages!r}"
@@ -18,6 +18,6 @@ def _psql(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded postgres state was not present at backup time"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded postgres state was not present at backup time"
@@ -19,6 +19,6 @@ def _psql(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation postgres state"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation postgres state"
@@ -18,6 +18,6 @@ def _psql(domain, sql):
 def test_upgrade_preserves_data(live_app):
-    assert _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives", (
+    assert (
-        "postgres data did not survive the upgrade"
+        _psql(live_app, "SELECT v FROM ci_marker;") == "upgrade-survives"
-    )
+    ), "postgres data did not survive the upgrade"
@@ -214,7 +214,7 @@ def handshake(
            elif msg_type == MSG_REJECT:
                f = _dec_fields(payload)
                result["error"] = (
-                    f"Rejected: {REJECT_TYPES.get(f.get(1, 0), 'Unknown')} — {f.get(2, '')}"
+                    f"Rejected: {REJECT_TYPES.get(f.get(1, 0), 'Unknown')} " f"— {f.get(2, '')}"
                )
                return result
            elif msg_type == MSG_CHANNELSTATE:
@@ -25,7 +25,7 @@ def test_handshake_completes_with_channel_presence(live_app):
    assert r["server_version"] is not None, "server did not send a Version message"
    assert r["auth_accepted"], f"authentication not accepted — {r.get('error')}"
    # Channel presence: the server must expose at least the root channel (beyond a bare TCP open).
-    assert len(r["channels"]) >= 1, (
+    assert (
-        f"server reported no channels (expected >=1 root channel) — {r!r}"
+        len(r["channels"]) >= 1
-    )
+    ), f"server reported no channels (expected >=1 root channel) — {r!r}"
    assert r["server_sync"], f"ServerSync handshake did not complete — {r.get('error')}"
@@ -23,6 +23,6 @@ def _sqlite(domain, sql):
 def test_backup_captures_state(live_app):
-    assert _sqlite(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "the seeded mumble sqlite marker was not present at backup time"
+        _sqlite(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "the seeded mumble sqlite marker was not present at backup time"
@@ -25,6 +25,6 @@ def _sqlite(domain, sql):
 def test_restore_returns_state(live_app):
-    assert _sqlite(live_app, "SELECT v FROM ci_marker;") == "original", (
+    assert (
-        "restore did not return the pre-mutation mumble sqlite marker (data-integrity failure)"
+        _sqlite(live_app, "SELECT v FROM ci_marker;") == "original"
-    )
+    ), "restore did not return the pre-mutation mumble sqlite marker (data-integrity failure)"
@@ -91,6 +91,6 @@ def test_login_endpoint_returns_json(live_app):
    assert body is not None, f"/rest/login returned no parseable JSON: state={state}"
    # If it's a dict, it's the expected envelope; if it's a list, n8n shouldn't do that on this
    # endpoint, but accept either; only reject obvious non-shapes.
-    assert isinstance(body, dict | list), (
+    assert isinstance(
-        f"/rest/login returned unexpected JSON type {type(body).__name__}: {body!r}"
+        body, dict | list
-    )
+    ), f"/rest/login returned unexpected JSON type {type(body).__name__}: {body!r}"
@@ -72,9 +72,9 @@ def test_rest_settings_returns_json_with_known_keys(live_app):
    # (e.g. version 3.2.0+2.20.6).
    assert isinstance(body, dict), f"/rest/settings returned non-dict JSON: {type(body).__name__}"
    data = body.get("data") if "data" in body else body
-    assert isinstance(data, dict), (
+    assert isinstance(
-        f"/rest/settings response missing 'data' envelope: keys={list(body.keys())[:10]}"
+        data, dict
-    )
+    ), f"/rest/settings response missing 'data' envelope: keys={list(body.keys())[:10]}"
    # Bootstrap keys the editor SPA relies on across versions:
    #   - `userManagement`: the auth-mode dict (whether owner-setup is needed, smtp/email mode).
    #   - `defaultLocale`: i18n bootstrap; present on every n8n install.
--- a/Show More
+++ b/Show More