orchestrator-host: symlink the opencode installer's binary into ~/.local/bin; README: nginx file perms, first-rebuild wrapper, pre-cutover checks

Found on the first activation of #cc-ci on 195.201.88.249: the opencode installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and nginx's pre-start config check runs as the nginx user, so the staged cert/htpasswd must be root:nginx 0640, not root:root 0600. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
nix/hosts/cc-ci: real hardware + networking from the box; document the infect fixes
2026-09-07 20:24:47 +00:00 · 2026-09-07 20:15:16 +00:00 · 2026-09-07 19:58:33 +00:00 · 2026-08-31 20:47:10 +00:00 · 2026-08-31 20:46:55 +00:00 · 2026-08-31 20:31:59 +00:00
60 changed files with 2404 additions and 239 deletions
@@ -0,0 +1,97 @@
+---
+name: cc-ci-cleanup
+description: Tidy the fleet's open recipe PRs. Reconciles every mirror from TRUE upstream first (which alone closes PRs upstream already merged), then surveys every open PR deterministically, CLOSES the ones that can no longer be merged or were never meant to be (CI sweep artifacts, obsolete bumps, superseded duplicates) with a reason, and reports prioritised action items for the ones that SHOULD merge — what specifically is blocking each. NEVER merges a recipe PR. Invoke as /cc-ci-cleanup [recipe ...] [--dry-run].
+---
+
+# cc-ci-cleanup
+
+Open recipe PRs accumulate and rot. Some were never meant to merge (CI sweep artifacts), some were
+overtaken (upstream merged the same change, or a newer PR supersedes them), and some genuinely should
+land but are quietly blocked. Left alone the list becomes noise, and a real CVE fix hides in it.
+
+This skill separates those three, acts on the first two, and hands you a short list for the third.
+
+**Boundaries.** It **CLOSES** irrelevant PRs and **NEVER MERGES** any recipe PR — those change what
+deploys on other people's infrastructure, so a human merges them (see AGENTS.md). Closing is the only
+write it performs, always with a comment saying why.
+
+## Arguments
+- `<recipe> …` — limit to these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
+- `--dry-run` — classify and report, close nothing.
+
+## Procedure
+
+### 1. Reconcile every mirror from TRUE upstream — MANDATORY, FIRST
+```
+cc-ci-plan/reconcile-upstream.sh --all        # or: reconcile-upstream.sh <recipe>...
+```
+**Do not skip this and do not reorder it.** Every signal in step 2 is measured against the mirror's
+`main`; against a stale mirror they are all wrong. This step also does a chunk of the cleanup by
+itself — it closes any PR whose changes upstream has already merged.
+
+> On the first real run (2026-08-11) this alone closed **three** PRs that looked pending and were
+> already merged upstream: discourse #6 (carrying **140 CVEs**), keycloak #6 (**12 CVEs**), n8n #5.
+> All three had been reported to the operator as outstanding work. mailu #6 went the same way earlier
+> the same day. Reconciling is not hygiene, it is how you avoid recommending work that is already done.
+
+### 2. Survey every open PR (deterministic — no judgement yet)
+```
+python3 cc-ci-plan/pr-survey.py [recipe ...]          # add --json for the raw facts
+```
+Per PR it measures: `behind_main`, `ahead`, `mergeable`, `diff_files`, the images it **adds**, which
+of those are **already in main**, `obsolete`, the newest `!testme` verdict + build, `branch_kind`,
+and age/idle days. It decides nothing — that is this skill's job.
+
+### 3. Classify
+
+**CLOSE — cannot merge, or was never meant to.** Each needs a *positive* reason, not an absence:
+
+| signal | why it is closeable |
+|---|---|
+| `branch_kind: ci-artifact` (`ci/*`) | regall/cfold sweeps and `!testme` probes — harness artifacts, never intended to merge |
+| `obsolete: true` | every image it adds is **already pinned in main** — it has nothing left to contribute |
+| superseded | a newer PR on the same recipe makes the same bump (name both numbers in the comment) |
+| `diff_files: 0` | genuinely empty diff — nothing to merge |
+
+**NEVER close on:**
+- `DIFF-UNREADABLE` — the diff could not be fetched, which is NOT an empty diff. gitea #4 reads that
+  way (force-pushed branch) while being a verified, green, needed fix.
+- any field that came back `null`/unknown.
+- a PR that carries a **CVE fix** and is the only thing carrying it, even if it looks stale — report it
+  instead. Losing a security fix to tidiness is far worse than a long PR list.
+- `--dry-run`.
+
+**NEEDS WORK — should merge, something blocks it.** Give the *specific* next action:
+| signal | action item |
+|---|---|
+| `mergeable: false` | conflicts — rebase the branch on `main` and re-run `!testme` |
+| `behind_main > 0` | out of date — rebase, then re-verify (a green from before main moved proves nothing) |
+| `ci: failed` | diagnose via `/ci-test-review`; classify recipe-bug vs stale test |
+| `ci: never-run` | run `!testme` |
+| blocked on the operator | say exactly what is needed (a secret, an upstream release, a decision) |
+
+**READY — green, current, no conflicts.** Action item is simply: review and merge.
+
+### 4. Close the CLOSE set (skip entirely under `--dry-run`)
+Comment first, then close. The comment must say **which signal** made it closeable and **what to do
+if that is wrong** ("reopen if …"), so a wrong call is cheap to undo. Never close silently.
+
+### 5. Report
+Order by what deserves attention, not by recipe name:
+
+1. **CVE-carrying PRs that should merge** — most severe first, with the CVE ids.
+2. Other **READY** PRs (green + current).
+3. **NEEDS WORK**, each with its one specific action.
+4. **CLOSED this run**, with the reason for each.
+5. Anything **deliberately left alone** despite looking stale, and why.
+
+End with a one-line summary: `N open → C closed, R ready to merge, W need work`.
+
+## Guardrails
+- **Never merge a recipe PR.** Create/verify/close only; the operator merges.
+- **Reconcile first, always.** Judging a PR against a stale mirror is how you close good work or
+  recommend work that is already done.
+- **Close only on a positive signal**, never on "looks old". Age alone is not a reason — several
+  60-day-old PRs here are green and mergeable.
+- **Never close a lone CVE fix.** Report it, however stale.
+- Every close gets a comment with its reason and a reopen hint.
@@ -104,6 +104,27 @@ CRITICAL came from, and an image with no window is not counted at all.
  distinction was the difference between two false zeros and the truth (both recipes turned out fine,
  but nothing in the survey said so).

+### 2c. Know which recipes CANNOT see CVEs at all
+```
+python3 cc-ci-plan/audit-sources.py --security-sources
+```
+A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
+measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
+
+**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory
+feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by
+declaring an NVD CPE in their registry:
+```
+- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
+```
+**If this sweep ever reports a blind recipe again, that is the fix**: find the product's CPE at
+nvd.nist.gov and add the line. Prefer a real advisory feed or an attributable changelog when one
+exists — NVD lags the vendor — but a lagging source beats no source, and it turns a `?` into a
+number.
+
+An *unparseable page* is NOT the same thing: it is harmless when the same project also publishes an
+advisory feed (redis, gitea, minio, clickhouse all do). Only "no usable source for this image" counts.
+
 ### 3. Run the advisory scan over that window
 ```
 python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
@@ -18,7 +18,7 @@ RECIPE="${1:?usage: testme-on-pr.sh <recipe> <pr-index>}"
 PRIDX="${2:?usage: testme-on-pr.sh <recipe> <pr-index>}"
 TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
 set -a; . "$TESTENV"; set +a
-: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
+: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
 NS="${GITEA_NAMESPACE:-recipe-maintainers}"
 API="https://${GITEA_URL}/api/v1"; AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
 POST="${POST:-1}"; MAX_WAIT="${MAX_WAIT:-480}"; INTERVAL="${INTERVAL:-30}"
@@ -13,8 +13,8 @@ a health gate, not as silent drift.

 > **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
 > the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
-> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
-> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
+> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo
+> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci
 > CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
 > machine — that's `/cc-ci-server-update`, NOT this skill.

@@ -72,7 +72,7 @@ deploy).
 ### 3. Build (catch errors before any activation)

 ```
-cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
+cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15
 readlink -f result
 ```
 Build failure → fix on the branch (option renames etc.) before going further. Never activate a
@@ -85,7 +85,7 @@ activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route o
 reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
 `hetzner-server-recovery`) — lands back on the last-known-good generation.
 ```
-cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
+cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \
  > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
 # after it settles (poll; tailscaled/sshd may blip):
 readlink /run/current-system            # should be the new store path
@@ -100,7 +100,7 @@ switch.
 ### 5. Switch (make permanent — only after 4 is healthy)

 ```
-cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
+cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10
 ```
 (If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
 detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
@@ -127,7 +127,7 @@ git add flake.lock                                # flake.nix too if the channel
 git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))

 nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
-Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
+Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green."
 git push -u origin HEAD
 ```
 Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
@@ -31,6 +31,12 @@ Then present the roster grouped as follows, and close with the situation guide.
  PR). `--with-tests` also fixes that recipe's stale test.
 - **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.

+**Keeping the PR list honest**
+- **/cc-ci-cleanup** — reconciles every mirror from true upstream (which alone closes PRs upstream
+  already merged), then closes the open recipe PRs that can no longer merge or were never meant to
+  (CI sweep artifacts, obsolete bumps, superseded duplicates) and reports what is actually blocking
+  the ones that should land. Never merges.
+
 **Security (CVEs)**
 - **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
  upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
@@ -81,6 +87,7 @@ ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-ccte
 | "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
 | "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
 | "The report site is stale/missing a week" | `/recipe-report` |
+| "The open PR list is a mess / what should I merge?" | `/cc-ci-cleanup` |
 | "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
 | "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
 | "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
@@ -30,15 +30,18 @@ the orchestrator watches from outside.

 Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
 to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
-watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
-operator reconnects to it (that's why the startup notification matters). The orchestrator now runs on
-a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.30`, public
-`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
-`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
-`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
-`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
+watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
+`cc-ci-orchestrator.service` (`agents.py up orchestrator`) — the operator reconnects to it (that's
+why the startup notification matters). Since 2026-09 the orchestrator runs on the **same Hetzner
+host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
+`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
+repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
+The full provisioning + deploy guide is `README.md`; the move is recorded in
+`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
+shared with notplants) are in `archive/`. Rebuild this host with
+`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
 `nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
-before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
+before the `switch`** (general policy for nix deploys to this host: `test`
 leaves the bootloader and system profile untouched, so a reboot always recovers to the
 last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
 `.cc-ci-logs/server-update-2026-08-03.md`).
@@ -115,3 +118,29 @@ When the orchestrator, Builder, or assistant makes intentional repository change
 promptly and push them to `git.autonomic.zone` in append-only fashion (never force-push). Match the
 existing commit author and message style in this repo. Do not bundle unrelated worktree changes you
 did not make; stage only the intended files.
+
+## Ship as PRs, merge them yourself, operator reviews retrospectively
+
+**This applies to the two INFRASTRUCTURE repos — `recipe-maintainers/cc-ci-orchestrator` (here) and
+`recipe-maintainers/cc-ci` (the CI product).** For work in either:
+
+1. Branch, don't commit straight to `main`.
+2. Open a PR with a description written to be read **after** the fact: what changed, why, and what
+   evidence says it works (test output, a verified run, a before/after number). The PR *is* the
+   review artifact and the historical record.
+3. **Merge it yourself once it is verified** — do not wait for review. The invocation is the
+   authorization; blocking on review would stall the pipeline these repos exist to run.
+4. The operator reviews **retrospectively**, from the PR.
+
+So the PR is not a gate — it is how the work stays legible. A PR that merely says "fix scanner" has
+failed at its only job.
+
+> ### This does NOT extend to RECIPE repos
+> Recipe PRs — any `coop-cloud/<recipe>` or its `recipe-maintainers/<recipe>` mirror — are
+> **created and verified but NEVER merged by an agent**. Those change what deploys on other people's
+> infrastructure, so a human merges them. The split is deliberate: agents own the tooling, the
+> operator owns the recipes.
+
+If work has already landed on `main` without a PR, do not rewrite published history to fix it.
+Create a branch pinned at the pre-work commit and open the PR against that, so the diff is still
+reviewable and merging only advances the pointer (see PRs #2-#5, 2026-08-11).
@@ -1,47 +1,344 @@
 # cc-ci-orchestrator

-Orchestrator workspace for building the **cc-ci** Co-op Cloud recipe CI server. The plan, launch
-tooling, and loop prompts live in [`cc-ci-plan/`](cc-ci-plan/); see [`AGENTS.md`](AGENTS.md) for the
-roles and operating model. Secrets (`.testenv`) are gitignored — never commit them.
+The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
+the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
+host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
+one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
+module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.

-## Run the orchestrator in tmux (survives disconnects + closing your laptop)
+| | where |
+|---|---|
+| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
+| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
+| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
+| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
+| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
+| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
+| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |

-Keep this supervising session alive on the host with tmux, and use `--remote-control` so you can
-watch/steer it from **claude.ai/code** (or the mobile app).
+Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
+commit them.
+
+---
+
+# Deploying a cc-ci host from scratch
+
+This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
+server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
+can repeat it. Read it once before starting; the order matters.
+
+## 0. What you need in hand
+
+- A **Hetzner Cloud** project you can create servers in (console login or an API token).
+- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
+  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
+- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
+  read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
+  `autonomic-bot-gitea-ed25519`, has it).
+- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
+  you are starting fresh you create them (each row says how).
+- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
+
+## 1. Provision the server on Hetzner (Debian image)
+
+In the Hetzner Cloud console (or with `hcloud server create`):
+
+| setting | value | why |
+|---|---|---|
+| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
+| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
+| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
+| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
+| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
+| Name | `cc-ci` | becomes the hostname |
+| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
+
+Check you can log in: `ssh root@<ip> hostname`.
+
+## 2. Convert Debian → NixOS with nixos-infect
+
+[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
+reboots. Run it detached so the SSH session dropping does not kill it:

 ```bash
-# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
-#      /exit        (inside Claude)   or Ctrl-D
-
-# 1. Start a detachable tmux session on this host
-tmux new -s orchestrator
-
-# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
-claude --resume autonomous-orchestrator \
-  --remote-control "autonomous-orchestrator" \
-  --dangerously-skip-permissions
-#    - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
-#    - Or resume by the stable session id (more deterministic in a fresh pane):
-#        claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
-#          --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
-
-# 3. Detach — the process keeps running:  press Ctrl-b, then d
+ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
+#!/usr/bin/env bash
+set -euo pipefail
+# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
+INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
+export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
+export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
+export NIXOS_IMPORT=""             # the real config comes from the flake in §5
+# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
+# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
+export NO_SWAP=true
+curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
+EOF
+chmod +x /root/infect.sh
+nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
 ```

-**Reconnect later**
- On this host: `tmux attach -t orchestrator`
- From anywhere: **claude.ai/code** → the `autonomous-orchestrator` session
+It downloads Nix, builds a NixOS system (5–10 min; follow with
+`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
+`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.

-**Why it survives:** tmux keeps the `claude` process alive across SSH disconnects and your laptop
-closing; remote-control runs *outbound* from this host to Anthropic, so it stays connected
-regardless of the viewer. After a host reboot, re-run steps 1–2.
+### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

-> Two different "names": `--resume <name|id>` selects the *conversation* to restore (shown in the
-> `/resume` picker); the `--remote-control "<name>"` value is only the web display label and resumes
-> nothing. Resuming reuses the same session id each time (stays `34a8…`) — don't pass
-> `--fork-session` unless you intend to branch a new conversation.
->
-> Already inside a live session and just want the web surface? Run `/remote-control` — no exit/resume.
+All three bit on the first attempt; the script above and §3 already include the fixes, this is
+so you recognise them if they come back in another form.
+
+1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
+   The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
+   Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
+2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
+   (`dbus`, `systemd-logind`, `sshd`, networking …) with
+   `Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
+   nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
+   to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
+   lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
+   NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
+   (`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
+   move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
+   `etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
+   `root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
+   `disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
+3. **Boots, units fine, but no network.** The generated `networking.nix` has
+   `defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
+   route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
+   `nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
+   mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
+   so turn it off for that one build:
+   ```bash
+   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
+   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
+   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
+     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
+     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
+     ln -sfn /nix/var/nix/profiles/system /run/current-system
+     nixos-rebuild boot --option sandbox false'
+   ```
+   The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
+   from rescue mode) is what told these apart.
+
+> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
+> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
+> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
+> wraps these (token in `/srv/cc-ci/.hcloud-token`).
+
+## 3. Capture the machine-specific config into this repo
+
+nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
+first two matter; the flake replaces `configuration.nix`.
+
+```bash
+scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
+scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix
+```
+
+Then in `nix/hosts/cc-ci/`:
+
+- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
+  `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
+- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
+  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
+  the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
+  (as on the 2026-09 box) can stay.
+- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
+  is the release you installed (never change it later).
+- `ssh-keys`: the root keys.
+
+Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
+
+## 4. Stage the workspace and secrets on the new host
+
+Everything in this section is **outside git**. Do it as root over SSH, in this order.
+
+### 4a. Tailscale
+
+```bash
+# a reusable (or fresh) tailnet auth key from the tailscale admin console
+install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
+```
+
+### 4b. The CI server's checkout and its one out-of-band secret
+
+```bash
+# root's deploy key for the private cc-ci-secrets submodule
+install -d -m700 /root/.ssh
+install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
+cat > /root/.ssh/config <<'EOF'
+Host git.autonomic.zone
+    Port 2222
+    User git
+    IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
+    IdentitiesOnly yes
+EOF
+# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
+git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
+# the master (recovery) age key — the only sops recipient a fresh host can be
+install -d -m700 /var/lib/sops-nix
+install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
+```
+
+`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
+key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
+`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
+`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.
+
+### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
+run §5 once first if this is a fresh host, then come back)
+
+```bash
+sudo -iu loops
+git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
+sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
+cd /srv/cc-ci-orch
+git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
+git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
+mkdir -p .cc-ci-logs .sops
+```
+
+Then the files below (`install -m600 -o loops -g users`):
+
+| file | what | source |
+|---|---|---|
+| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
+| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
+| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
+| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
+| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
+| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
+| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
+| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
+| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
+
+`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
+`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
+
+## 5. Build and activate
+
+From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
+
+```bash
+# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
+# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
+# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
+# system has none — hence nix-shell), HOME=/root (so root's `git config --global
+# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
+# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
+git config --global --add safe.directory '*'
+systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
+  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
+journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)
+```
+
+`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
+networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
+Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
+
+The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
+and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
+oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
+deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
+
+```bash
+systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
+tailscale status | head -3
+docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
+systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
+systemctl list-timers 'cc-ci-*' nightly-sweep
+sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
+# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
+# /var/lib/acme is restored or a cert has been issued):
+curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
+curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
+curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
+dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
+```
+
+Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
+the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
+it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
+during the install; the box is reachable on its public IP throughout.
+
+When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
+default). **If you are migrating from another host, do §6 before letting it serve anything**: right
+after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
+the new box does not process PR comments or start a second weekly run while the old host is live:
+
+```bash
+docker service scale ccci-bridge_app=0
+systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
+```
+
+## 6. Migrating: restore state from the previous host
+
+Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
+stopped on the new host while its directory is copied:
+
+| path | holds | notes |
+|---|---|---|
+| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
+| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
+| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
+| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
+| `/var/lib/acme-dns` | the acme-dns zone DB | |
+| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
+| `/root/.abra` | abra's per-app env files for the deployed stacks | |
+| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
+| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
+
+## 7. Cutover and verification
+
+1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
+   the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
+   are. Wait for propagation (`dig +short ci.commoninternet.net`).
+2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
+   (acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
+   (dashboard, valid cert), same for `report.ci` and `drone.ci`.
+3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
+   `systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
+   orchestrator. New host: `docker service scale ccci-bridge_app=1`;
+   `systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
+4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
+   open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
+5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
+   (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
+   claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
+6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
+
+## 8. Day 2
+
+- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
+  `nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
+  this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
+- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
+  `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
+  so the deployed checkout the sweep runs from matches.
+- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
+  the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
+  `switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
+  `/hetzner-server-recovery`).
+
+---
+
+# Operating the orchestrator session
+
+The steering session is a long-lived interactive Claude Code session under tmux with
+`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
+`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
+(backend + model in `cc-ci-plan/agents.toml`).
+
+```bash
+# attach on the host
+sudo -iu loops tmux attach -t cc-ci-orchestrator
+# or resume the conversation by hand in a fresh tmux pane
+claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
+# already inside a live session and just want the web surface? /remote-control
+```
+
+`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
+only the web display label. Don't pass `--fork-session` unless you mean to branch.

 ## Kick off / supervise the loops

@@ -53,5 +350,5 @@ cd /srv/cc-ci/cc-ci-plan
 ./launch.sh stop
 ```

-Full supervision guide, credential map, and the Incus VM fallback are in
-[`cc-ci-plan/kickoff.md`](cc-ci-plan/kickoff.md) and [`cc-ci-plan/plan.md`](cc-ci-plan/plan.md) §1.5.
+Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
+`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.
@@ -0,0 +1,22 @@
+# archive/ — how cc-ci and its orchestrator were built and moved, before the combined host
+
+Historical record only. Nothing in here is deployed or evaluated. It was moved out of the live
+tree on 2026-09-07 when the CI server and the orchestrator were consolidated onto one Hetzner
+host (`nixosConfigurations.cc-ci` in `../flake.nix`; deploy guide in `../README.md`; the plan
+that did it is `../cc-ci-plan/plan-cc-ci-combined-host.md`).
+
+| path | what it was |
+|---|---|
+| `nix/configuration-incus-vm.nix` | Channel-based NixOS config of the first orchestrator VM on b1 (Incus, 2 GB). Ran the loops as root; hard-coded the dead Incus cc-ci IP. Replaced by the Hetzner host 2026-05-31. |
+| `nix/README.md` | The README for that Incus VM config. |
+| `nix/cc-ci-orchestrator-hetzner/` | The orchestrator's own Hetzner `cpx22` host (`168.119.126.100`, tailnet `cc-ci-orchestrator-1`), 2026-05-31 → 2026-09. From 2026-08-20 the live copy of this config was `notplants-nix`'s `notplants-orchestrator` host (the box became a shared agent host for several projects); this one had drifted and still carried lichen/project-orchestrator units. Superseded by `../nix/hosts/cc-ci` + `../nix/modules/orchestrator-host.nix`. |
+| `nix/atproto-likes.nix` | A notplants (not cc-ci) service that lived on the shared box; kept by notplants-nix. |
+| `terraform/` | OpenTofu for the `cpx22` orchestrator server (Debian 12 → nixos-infect at `nixos-24.11`). The combined host was provisioned by hand instead; the README documents that path. Note its `user-data.sh` would fail on the Debian 13 image (nixos-infect's temp swapfile on a tmpfs `/tmp`) — see the README's `NO_SWAP=true` note. |
+| `plans/plan-orchestrator-migration.md` | Pi → Incus VM move of the orchestrator (2026-05). |
+| `plans/plan-orchestrator-hetzner-migration.md` | Incus VM → Hetzner `cpx22` move of the orchestrator (2026-05-31). Has the reboot-resilience design (`cc-ci-loops.service`). |
+| `plans/plan-migrate-cc-ci-to-hetzner.md`, `plans/plan-cc-ci-hetzner-migration.md`, `plans/plan-cc-ci-hetzner-terraform.md` | The CI server's own move from the `cc-nix-test` Incus VM to Hetzner `cpx32` (`91.98.47.73`, 2026-05-31), and the terraform that provisioned it (lives in the cc-ci repo). |
+| `plans/plan-repo-consolidation.md` | The earlier repo layout consolidation. |
+
+The cc-ci server's own history (machine-docs, decisions, the clean-room rebuild that proved
+"two repos + one age key + one `nixos-rebuild switch`") is in the cc-ci repo under
+`machine-docs/` and `docs/`.
@@ -14,7 +14,11 @@
  # with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
  # the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
  # because pure evaluation cannot import a path outside the flake.
-  imports = [ ../../atproto-likes.nix ];
+  # cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
+  # the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
+  # a general agent box that also serves unrelated projects — can be reasoned about separately
+  # from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
+  imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];

  services.openssh = {
    enable = true;
@@ -127,7 +131,8 @@ SSHCFG

  # opencode web server — one shared instance; agent sessions attach to it for web visibility.
  # Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
-  # Provider creds are read from /srv/cc-ci/.testenv at startup.
+  # Public provider configuration is tracked in the cc-ci repo; credentials
+  # remain in /srv/cc-ci/.testenv.
  systemd.services.opencode-web = {
    description = "opencode web server for cc-ci agents";
    wantedBy = [ "multi-user.target" ];
@@ -137,7 +142,10 @@ SSHCFG
      Type = "simple";
      User = "loops"; Group = "users";
      WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
-      EnvironmentFile = "/srv/cc-ci/.testenv";
+      EnvironmentFile = [
+        "-/srv/cc-ci/cc-ci/.env.public"
+        "/srv/cc-ci/.testenv"
+      ];
      ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
      ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
      Restart = "on-failure";
@@ -225,72 +233,7 @@ SSHCFG
    };
  };

-  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
-  systemd.services.cc-ci-loops = {
-    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
-    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
-      # default (control-group) systemd kills every leftover process in the cgroup when the unit
-      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
-      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
-      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
-      KillMode = "process";
-      Type = "oneshot"; RemainAfterExit = true;
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci/cc-ci";
-      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
-      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
-    };
-    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
-    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
-    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
-    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
-      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
-      export PATH="/home/loops/.local/bin:$PATH"
-      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
-        echo "workspace not staged yet — skipping loop start"
-    '';
-  };

-  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
-  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
-  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
-  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
-  # the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
-  # attaches to the shared opencode web server (opencode-web.service below) and is reachable for
-  # Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
-  # watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
-  # supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
-  # the same reboot-resilience the other two orchestrators already have.
-  systemd.services.cc-ci-orchestrator = {
-    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
-    wantedBy = [ "multi-user.target" ];
-    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
-      # must not tear down the (shared) tmux server and every agent session with it.
-      KillMode = "process";
-      Type = "oneshot"; RemainAfterExit = true;
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci-orch";
-    };
-    environment = { HOME = "/home/loops"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      proj="/srv/cc-ci-orch"
-      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
-        >> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
-      cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
-    '';
-  };

  # p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
  # (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
@@ -356,76 +299,7 @@ SSHCFG
    '';
  };

-  # Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
-  # verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
-  # phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
-  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
-  systemd.services.cc-ci-upgrade-all = {
-    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci";
-      # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
-      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
-      # (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
-      #   LOOP_BACKEND=opencode
-      #   LOOP_MODEL=opencode-go/glm-5.2
-      # No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
-      # (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
-      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
-    };
-    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
-    '';
-  };

-  systemd.timers.cc-ci-upgrade-all = {
-    description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
-    wantedBy = [ "timers.target" ];
-    timerConfig = {
-      # 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
-      OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
-      Persistent = true;  # if the box was down at the scheduled time, run once on next boot
-    };
-  };

-  # Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
-  # log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
-  # disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
-  # CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
-  # (zero model tokens). Only when a run has stalled/died before completing does it launch a
-  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
-  systemd.services.cc-ci-upgrade-supervisor = {
-    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
-    after = [ "network-online.target" "tailscaled.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci";
-      # Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
-      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
-    };
-    environment = { HOME = "/home/loops"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
-    '';
-  };

-  systemd.timers.cc-ci-upgrade-supervisor = {
-    description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
-    wantedBy = [ "timers.target" ];
-    timerConfig = {
-      OnCalendar = "*-*-* *:07:00";  # every hour at :07 (offset from the weekly :00 fire)
-      Persistent = false;            # a missed hourly check is moot — the next hour re-checks
-    };
-  };
 }
@@ -867,6 +867,129 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
  (/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
  config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
  invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
-  LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
-  modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
-  real launcher) and landed in a different project.
+   LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
+   modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
+   real launcher) and landed in a different project.
+
+## Session 2026-08-15 19:25 UTC — opencode glm-5.2
+
+**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
+been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
+progressing (surveying the 9 remaining recipes). Watching it through to completion.
+
+**What happened (the stall):**
+- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
+  mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
+  had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
+  alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
+- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
+  to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
+  INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
+
+**What I did this session:**
+- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
+  balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
+  (267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
+  I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
+  `socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
+  giant session is unresumable; a fresh start is the only path.
+- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
+- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
+  wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
+  stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
+  stale images on cc-ci (disk 29%), and started a FRESH small session
+  `ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
+  (positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
+  fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
+  lasuite-drive #6, never duplicates), so scoping is safe.
+- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
+  in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
+
+**Phase / loop state:**
+- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
+- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
+- cc-ci server: healthy (disk 29%, runner active).
+
+**Open items for next session:**
+- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
+  upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
+  to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
+  itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
+  the watchdog auto-resumes the SAME (small) session — that works now.
+- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
+  the 2.58M-token request). It's archived; leave it.
+- After the run completes + report publishes, operator review queue = this week's recipe PRs.
+- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
+  supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
+  double-write. No action needed unless it interferes.
+
+**Notes:**
+- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
+  the supervisor kept firing because the run never reached "progressing". Now that balance is restored
+  and a fresh small session is running, the gate should see progress and stand down.
+- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
+  start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
+
+## Session 2026-08-31 17:17 UTC — restricted acme-dns renewal staged
+
+**What happened:** Implemented and deployed the cc-ci restricted ACME DNS-01 design in cc-ci PRs
+#24–#27 (all merged). The initial `sqlite3` backend setting was detected during staging bootstrap
+and corrected to acme-dns 2.x `sqlite` before any account registration succeeded.
+
+**Live state:**
+- `acme-dns` is authoritative for `acme.commoninternet.net` on `91.98.47.73`, with public UDP/TCP
+  53 verified directly; it serves the apex NS and `ns-acme.commoninternet.net A 91.98.47.73`.
+- Its HTTP update API is bound solely to `127.0.0.1:8080`; registration is disabled (local POST
+  returns 403). The one restricted Lego account is stored mode `0600` under `/var/lib/acme/`, outside
+  Git and the Nix store. No Gandi credentials were used or stored.
+- Nix ACME is in Let's Encrypt staging mode for `ci.commoninternet.net` and its wildcard; its renewal
+  timer is enabled. Traefik handoff remains production-gated so staging cannot replace the live cert.
+- Every server deployment used `nixos-rebuild test`, reachability/service verification, then
+  `nixos-rebuild switch` on `#cc-ci-hetzner`.
+
+**Manual Gandi handoff (only blocker):** Add the parent-zone A+NS delegation and permanent CNAME:
+`ns-acme.commoninternet.net. A 91.98.47.73`,
+`acme.commoninternet.net. NS ns-acme.commoninternet.net.`, and
+`_acme-challenge.ci.commoninternet.net. CNAME
+9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` Once propagated, resume with a
+staging issuance, then production cutover/handoff verification. Do not obtain or add a Gandi token.
+
+## Session 2026-08-31 19:05 UTC — ACME production issuance complete
+
+**Completed:** The operator added the narrow Gandi delegation and permanent CNAME. Staging then
+validated both `ci.commoninternet.net` and `*.ci.commoninternet.net`; production issuance validated
+the same SAN pair and produced a Let's Encrypt certificate valid through 2026-11-29. cc-ci PRs #28–#30
+were merged to promote production, make Traefik roll certificate-secret changes even when recipe
+upgrades are held, and remove the retired SOPS wildcard-file injection.
+
+**Live cc-ci state:**
+- `/var/lib/acme/ci.commoninternet.net` is the sole source of the production cert/key; the runtime
+  pair matches it, is root-only for the key, and Traefik now references matching versioned Swarm
+  secrets. Reboot-safe `nixos-rebuild switch` completed at cc-ci `7147d77` after the required test
+  activations and health checks.
+- acme-dns remains authoritative only for the delegated subzone, its update API is localhost-only,
+  registration is disabled, and the restricted-account JSON is mode `0600`. The Nix ACME renewal
+  timer is active. No Gandi credential was added.
+
+**External blocker, not cc-ci-owned:** Direct TLS to `91.98.47.73:443` now presents the new valid
+certificate. The operator-managed public gateway at `143.244.213.108` currently accepts TCP then
+closes TLS before presenting a certificate, so its passthrough to cc-ci needs repair/verification.
+The cc-ci plan explicitly marks the gateway as operator infrastructure; do not change the DNS
+delegation or add a Gandi token to work around it.
+
+## Session 2026-08-31 20:45 UTC — weekly DeepSeek Flash run and report complete
+
+**Completed:** Started the weekly upgrader explicitly on `opencode/deepseek-v4-flash`; its parent
+and every recipe subagent were confirmed with that persisted model. All nine eligible recipe PRs
+completed `!testme` successfully. The public report was then restarted before publication when the
+first handoff was found to use the old GLM model; the replacement report session was confirmed
+`opencode/deepseek-v4-flash` and published successfully at
+`https://report.ci.commoninternet.net/week-2026-08-31.html` (HTTP 200).
+
+**Follow-up fixes:** Orchestrator PR #17 (`1b75d98`) made `testme-on-pr.sh` apply the documented
+public Gitea-host default. cc-ci PR #31 (`769fd29`) adds that public hostname to `.env.public`.
+Both commits were scanned clean and contain no coauthor trailers. No recipe PR was merged.
+
+**Security note:** A subagent briefly enabled shell tracing while debugging the verifier, exposing
+runtime credentials in its private agent trace. No values were committed or put in this journal,
+but rotate the affected `/srv/cc-ci/.testenv` credentials as a precaution.
@@ -39,6 +39,8 @@ keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c i
 ```
 advisory-scan.py <recipe> [--from <version>] [--to <version>]
                          [--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
+
+advisory-scan.py <recipe> --compose-to <URL> [--compose-from <URL>]   # windows derived, not typed
 ```

 | Input | Meaning |
@@ -46,6 +48,8 @@ advisory-scan.py <recipe> [--from <version>] [--to <version>]
 | `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
 | `--from` / `--to` | The **primary app image's** version window being upgraded across |
 | `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
+| `--compose-to URL` | **Derive every window by diffing this compose against its baseline**, instead of typing `--from/--to/--image`. Point it at a PR's `compose.yml`. |
+| `--compose-from URL` | Baseline for the above. Default: the same repo's **default branch, resolved from the API** — never assumed to be `main`. |
 | `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
 | `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
 | `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
@@ -103,10 +107,43 @@ URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as temp
 human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.

 This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
-page names neither. A CVE found **only** here carries no version data, so pass 1 cannot place it — it
-goes to pass 2 (§6).
+page names neither.

-### 2c. OSV.dev — supplementary
+**When the page is a changelog organised by release, each CVE is attributed to the release heading it
+appears under** (`Changes with nginx 1.31.3`, `## v1.31.3`, …) and that becomes its fixed-in version.
+Without this, a project that publishes no advisory feed can never contribute a CVE:
+
+> **nginx publishes NO GitHub security advisories.** Every nginx CVE we can see comes from
+> `nginx.org/en/CHANGES`. Scraping ids out of it without attributing them to a release left them with
+> no patched version, so they were never classifiable — and every nginx bump in the fleet reported
+> **0** forever. nginx is a sidecar in most recipes. Measured: `1.31.1 → 1.31.3` fixes **six** CVEs
+> (three in .2, three in .3); lasuite-docs#7 went 0 → 6 and lasuite-drive#6 went 0 → 3 on this alone.
+
+A changelog CVE is tied to a window by the **image name appearing in the page URL** (window `nginx` ↔
+`nginx.org/...`). A CVE found on a vendor page with no attributable release still has no version data,
+so pass 1 cannot place it — it goes to pass 2 (§6).
+
+### 2c. NVD by CPE — the fallback for projects that publish nothing
+
+Declared per recipe in the registry as `nvd-cpe: <image-key> = <cpe:2.3:...>`.
+
+> **Why it exists.** Two recipes could not see CVEs *at all*: `mattermost-lts` (empty GitHub advisory
+> feed, security bulletins rendered client-side so a text sweep finds nothing) and `mumble` (nothing
+> published anywhere the registry points). Their scans returned `?` — nothing measured. NVD is
+> CPE-indexed and carries structured ranges, so it answers where the vendor does not: mattermost
+> 10.5.0 → 10.12.4 now scores **165**, and mumble finds `CVE-2025-71264` (fixed 1.6.870).
+
+Two range forms, both used:
+
+| NVD field | meaning | how it is judged |
+|---|---|---|
+| `versionEndExcluding X` | fixed in X exactly | a normal patched version (§4a) |
+| `versionEndIncluding X` | affected **up to and including** X; fix version unpublished | fixed when the upgrade crosses X, i.e. `from ≤ X < to` |
+
+**NVD lags the vendor** — it had neither gitea CVSS-9.8 RCE at publication — so this is a fallback,
+never a replacement for 2a/2b. Unauthenticated calls are rate-limited (~5/30s), hence the retry.
+
+### 2d. OSV.dev — supplementary

 Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.

@@ -140,8 +177,44 @@ Two invariants govern this step, both learned from a wrong answer in production.
 > `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
 > cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".

+### 3b. Deriving the windows from a compose diff (`--compose-to`)
+
+Typing `--from/--to/--image` by hand means someone has to remember that the recipe also bumped its
+redis. That is how sidecar CVEs went uncounted for months. This mode reads the windows off the diff:
+
+1. Fetch both compose files (baseline = the repo's **default branch from the API**, since several
+   recipes keep a stale `main` beside a live `master`).
+2. Parse `{service: (image-repo, tag)}` — keyed by **service, not image repo**, because an upgrade
+   may change the repo itself (plausible moved `plausible/analytics` →
+   `ghcr.io/plausible/community-edition`; keyed by repo that reads as one image vanishing and an
+   unrelated one appearing, losing the app window entirely).
+3. Every service whose tag or repo changed becomes a window. The `app` service drives `--from/--to`
+   (coop-cloud convention: it is the recipe's primary image); the rest become `--image` windows.
+   Unchanged images produce no window — inventing one would be a false count.
+4. The derived windows are printed to stderr before the scan, so the inputs are auditable.
+
+Image names are matched against advisory sources **both ways** — an image name is often longer than
+its source repo (`clickhouse/clickhouse-server` vs `ClickHouse/ClickHouse`) and sometimes shorter
+(`redis` vs `redis/redis`).
+
+Verified on plausible PR #5: from the compose URL alone it derives `v2.0.0 → v3.2.1` plus
+`clickhouse-server 23.4.2.11-alpine → 24.12-alpine`, and reports **6** — identical to the
+hand-specified args.
+
+`--from/--to/--image` remain available for finer-grained checks (scanning a window that is not a
+literal compose diff, e.g. "what would the compatibility-safe target fix?").
+
 ### 4a. By patched version (preferred — exact)

+**A fix on the line you are upgrading FROM was already yours.** Projects that maintain several lines
+patch them all at once: mattermost fixed `CVE-2025-11794` in 10.11.4, 10.12.1 *and* 10.5.12. An
+upgrade 10.11.22 → 10.12.4 crosses 10.12.1, so a naive window test counts it — but 10.11.22 is
+already past 10.11.4, so the deployment had the fix before the upgrade. Counting it credits the
+upgrade with work it did not do. This check is **skipped for placeholder versions** (`7.4.X` parses
+to a bare `7.4`, which would read as "already fixed at 7.4" and silently drop a real fix — exactly
+how redis `CVE-2024-46981` was lost when the rule was first added).
+
+
 `patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
 every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
 satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this
@@ -44,7 +44,9 @@ import json
 import os
 import re
 import sys
+import time
 import urllib.error
+import urllib.parse
 import urllib.request

 REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", "/srv/cc-ci/cc-ci-plan/upstream")
@@ -164,6 +166,27 @@ def _vkey(v: str | None) -> tuple:
    return tuple(out)


+def _already_fixed_on_from_line(kf: tuple, cands: list[tuple]) -> bool:
+    """Was it ALREADY fixed on the line we are upgrading FROM?
+
+    The mirror image of _superseded_on_target_line, and just as necessary. mattermost fixes each CVE
+    across several maintained lines at once — CVE-2025-11794 is patched in 10.11.4, 10.12.1 and
+    10.5.12. Upgrading 10.11.22 -> 10.12.4 crosses 10.12.1, so a naive window test counts it; but
+    10.11.22 is already past 10.11.4, so the deployment HAD the fix before the upgrade. Counting it
+    credits the upgrade with work it did not do."""
+    if len(kf) < 2:
+        return False
+    line = kf[:2]
+    for c in cands:
+        if len(c) < 2 or c[:2] != line:
+            continue
+        n = max(len(kf), len(c))
+        pad = lambda z: z + (0,) * (n - len(z))
+        if pad(c) <= pad(kf):
+            return True
+    return False
+
+
 def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool:
    """Does a patched version on the TARGET's own release line sit ABOVE the target?

@@ -272,6 +295,51 @@ def github_advisories(urls: list[str]) -> list[dict]:
    return results


+# Release headings in a vendor changelog. nginx's CHANGES uses "Changes with nginx 1.31.3", most
+# markdown changelogs use "## 1.31.3" / "## v1.31.3".
+_HEADING_RE = re.compile(
+    r"^\s*(?:#{1,4}\s*)?(?:Changes with\s+\S+\s+|Version\s+|Release\s+)?v?(\d+\.\d+(?:\.\d+)*)\s*$"
+    r"|^\s*Changes with\s+\S+\s+(\d+\.\d+(?:\.\d+)*)", re.I)
+
+
+def _changelog_versions(text: str) -> dict:
+    """{cve: version} for a changelog that is ORGANISED BY RELEASE.
+
+    Why this exists: nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes
+    from nginx.org/en/CHANGES, and scraping ids out of it without attributing them to a release
+    leaves them with no patched version — so they can never be classified, and an nginx bump reports
+    0 CVEs forever. nginx 1.31.1 -> 1.31.3 in fact fixes SIX (three in .2, three in .3), and nginx is
+    a sidecar in most of the fleet, so that was a fleet-wide blind spot.
+
+    Attributes each CVE to the nearest PRECEDING release heading — the release that fixed it.
+    """
+    plain = re.sub(r"<[^>]+>", " ", text)
+    out, cur = {}, None
+    for line in plain.splitlines():
+        m = _HEADING_RE.match(line)
+        if m:
+            cur = m.group(1) or m.group(2)
+            continue
+        if cur:
+            for cve in CVE_RE.findall(line):
+                out.setdefault(cve, cur)
+    return out
+
+
+_BLOB_RE = re.compile(r"^https://github\.com/([^/]+)/([^/]+)/blob/(.+)$")
+
+
+def _raw_if_blob(url: str) -> str:
+    """A GitHub *blob* URL is an HTML viewer, not the file.
+
+    The registry pointed ONLYOFFICE's CHANGELOG.md at its blob page. Fetching that returns 636KB of
+    markup in which the release headings do not survive HTML-stripping, so 24 CVEs were visible and
+    NONE attributable to a release — the same shape of blind spot as nginx. The raw URL attributes
+    all 24. Normalising here fixes every registry entry at once, present and future."""
+    m = _BLOB_RE.match(url)
+    return f"https://raw.githubusercontent.com/{m.group(1)}/{m.group(2)}/{m.group(3)}" if m else url
+
+
 def vendor_pages(urls: list[str]) -> list[dict]:
    """Fetch each registry URL and regex out CVE ids, with a little surrounding context."""
    out = []
@@ -284,20 +352,93 @@ def vendor_pages(urls: list[str]) -> list[dict]:
            # correct; counting them as failures would wrongly mark the recipe's count unreliable.
            out.append({"source": u, "status": "skipped: template URL (not fetchable)", "cves": [], "context": {}})
            continue
-        entry = {"source": u, "status": "ok", "cves": [], "context": {}}
+        entry = {"source": u, "status": "ok", "cves": [], "context": {}, "fixed_in": {}}
        try:
-            text = _fetch(u)
+            text = _fetch(_raw_if_blob(u))
            plain = re.sub(r"<[^>]+>", " ", text)
            for cve in sorted(set(CVE_RE.findall(plain))):
                entry["cves"].append(cve)
                i = plain.find(cve)
                entry["context"][cve] = re.sub(r"\s+", " ", plain[max(0, i - 160) : i + 200]).strip()
+            entry["fixed_in"] = _changelog_versions(text)
        except Exception as e:  # noqa: BLE001
            entry["status"] = f"error: {type(e).__name__}: {e}"
        out.append(entry)
    return out


+NVD_API = "https://services.nvd.nist.gov/rest/json/cves/2.0"
+NVD_CPE_RE = re.compile(r"^\s*[-*]?\s*nvd-cpe:\s*(\S+)\s*=\s*(cpe:2\.3:[^\s`]+)", re.M | re.I)
+
+
+def registry_cpes(recipe: str, registry_dir: str) -> list[tuple[str, str]]:
+    """[(image-key, cpe)] declared in the recipe's registry as `nvd-cpe: <key> = <cpe>`."""
+    path = os.path.join(registry_dir, f"{recipe}.md")
+    try:
+        return [(m.group(1), m.group(2)) for m in NVD_CPE_RE.finditer(open(path).read())]
+    except OSError:
+        return []
+
+
+def nvd_advisories(cpe: str, key: str) -> dict:
+    """CVEs for a CPE from NVD, with the version data the classifier needs.
+
+    THE FALLBACK FOR PROJECTS THAT PUBLISH NOTHING MACHINE-READABLE. mattermost's GitHub advisory
+    feed is empty and its security bulletins are client-side rendered; mumble publishes neither. Both
+    scanned as `?` — nothing measured — until here. NVD is CPE-indexed and carries structured ranges:
+
+      versionEndExcluding X  -> fixed in X exactly            (a patched version)
+      versionEndIncluding X  -> affected up to and INCLUDING X, fixed in some later release. The
+                                exact fix version is unknown, but the upgrade fixes it whenever it
+                                crosses X — recorded as `affected_max` and judged in the classifier.
+
+    NVD LAGS the vendor (it had neither gitea CVSS-9.8 RCE at publication), so this is a fallback,
+    never a replacement for 2a/2b. Unauthenticated calls are rate-limited to ~5/30s, hence the retry.
+    """
+    entry = {"source": f"nvd:{key}", "status": "ok", "advisories": []}
+    url = f"{NVD_API}?resultsPerPage=2000&virtualMatchString={urllib.parse.quote(cpe)}"
+    data = None
+    for attempt in range(3):
+        try:
+            data = json.loads(_fetch(url))
+            break
+        except Exception as e:  # noqa: BLE001
+            if attempt == 2:
+                entry["status"] = f"error: {type(e).__name__}"
+                return entry
+            time.sleep(8)
+    for v in (data or {}).get("vulnerabilities", []):
+        c = v.get("cve") or {}
+        cid = c.get("id")
+        if not cid:
+            continue
+        fixed, affected_max = set(), set()
+        for cfg in c.get("configurations", []):
+            for node in cfg.get("nodes", []):
+                for m in node.get("cpeMatch", []):
+                    if m.get("versionEndExcluding"):
+                        fixed.add(m["versionEndExcluding"])
+                    elif m.get("versionEndIncluding"):
+                        affected_max.add(m["versionEndIncluding"])
+        sev = None
+        for mk in ("cvssMetricV31", "cvssMetricV30", "cvssMetricV2"):
+            got = (c.get("metrics") or {}).get(mk) or []
+            if got:
+                sev = (got[0].get("cvssData") or {}).get("baseSeverity")
+                break
+        entry["advisories"].append({
+            "cve": cid, "ghsa": None, "severity": (sev or "").lower() or None,
+            "summary": next((d.get("value") for d in c.get("descriptions", [])
+                             if d.get("lang") == "en"), "")[:200],
+            "vulnerable_range": None,
+            "patched": "; ".join(sorted(fixed)) or None,
+            "affected_max": "; ".join(sorted(affected_max)) or None,
+            "url": f"https://nvd.nist.gov/vuln/detail/{cid}",
+            "published_at": c.get("published"), "description": None, "cvss": None,
+        })
+    return entry
+
+
 def osv(recipe: str, version: str | None) -> dict | None:
    pkg = OSV_PACKAGES.get(recipe)
    if not pkg or not version:
@@ -595,7 +736,8 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
        e = report["cves"].setdefault(cve, {"sources": [], "severity": None, "ghsa": None,
                                            "vulnerable_range": None, "patched": None,
                                            "context": None, "published_at": None,
-                                            "description": None, "url": None, "cvss": None})
+                                            "description": None, "url": None, "cvss": None,
+                                            "changelog_fixed_in": None, "affected_max": None})
        if src not in e["sources"]:
            e["sources"].append(src)
        for k, v in extra.items():
@@ -612,11 +754,22 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
                       context=a.get("summary"), published_at=a.get("published_at"),
                       description=a.get("description"), url=a.get("url"), cvss=a.get("cvss"))

+    for key, cpe in registry_cpes(recipe, registry_dir):
+        entry = nvd_advisories(cpe, key)
+        report["sources"].append({"source": entry["source"], "status": entry["status"],
+                                  "found": len(entry.get("advisories") or [])})
+        for a in entry.get("advisories", []):
+            record(a["cve"], entry["source"], severity=a.get("severity"),
+                   patched=a.get("patched"), affected_max=a.get("affected_max"),
+                   context=a.get("summary"), published_at=a.get("published_at"),
+                   url=a.get("url"))
+
    for entry in vendor_pages(urls):
        report["sources"].append({"source": entry["source"], "status": entry["status"],
                                  "found": len(entry.get("cves", []))})
        for cve in entry.get("cves", []):
-            record(cve, entry["source"], context=entry["context"].get(cve))
+            record(cve, entry["source"], context=entry["context"].get(cve),
+                   changelog_fixed_in=(entry.get("fixed_in") or {}).get(cve))

    for version in filter(None, (v_from, v_to)):
        o = osv(recipe, version)
@@ -651,17 +804,39 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
    #
    # A source with no window is not classified: its advisories are listed as unclassified so they
    # stay visible without inflating the count.
-    gh_sources = [x["source"] for x in report["sources"] if x["source"].startswith("github-advisories:")]
+    gh_sources = [x["source"] for x in report["sources"]
+                  if x["source"].startswith(("github-advisories:", "nvd:"))]
    primary = gh_sources[0] if (gh_sources and (v_from or v_to)) else None
    report["primary_source"] = primary

    windows = {}                                    # source name -> (from, to)
+    window_key = {}                                 # source name -> the image name it covers
    if primary:
        windows[primary] = (v_from, v_to)
+        window_key[primary] = primary.split("/")[-1]
+        # The app's window must also cover its NVD entry. NVD sources are keyed by IMAGE name
+        # (`mattermost-team-edition`) while the advisory feed is keyed by REPO (`mattermost/
+        # mattermost`), so without this the fallback source that exists precisely because the feed
+        # is empty would itself go unwindowed — and mumble/mattermost would still report nothing.
+        pname = primary.split("/")[-1].lower()
+        for src in gh_sources:
+            if src.startswith("nvd:") and src not in windows:
+                k = src.split(":", 1)[1].lower()
+                if pname in k or k in pname:
+                    windows[src] = (v_from, v_to)
+                    window_key[src] = k
    for key, wf, wt in (images or []):
        for src in gh_sources:
-            if key.lower() in src.lower() and src not in windows:
+            if src in windows:
+                continue
+            # Match BOTH ways: an image name is often longer than its source repo
+            # (`clickhouse/clickhouse-server` vs source `ClickHouse/ClickHouse`) and sometimes
+            # shorter (`redis` vs `redis/redis`). One-directional matching silently dropped the
+            # clickhouse window when the key was derived from a compose file.
+            k, name = key.lower(), src.split("/")[-1].lower()
+            if k in src.lower() or name in k:
                windows[src] = (wf, wt)
+                window_key[src] = key
    report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()}

    def _classify_window(src, wf, wt):
@@ -680,12 +855,31 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
                    continue
                patched = e.get("patched") or ""
                cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
+                # NEVER on a placeholder: "7.4.X" parses to the bare 7.4, which then reads as
+                # "already fixed at 7.4" and silently drops a real fix (redis CVE-2024-46981).
+                # A placeholder means the fix version is unknown — that is the indeterminate path.
+                if (kf and kt and not PLACEHOLDER_RE.search(patched)
+                        and _already_fixed_on_from_line(kf, cands)):
+                    # already had it before the upgrade
+                    e.setdefault("classification", "outside-window")
+                    continue
                if kf and kt and _superseded_on_target_line(kt, cands):
                    # The target's own line got the fix LATER than the target: not fixed here.
                    e.setdefault("classification", "outside-window")
                    continue
                if kf and kt and any(_within(kf, kt, c) for c in cands):
                    got.add(cve)
+                elif kf and kt and e.get("affected_max"):
+                    # NVD's `versionEndIncluding X`: affected up to and INCLUDING X, fixed in some
+                    # later release. The exact fix version is unpublished, but the upgrade delivers
+                    # it whenever it crosses X — i.e. from <= X < to.
+                    for t in re.findall(r"\d+(?:\.\d+)*", e["affected_max"]):
+                        x = _vkey(t)
+                        n = max(len(kf), len(kt), len(x))
+                        pad = lambda z: z + (0,) * (n - len(z))
+                        if x and pad(kf) <= pad(x) < pad(kt):
+                            got.add(cve)
+                            break
                elif not patched or PLACEHOLDER_RE.search(patched):
                    # No fix version published ("TBD") or only a placeholder ("7.4.X" — which could
                    # be 7.4.1, inside the window). We cannot say either way, so say so.
@@ -719,6 +913,32 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
            report["cves"][cve]["classification"] = f"fixed-by-this-upgrade ({method}) via {src}"
            fixed_set.add(cve)

+    # A CVE seen only in a vendor CHANGELOG has no advisory feed behind it, but the changelog says
+    # which release fixed it (see _changelog_versions). Tie it to a window by the image name
+    # appearing in the page URL — nginx's window is `nginx`, and its changelog is nginx.org/... .
+    # Without this, projects that publish no GitHub advisories (nginx being the big one) can never
+    # contribute a CVE, and every nginx bump in the fleet silently reports 0.
+    from_changelog = {}
+    for cve, e in report["cves"].items():
+        if cve in fixed_set or not e.get("changelog_fixed_in"):
+            continue
+        for src, (wf, wt) in windows.items():
+            key = (window_key.get(src) or "").lower()
+            if not key:
+                continue
+            if not any(key in s_.lower() for s_ in e["sources"] if s_.startswith("http")):
+                continue
+            kf, kt = _vkey(wf), _vkey(wt)
+            cand = _vkey(e["changelog_fixed_in"])
+            if kf and kt and cand and _within(kf, kt, cand):
+                e["classification"] = (f"fixed-by-this-upgrade (named under {e['changelog_fixed_in']} "
+                                       f"in the vendor changelog) via {src}")
+                fixed_set.add(cve)
+                from_changelog[cve] = e["changelog_fixed_in"]
+            break
+    if from_changelog:
+        report["resolved_by_changelog"] = from_changelog
+
    unknown = []
    for cve, e in report["cves"].items():
        if cve in fixed_set:
@@ -893,6 +1113,118 @@ def markdown(rep: dict) -> str:
    return "\n".join(L)


+def _gitea_auth(url: str) -> dict:
+    """Basic auth for the private mirror, using public config plus .testenv.
+
+    Sent as a HEADER, never embedded in the URL: in-URL credentials leak into shell history, process
+    lists and error messages, and urllib mis-parses a password containing a colon."""
+    host = re.sub(r"^https?://", "", url).split("/")[0]
+    env = {}
+    for path in (
+        "/srv/cc-ci/cc-ci/.env.public",
+        os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv"),
+    ):
+        try:
+            lines = open(path)
+        except OSError:
+            continue
+        for ln in lines:
+            if "=" in ln and not ln.strip().startswith("#"):
+                k, v = ln.strip().split("=", 1)
+                env[k] = v.strip().strip("\"'")
+    if host != env.get("GITEA_URL", "git.autonomic.zone"):
+        return {}
+    u, pw = env.get("GITEA_USERNAME"), env.get("GITEA_PASSWORD")
+    if not (u and pw):
+        return {}
+    import base64 as _b64
+    return {"Authorization": "Basic " + _b64.b64encode(f"{u}:{pw}".encode()).decode()}
+
+
+def _compose_images(url: str) -> dict[str, tuple[str, str]]:
+    """{service: (image-repo, tag)} for a compose file.
+
+    Keyed by SERVICE, not by image repo, because an upgrade may change the repo itself: plausible
+    moved `plausible/analytics` -> `ghcr.io/plausible/community-edition`. Keyed by repo that reads
+    as one image vanishing and an unrelated one appearing, and the app's version window is lost —
+    which is exactly the upgrade most worth scanning."""
+    txt = _fetch(url, _gitea_auth(url))
+    out, svc = {}, None
+    in_services = False
+    for line in txt.splitlines():
+        if re.match(r"^services:\s*$", line):
+            in_services = True
+            continue
+        if in_services and re.match(r"^\S", line):
+            in_services = False
+        if not in_services:
+            continue
+        m = re.match(r"^  (\S+):\s*$", line)
+        if m:
+            svc = m.group(1)
+            continue
+        m = re.match(r"^\s+image:\s*[\"']?([^\"'\s]+)", line)
+        if m and svc:
+            ref = m.group(1).split("@", 1)[0]
+            if "${" in ref or "$(" in ref:
+                continue
+            repo, _, tag = ref.rpartition(":")
+            if repo and tag:
+                out[svc] = (repo, tag)
+    return out
+
+
+def _default_branch_compose(url: str) -> str | None:
+    """Same repo as `url`, but its DEFAULT branch — resolved from the API, never assumed.
+
+    Several coopcloud recipes keep a stale `main` beside the real default `master` (gitea's `main`
+    is 1.24.2-rootless while `master` has 1.27.1-rootless), so guessing the branch produces a
+    confidently wrong baseline."""
+    m = re.match(r"(https?://[^/]+)/([^/]+)/([^/]+)/(?:raw|src)/branch/[^/]+/(.*)$", url)
+    if not m:
+        return None
+    host, owner, repo, path = m.groups()
+    try:
+        meta = json.loads(_fetch(f"{host}/api/v1/repos/{owner}/{repo}", _gitea_auth(host)))
+        br = meta.get("default_branch")
+    except Exception:  # noqa: BLE001
+        return None
+    return f"{host}/{owner}/{repo}/raw/branch/{br}/{path}" if br else None
+
+
+def windows_from_compose(to_url: str, from_url: str | None = None) -> tuple[list, str | None]:
+    """Derive the scan's version windows by DIFFING two compose files.
+
+    This is the deterministic alternative to a human (or a model) deciding which `--image` args a
+    given upgrade needs. Point it at a PR's compose and it reads the windows straight off the diff:
+    every image whose tag changed becomes a window, every image that did not change is correctly
+    left out, and nothing depends on anyone remembering that the recipe also bumped its redis.
+
+    Returns (windows, note) where windows is [(image-name, from, to)].
+    """
+    if from_url is None:
+        from_url = _default_branch_compose(to_url)
+        if not from_url:
+            raise SystemExit("could not resolve a baseline compose; pass --compose-from explicitly")
+    new, old = _compose_images(to_url), _compose_images(from_url)
+    app, others = None, []
+    for svc, (repo, tag) in sorted(new.items()):
+        if svc not in old:
+            continue
+        prev_repo, prev_tag = old[svc]
+        if prev_tag == tag and prev_repo == repo:
+            continue
+        # The `app` service is the recipe's primary image by coop-cloud convention; its window drives
+        # --from/--to so the scan's primary advisory source is judged against it. Everything else is
+        # a sidecar window keyed by its image name.
+        if svc == "app":
+            app = (repo.split("/")[-1], prev_tag, tag)
+        else:
+            others.append((repo.split("/")[-1], prev_tag, tag))
+    wins = ([app] if app else []) + others
+    return wins, f"baseline {from_url}"
+
+
 def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("recipe")
@@ -904,6 +1236,13 @@ def main() -> int:
                         "fix version published), fetch their full text + references and append a "
                         "block for the agent to judge. Additive: it never changes the count above.")
    ap.add_argument("--registry", default=REGISTRY_DIR)
+    ap.add_argument("--compose-to", default=None, metavar="URL",
+                    help="derive the windows by DIFFING this compose against its baseline, instead "
+                         "of passing --from/--to/--image by hand. Point it at a PR's compose.yml "
+                         "(e.g. .../raw/branch/<pr-branch>/compose.yml).")
+    ap.add_argument("--compose-from", default=None, metavar="URL",
+                    help="baseline compose for --compose-to. Default: the same repo's DEFAULT "
+                         "branch, resolved from the API (never assumed to be `main`).")
    ap.add_argument("--image", action="append", default=[], metavar="NAME=FROM:TO",
                    help="a sidecar image and the versions it moved between, e.g. "
                         "--image redis=7.4:8.10 (repeatable). NAME matches a source repo name; "
@@ -911,6 +1250,20 @@ def main() -> int:
                         "being left unclassified.")
    a = ap.parse_args()
    images = []
+    if a.compose_to:
+        wins, note = windows_from_compose(a.compose_to, a.compose_from)
+        if not wins:
+            print(f"### Advisory scan — {a.recipe}\n\n**No image versions changed between the two "
+                  f"compose files, so this upgrade fixes no CVEs by definition.**\n\n_{note}_")
+            return 0
+        print(f"_derived from compose diff ({note}):_", file=sys.stderr)
+        for n_, f_, t_ in wins:
+            print(f"_  {n_}: {f_} → {t_}_", file=sys.stderr)
+        # The `app` service (first entry when present) drives --from/--to; the rest are --image
+        # windows. Passing every window as --image too is harmless: each is matched by name against
+        # the advisory sources, and an unmatched name is simply ignored.
+        a.v_from, a.v_to = a.v_from or wins[0][1], a.v_to or wins[0][2]
+        images = list(wins[1:])
    for spec in a.image:
        name, _, rng = spec.partition('=')
        vf, _, vt = rng.partition(':')
@@ -34,7 +34,7 @@ supports_resume = false
 prompt_delivery = "ping"       # send after the TUI connects
 connect_delay   = 12
 submit_key      = "C-m"
-preamble        = "set -a; . /srv/cc-ci/.testenv; set +a"
+preamble        = "set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a"
 stall_idle      = 900
 active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
 limit_re  = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
@@ -42,6 +42,11 @@ _spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HE
 RI = importlib.util.module_from_spec(_spec)
 _spec.loader.exec_module(RI)

+# advisory-scan supplies the source-fetching + changelog-attribution used by --security-sources
+_aspec = importlib.util.spec_from_file_location("advisory_scan", os.path.join(HERE, "advisory-scan.py"))
+A = importlib.util.module_from_spec(_aspec)
+_aspec.loader.exec_module(A)
+
 REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream"))
 USED_RECIPES = os.path.join(HERE, "used-recipes.md")
 DEPRECATION_RE = re.compile(
@@ -108,6 +113,45 @@ def newest_tag_date(registry: str, repo: str, tag: str) -> str | None:
    return (d.get("results") or [{}])[0].get("last_updated")


+def security_source_audit(recipe: str) -> list[dict]:
+    """Per source: are its CVEs USABLE, or merely visible?
+
+    The nginx lesson. nginx publishes no GitHub advisories; all its CVEs live in nginx.org/en/CHANGES.
+    The scan saw them and could do nothing with them, because nothing said which release fixed which
+    CVE — so every nginx bump in the fleet reported 0. Attribution (advisory-scan §2b) fixed that for
+    changelogs organised by release, but a page that lists CVEs with NO release structure is still a
+    blind spot: visible, uncountable. This finds those.
+
+    Per source: `advisory-feed` (structured, best), `changelog` (CVEs attributable to a release),
+    `unattributable` (CVEs present but no release structure — BLIND), or `no-cve-data`.
+    """
+    urls, _ = _registry_urls(recipe)
+    out = []
+    # NVD CPE entries are a first-class source: for projects publishing nothing machine-readable
+    # (mattermost, mumble) they are the ONLY structured source, and omitting them here made two
+    # recipes look permanently blind after they had been fixed.
+    for key, cpe in A.registry_cpes(recipe, REGISTRY_DIR):
+        e = A.nvd_advisories(cpe, key)
+        n = len(e.get("advisories") or [])
+        out.append({"source": e["source"] + f"  ({cpe.split(':')[4]}/{cpe.split(':')[3]})",
+                    "kind": "advisory-feed" if n else "no-cve-data",
+                    "status": e["status"], "cves": n, "usable": n})
+    for entry in A.github_advisories(urls):
+        out.append({"source": entry["source"], "kind": "advisory-feed",
+                    "status": entry["status"], "cves": len(entry.get("advisories") or []),
+                    "usable": len(entry.get("advisories") or [])})
+    for entry in A.vendor_pages(urls):
+        if entry["status"].startswith("skipped"):
+            continue
+        n = len(entry.get("cves") or [])
+        attributed = len(entry.get("fixed_in") or {})
+        kind = ("no-cve-data" if n == 0 else
+                "changelog" if attributed else "unattributable")
+        out.append({"source": entry["source"], "kind": kind, "status": entry["status"],
+                    "cves": n, "usable": attributed})
+    return out
+
+
 def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
    out = {"recipe": recipe, "findings": [], "images": [], "sources": []}
    try:
@@ -214,9 +258,41 @@ def main() -> int:
    ap.add_argument("--ssh", default=None)
    ap.add_argument("--quiet-days", type=int, default=365)
    ap.add_argument("--json", action="store_true")
+    ap.add_argument("--security-sources", action="store_true",
+                    help="audit whether each recipe's CVE sources are USABLE (structured advisory "
+                         "feed / release-attributable changelog) or merely visible")
    a = ap.parse_args()

    recipes = a.recipes or all_recipes()
+    if a.security_sources:
+        # What matters is whether the RECIPE can see CVEs at all — not whether some individual page
+        # is unparseable. A page with no release structure is harmless when the same project also
+        # publishes an advisory feed (redis, gitea, minio, clickhouse all do); it is only a blind
+        # spot when nothing else covers that project.
+        blind_recipes, noisy = [], 0
+        for r in recipes:
+            rows = security_source_audit(r)
+            feeds = [x for x in rows if x["kind"] == "advisory-feed" and x["cves"] > 0]
+            logs = [x for x in rows if x["kind"] == "changelog"]
+            unattr = [x for x in rows if x["kind"] == "unattributable"]
+            noisy += len(unattr)
+            usable = len(feeds) + len(logs)
+            if usable == 0:
+                blind_recipes.append(r)
+                print(f"!! {r}: NO USABLE CVE SOURCE — {len(unattr)} unparseable page(s), "
+                      f"0 advisory feeds, 0 attributable changelogs")
+                for x in rows:
+                    print(f"      {x['kind']:15} {x['source'][:64]} ({x['cves']} CVEs)")
+            else:
+                print(f"OK {r}: {len(feeds)} advisory-feed(s), {len(logs)} changelog(s)"
+                      + (f", {len(unattr)} unparseable page(s) (redundant — covered by a feed)"
+                         if unattr else ""))
+                for x in logs:
+                    print(f"      changelog  {x['source'][:62]} ({x['usable']}/{x['cves']})")
+        print(f"\n{len(recipes)} recipes · {len(blind_recipes)} with NO usable CVE source"
+              + (f": {', '.join(blind_recipes)}" if blind_recipes else "")
+              + f" · {noisy} unparseable page(s) elsewhere (harmless where a feed covers them)")
+        return 0
    reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes]
    if a.json:
        print(json.dumps(reports, indent=2))
@@ -112,7 +112,7 @@ def start(mode="resume"):
        if not Path(OPENCODE_BIN).exists():
            die(f"opencode not found at {OPENCODE_BIN}")
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {WORKDIR}"
        )
        log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
@@ -154,7 +154,7 @@ def start(mode="resume"):
                f"OPENCODE_CONFIG_CONTENT={shlex.quote(json.dumps({'model': LOOP_MODEL}))} "
            )
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"{model_env}NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {shlex.quote(WORKDIR)}"
        )
        log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
@@ -109,7 +109,7 @@ def start(mode, date):
        cwd = "/srv/cc-ci-orch/cc-ci"
        share_flag = "--share" if OPENCODE_SHARE else ""
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
            f"--title '{SESSION}' --dir {cwd} \"$(cat '{kf}')\""
        )
@@ -107,7 +107,7 @@ def spawn_supervisor(sid, reason):
    kf = Path(LOG_DIR) / f".kickoff-{SUP_SESSION}.txt"
    kf.write_text(build_kickoff(sid, reason))
    share = "--share" if OPENCODE_SHARE else ""
-    cmd = (f"set -a; . {WORKDIR}/.testenv; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
+    cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
           f"--attach '{OPENCODE_SERVER}' --title '{SUP_SESSION}' --dir {WORKDIR} \"$(cat '{kf}')\"")
    _sh(["tmux", "new-session", "-d", "-s", SUP_SESSION, "-c", WORKDIR, cmd])
    _sh(["tmux", "pipe-pane", "-o", "-t", SUP_SESSION, f"cat >> '{LOG_DIR}/{SUP_SESSION}.log'"])
@@ -216,7 +216,7 @@ def start(mode="use-or-create"):
        # so they must come AFTER `run` (a global `opencode --model X run` is ignored).
        share_flag = "--share" if OPENCODE_SHARE else ""
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
            f"--title '{SESSION}' \"$(cat '{kf}')\""
        )
@@ -521,7 +521,7 @@ def resume(reason="manual"):
            "write+push the weekly summary, then `python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`, print "
            "'" + DONE_MARKER + "', and go idle.")
    share = "--share" if OPENCODE_SHARE else ""
-    cmd = (f"set -a; . /srv/cc-ci/.testenv; set +a; {OPENCODE_BIN} run -s {sid} --continue "
+    cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run -s {sid} --continue "
           f"--model '{MODEL}' {share} --attach '{OPENCODE_SERVER}' --dir '{WORKDIR}' \"$(cat '{kf}')\"")
    subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", WORKDIR, cmd])
    subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_FILE}'"])
@@ -323,7 +323,7 @@ def start_agent(role, session, workdir):
        # absolute repo paths for builder/adversary work.
        session_cwd = "/srv/cc-ci-orch/cc-ci"
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {session_cwd}"
        )
        log(f"starting {session} (backend=opencode, phase={pid}, model={model or 'default'})")
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Load cc-ci runtime configuration. Source this file; do not execute it.
+#
+# Callers enable `set -a` so both files become exported environment variables.
+# Public configuration is loaded first, then the secret file. This preserves a
+# local secret override only where an emergency migration explicitly needs it.
+
+if [ -r /srv/cc-ci/cc-ci/.env.public ]; then
+  . /srv/cc-ci/cc-ci/.env.public
+fi
+
+if [ ! -r /srv/cc-ci/.testenv ]; then
+  echo "cc-ci secret environment is missing: /srv/cc-ci/.testenv" >&2
+  return 1
+fi
+
+. /srv/cc-ci/.testenv
@@ -71,7 +71,7 @@ log "writing report $REPORT ..."
  if [ -f "$SUMMARY" ]; then cat "$SUMMARY"; else echo "(no dated summary at $SUMMARY — /upgrade-all may have stalled on the usage limit or errored; check the cc-ci-upgrader session + $LOG)"; fi
  echo
  echo "## Open PRs per recipe (post-run)"
-  set -a; . /srv/cc-ci/.testenv 2>/dev/null; set +a
+  set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
  G="https://${GITEA_USERNAME}:${GITEA_PASSWORD}@${GITEA_URL}/api/v1"
  for r in $(curl -fsS "$G/orgs/recipe-maintainers/repos?limit=100" 2>/dev/null | python3 -c "import sys,json;print('\n'.join(sorted(x['name'] for x in json.load(sys.stdin))))" 2>/dev/null); do
    case "$r" in cc-ci|cc-ci-orchestrator|cc-ci-secrets|archived-*) continue;; esac
@@ -0,0 +1,102 @@
+# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator
+
+**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
+cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
+cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
+(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
+`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
+orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
+Last step (separate, after everything works on the current names): move both to `autonomic.zone`
+subdomains.
+
+## Facts (2026-09-07)
+
+| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
+|---|---|---|---|
+| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
+| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
+| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
+| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
+| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |
+
+Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
+`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
+(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
+`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
+`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
+volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
+rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
+oneshots recreate them from sops.
+
+Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
+key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
+`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
+`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
+htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.
+
+## Design
+
+**cc-ci repo** (`feat/nixos-module-export`):
+- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
+  host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
+  `environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
+  tailscale, no root keys, no stateVersion — the host supplies those.
+- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
+  `sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
+  path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
+- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.
+
+**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
+- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
+- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
+  `nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
+  orchestrator session and the weekly/hourly timers.
+- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
+  nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
+  (on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
+- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
+  both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
+  `--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
+- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
+- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
+- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
+  `nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.
+
+**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
+four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
+notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
+
+## Steps
+
+1. [ ] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
+   `hardware-configuration.nix` + `networking.nix`.
+2. [ ] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
+3. [ ] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
+   `#cc-ci` evaluates; PR.
+4. [ ] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
+   Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
+   new host does not double-process `!testme` or run a second weekly upgrade before cutover.
+5. [ ] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
+   /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
+6. [ ] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
+   drone, one direct `cc-ci-run` on custom-html-tiny).
+7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
+   drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
+8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
+   state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
+9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
+10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
+
+## Log
+
+- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
+  `notplants-orchestrator-ed25519`.
+- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
+  PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
+  nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
+  Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
+  Hetzner console (no API token for that project on this host).
+- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
+  `#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
+  `#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
+  hardware/networking copied from the old CI server — to be replaced by the infect output.
@@ -83,9 +83,10 @@ repo, a commit, a log, or the dashboard** (§9) — reference locations only.
 | What | Where | How to use |
 |---|---|---|
 | **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; `Host cc-ci` in `~/.ssh/config` (HostName `100.90.116.4`, no ProxyCommand) | Just run `ssh cc-ci` (logs in as **root**). The orchestrator VM is a direct tailnet peer — direct route, no proxy. Pubkey already in cc-ci's `/root/.ssh/authorized_keys`. |
-| **Gitea bot account** | `/srv/cc-ci/.testenv` → `GITEA_USERNAME` (`autonomic-bot`), `GITEA_PASSWORD`, `GITEA_URL` (`git.autonomic.zone`) | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
+| **Gitea bot account** | `/srv/cc-ci/cc-ci/.env.public` → `GITEA_USERNAME` (`autonomic-bot`); `/srv/cc-ci/.testenv` → `GITEA_PASSWORD`; `GITEA_URL` defaults to `git.autonomic.zone` | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |

-Load them in a shell with: `set -a; . /srv/cc-ci/.testenv; set +a` (don't echo the values).
+Load runtime configuration in a shell with: `set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a`
+(don't echo secret values). The helper loads tracked public configuration first and then `.testenv`.

 ### The Tailscale connection (how `ssh cc-ci` works)

@@ -0,0 +1,234 @@
+#!/usr/bin/env python3
+"""pr-survey — deterministic facts about every open recipe PR, for /cc-ci-cleanup to judge.
+
+Open recipe PRs rot in specific, detectable ways. This gathers the evidence; it does NOT decide
+anything — closing a PR is a judgement the skill makes, with these facts in hand.
+
+RUN `reconcile-upstream.sh --all` FIRST. Every signal below is measured against the mirror's `main`,
+and an unreconciled mirror makes all of them wrong: on 2026-08-11 three PRs (discourse #6 carrying
+140 CVEs, keycloak #6 carrying 12, n8n #5) looked pending against a stale mirror while upstream had
+already merged them. This tool refuses to guess about that — see `reconciled_recently`.
+
+Per PR:
+  behind_main      commits on main not in the branch — the "out of date" measure
+  ahead            commits on the branch not on main
+  mergeable        gitea's own verdict (false = conflicts, needs a rebase)
+  diff_files       files the PR touches (0 = nothing left to merge)
+  adds_images      the `+ image:` lines it introduces
+  already_in_main  those `+ image:` lines ALREADY present in main -> the bump landed another way
+  obsolete         true when every image it adds is already in main (nothing to contribute)
+  ci               newest `!testme` verdict + build number parsed from the PR comments
+  branch_kind      upgrade / fix / ci-artifact (`ci/*` sweep + probe branches) / other
+  age_days, stale_days (since last update)
+
+    pr-survey.py [recipe ...] [--json]
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+from datetime import datetime, timezone
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+USED_RECIPES = os.path.join(HERE, "used-recipes.md")
+TESTENV = os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")
+PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
+NS = "recipe-maintainers"
+
+
+def _env() -> dict:
+    e = {}
+    for path in (PUBLIC_ENV, TESTENV):
+        try:
+            lines = open(path)
+        except OSError:
+            continue
+        for ln in lines:
+            ln = ln.strip()
+            if "=" in ln and not ln.startswith("#"):
+                k, v = ln.split("=", 1)
+                e[k] = v.strip().strip('"').strip("'")
+    return e
+
+
+ENV = _env()
+GITEA = os.environ.get("GITEA_URL") or ENV.get("GITEA_URL", "git.autonomic.zone")
+_AUTH = base64.b64encode(
+    f"{os.environ.get('GITEA_USERNAME') or ENV.get('GITEA_USERNAME','')}:"
+    f"{os.environ.get('GITEA_PASSWORD') or ENV.get('GITEA_PASSWORD','')}".encode()
+).decode()
+
+
+def _get(path: str, raw: bool = False):
+    req = urllib.request.Request(
+        f"https://{GITEA}{path}",
+        headers={"Authorization": f"Basic {_AUTH}", "User-Agent": "cc-ci-pr-survey"},
+    )
+    with urllib.request.urlopen(req, timeout=60) as r:
+        body = r.read()
+    return body.decode(errors="replace") if raw else json.loads(body)
+
+
+def _days(iso: str | None) -> int | None:
+    if not iso:
+        return None
+    try:
+        d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
+    except ValueError:
+        return None
+    return (datetime.now(timezone.utc) - d).days
+
+
+def _branch_kind(ref: str) -> str:
+    if ref.startswith("ci/"):
+        return "ci-artifact"          # regall/cfold sweeps + testme probes; never meant to merge
+    if ref.startswith("upgrade"):
+        return "upgrade"
+    if re.match(r"^(fix|feat|chore|revert)", ref):
+        return "fix"
+    return "other"
+
+
+def _main_images(recipe: str) -> set[str]:
+    """Image refs pinned on the mirror's main — the baseline a PR is judged against."""
+    out = set()
+    for f in ("compose.yml",):
+        try:
+            txt = _get(f"/{NS}/{recipe}/raw/branch/main/{f}", raw=True)
+        except Exception:  # noqa: BLE001
+            continue
+        for m in re.finditer(r"^\s*image:\s*[\"']?([^\"'\s]+)", txt, re.M):
+            out.add(m.group(1))
+    return out
+
+
+def _ci_verdict(recipe: str, number: int) -> dict:
+    """Newest cc-ci !testme outcome recorded on the PR."""
+    try:
+        cs = _get(f"/api/v1/repos/{NS}/{recipe}/issues/{number}/comments?limit=100")
+    except Exception:  # noqa: BLE001
+        return {"verdict": "unknown", "build": None}
+    for c in reversed(cs):
+        b = c.get("body") or ""
+        if "cc-ci:testme" not in b:
+            continue
+        m = re.search(r"/cc-ci/(\d+)", b)
+        if "✅" in b or "passed" in b:
+            return {"verdict": "passed", "build": m.group(1) if m else None}
+        if "❌" in b or "failure" in b:
+            return {"verdict": "failed", "build": m.group(1) if m else None}
+        if "⏳" in b or "in progress" in b:
+            return {"verdict": "running", "build": m.group(1) if m else None}
+    return {"verdict": "never-run", "build": None}
+
+
+def survey_pr(recipe: str, pr: dict, main_images: set[str]) -> dict:
+    n = pr["number"]
+    head = pr["head"]["ref"]
+    row = {
+        "recipe": recipe, "number": n, "title": pr.get("title", ""), "head": head,
+        "url": pr.get("html_url"), "branch_kind": _branch_kind(head),
+        "age_days": _days(pr.get("created_at")), "stale_days": _days(pr.get("updated_at")),
+        "mergeable": pr.get("mergeable"),
+    }
+    try:
+        row["behind_main"] = _get(
+            f"/api/v1/repos/{NS}/{recipe}/compare/{urllib.parse.quote(head, safe='')}...main"
+        ).get("total_commits", 0)
+        row["ahead"] = _get(
+            f"/api/v1/repos/{NS}/{recipe}/compare/main...{urllib.parse.quote(head, safe='')}"
+        ).get("total_commits", 0)
+    except Exception:  # noqa: BLE001
+        row["behind_main"], row["ahead"] = None, None
+    # A FAILED diff fetch must never look like an empty diff: gitea#4 404s on .diff (force-pushed
+    # branch) and would otherwise be flagged EMPTY-DIFF and closed — while being a verified, green,
+    # needed fix. Unknown is its own state.
+    diff = None
+    try:
+        body = _get(f"/{NS}/{recipe}/pulls/{n}.diff", raw=True)
+        if body.lstrip().startswith(("diff --git", "From ")) or not body.strip():
+            diff = body
+    except Exception:  # noqa: BLE001
+        diff = None
+    row["diff_files"] = None if diff is None else len(re.findall(r"^diff --git ", diff, re.M))
+    adds = re.findall(r"^\+\s*image:\s*[\"']?([^\"'\s]+)", diff or "", re.M)
+    row["adds_images"] = sorted(set(adds))
+    row["already_in_main"] = sorted({i for i in set(adds) if i in main_images})
+    # Nothing left to contribute: it touches files but every image it introduces is already pinned.
+    # Only claim obsolete when the diff was actually READ. No diff, no verdict.
+    row["obsolete"] = diff is not None and bool(adds) and set(adds).issubset(main_images)
+    row["ci"] = _ci_verdict(recipe, n)
+    return row
+
+
+def all_recipes() -> list[str]:
+    out = []
+    for ln in open(USED_RECIPES):
+        p = ln.split()
+        if len(p) >= 2 and not ln.startswith(("#", "`")) and p[1] in ("weekly", "external"):
+            out.append(p[0])
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("recipes", nargs="*")
+    ap.add_argument("--json", action="store_true")
+    a = ap.parse_args()
+
+    rows = []
+    for r in (a.recipes or all_recipes()):
+        try:
+            prs = _get(f"/api/v1/repos/{NS}/{r}/pulls?state=open&limit=50")
+        except urllib.error.HTTPError as e:
+            rows.append({"recipe": r, "error": f"HTTP {e.code}"})
+            continue
+        if not prs:
+            continue
+        mi = _main_images(r)
+        for pr in prs:
+            rows.append(survey_pr(r, pr, mi))
+
+    if a.json:
+        print(json.dumps(rows, indent=2))
+        return 0
+
+    print(f"{len(rows)} open PR(s)\n")
+    for x in sorted(rows, key=lambda z: (z.get("recipe", ""), z.get("number", 0))):
+        if x.get("error"):
+            print(f"  {x['recipe']}: {x['error']}")
+            continue
+        flags = []
+        if x["obsolete"]:
+            flags.append("OBSOLETE(images already in main)")
+        if x["branch_kind"] == "ci-artifact":
+            flags.append("CI-ARTIFACT")
+        if x["diff_files"] == 0:
+            flags.append("EMPTY-DIFF")
+        if x["diff_files"] is None:
+            flags.append("DIFF-UNREADABLE(do not close on this)")
+        if x["mergeable"] is False:
+            flags.append("CONFLICTS")
+        if (x["behind_main"] or 0) > 0:
+            flags.append(f"BEHIND-{x['behind_main']}")
+        print(f"  {x['recipe']}#{x['number']:<3} {x['title'][:52]}")
+        print(f"      {x['branch_kind']:12} age={x['age_days']}d idle={x['stale_days']}d "
+              f"ci={x['ci']['verdict']}({x['ci']['build'] or '-'}) files={x['diff_files'] if x['diff_files'] is not None else '?'}")
+        if x["adds_images"]:
+            print(f"      adds: {', '.join(i.split('/')[-1] for i in x['adds_images'][:4])}")
+        if flags:
+            print(f"      >> {' | '.join(flags)}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -46,6 +46,7 @@ from datetime import datetime, timezone

 LOGDIR = "/srv/cc-ci/.cc-ci-logs"
 TESTENV = "/srv/cc-ci/.testenv"
+PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
 INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
 HOST_REPORTS = "/var/lib/cc-ci-reports"
 # Both kinds live in ONE archive, distinguished by a suffix on a common title.
@@ -56,14 +57,16 @@ KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),

 def _env():
    e = {}
-    try:
-        for ln in open(TESTENV):
+    for path in (PUBLIC_ENV, TESTENV):
+        try:
+            lines = open(path)
+        except FileNotFoundError:
+            continue
+        for ln in lines:
            ln = ln.strip()
            if "=" in ln and not ln.startswith("#"):
                k, v = ln.split("=", 1)
                e[k] = v.strip().strip('"').strip("'")
-    except FileNotFoundError:
-        pass
    return e


@@ -30,13 +30,12 @@ set -o errexit -o nounset -o pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ORCH="$(dirname "$HERE")"
 SSH="${SSH:-cc-ci}"
-TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
 RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
 USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"

 [ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
-set -a; . "$TESTENV"; set +a
-: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
+set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
+: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"

 if [ "${1:-}" = "--all" ]; then
  mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
@@ -537,6 +537,161 @@ class TestReleaseLineSemantics(unittest.TestCase):
        self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"])


+class TestAlreadyFixedOnFromLine(unittest.TestCase):
+    """A fix that landed on the line we upgrade FROM was already ours before the upgrade."""
+
+    def test_backport_to_our_own_line_is_not_credited(self):
+        # mattermost patches every maintained line at once. 10.11.22 -> 10.12.4 crosses 10.12.1, but
+        # 10.11.22 is already past 10.11.4, so the deployment HAD the fix. Counting it credits the
+        # upgrade with work it did not do.
+        rep = run_scan([gh("mattermost/mattermost",
+                           [adv("CVE-1", patched="10.11.4; 10.12.1; 10.5.12")])],
+                       v_from="10.11.22", v_to="10.12.4",
+                       urls=["https://github.com/mattermost/mattermost"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], [])
+
+    def test_a_fix_ABOVE_our_position_on_the_same_line_still_counts(self):
+        rep = run_scan([gh("mattermost/mattermost", [adv("CVE-2", patched="10.11.30; 10.12.1")])],
+                       v_from="10.11.22", v_to="10.12.4",
+                       urls=["https://github.com/mattermost/mattermost"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2"])
+
+    def test_placeholders_never_feed_this_rule(self):
+        # "7.4.X" parses to a bare 7.4, which would read as "already fixed at 7.4" and silently drop
+        # a real fix — this is exactly how redis CVE-2024-46981 was lost when the rule was added.
+        rep = run_scan([gh("redis/redis", [adv("CVE-3", patched="6.2.X, 7.2.X, 7.4.X")])],
+                       v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
+        self.assertIn("CVE-3", rep["indeterminate"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], [])
+
+
+class TestChangelogAttribution(unittest.TestCase):
+    """Projects that publish no advisory feed still say which release fixed what — in their changelog."""
+
+    CHANGES = """
+Changes with nginx 1.31.3        11 Aug 2026
+    *) Security: a flaw ... (CVE-2026-60005)
+    *) Security: another ... (CVE-2026-56434)
+
+Changes with nginx 1.31.2        04 Aug 2026
+    *) Security: something ... (CVE-2026-48142)
+
+Changes with nginx 1.31.1        21 Jul 2026
+    *) Security: older ... (CVE-2026-9256)
+
+Changes with nginx 1.20.0        01 Jan 2021
+    *) Security: ancient ... (CVE-2013-2028)
+"""
+
+    def test_each_cve_is_attributed_to_the_release_that_fixed_it(self):
+        got = A._changelog_versions(self.CHANGES)
+        self.assertEqual(got["CVE-2026-60005"], "1.31.3")
+        self.assertEqual(got["CVE-2026-48142"], "1.31.2")
+        self.assertEqual(got["CVE-2026-9256"], "1.31.1")
+        self.assertEqual(got["CVE-2013-2028"], "1.20.0")
+
+    def _scan(self, wfrom, wto):
+        # nginx publishes NO GitHub advisories — the feed is empty and the changelog is everything.
+        return run_scan(
+            [gh("nginx/nginx", [])],
+            [{"source": "https://nginx.org/en/CHANGES", "status": "ok",
+              "cves": sorted(A._changelog_versions(self.CHANGES)),
+              "context": {}, "fixed_in": A._changelog_versions(self.CHANGES)}],
+            images=[("nginx", wfrom, wto)], urls=["https://github.com/nginx/nginx"])
+
+    def test_window_counts_only_the_releases_it_crosses(self):
+        rep = self._scan("1.31.1", "1.31.3")   # 1.31.1 is the FROM, so its CVE is already fixed
+        self.assertEqual(set(rep["fixed_by_this_upgrade"]),
+                         {"CVE-2026-48142", "CVE-2026-56434", "CVE-2026-60005"})
+
+    def test_a_narrower_window_counts_fewer(self):
+        rep = self._scan("1.31.2", "1.31.3")
+        self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-56434", "CVE-2026-60005"})
+
+    def test_ancient_entries_are_not_swept_in(self):
+        # The changelog lists the project's whole history; only the crossed releases may count.
+        rep = self._scan("1.31.1", "1.31.3")
+        self.assertNotIn("CVE-2013-2028", rep["fixed_by_this_upgrade"])
+
+    def test_evidence_is_recorded(self):
+        rep = self._scan("1.31.1", "1.31.3")
+        self.assertEqual(rep["resolved_by_changelog"]["CVE-2026-60005"], "1.31.3")
+
+
+class TestComposeDerivedWindows(unittest.TestCase):
+    """Windows read off a compose diff, so nobody has to remember which --image args an upgrade needs."""
+
+    OLD = """
+services:
+  app:
+    image: "plausible/analytics:v2.0.0"
+  db:
+    image: pgautoupgrade/pgautoupgrade:18-alpine
+  plausible_events_db:
+    image: clickhouse/clickhouse-server:23.4.2.11-alpine
+volumes:
+  data:
+"""
+    NEW = """
+services:
+  app:
+    image: "ghcr.io/plausible/community-edition:v3.2.1"
+  db:
+    image: pgautoupgrade/pgautoupgrade:18-alpine
+  plausible_events_db:
+    image: clickhouse/clickhouse-server:24.12-alpine
+volumes:
+  data:
+"""
+
+    def _windows(self, old=None, new=None):
+        pages = {"to": new if new is not None else self.NEW,
+                 "from": old if old is not None else self.OLD}
+        with unittest.mock.patch.object(A, "_fetch", lambda u, h=None: pages["to" if "to" in u else "from"]), \
+             unittest.mock.patch.object(A, "_gitea_auth", lambda u: {}):
+            return A.windows_from_compose("http://x/to", "http://x/from")[0]
+
+    def test_app_service_leads_and_sidecars_follow(self):
+        w = self._windows()
+        self.assertEqual(w[0], ("community-edition", "v2.0.0", "v3.2.1"))
+        self.assertIn(("clickhouse-server", "23.4.2.11-alpine", "24.12-alpine"), w)
+
+    def test_unchanged_images_are_not_windows(self):
+        # pgautoupgrade is identical in both; inventing a window for it would be a false count.
+        self.assertNotIn("pgautoupgrade", [n for n, _, _ in self._windows()])
+
+    def test_a_changed_image_REPO_is_still_the_same_service(self):
+        # plausible/analytics -> ghcr.io/plausible/community-edition. Keyed by image repo this reads
+        # as one image vanishing and another appearing, and the app window is lost entirely.
+        w = self._windows()
+        self.assertTrue(any(n == "community-edition" and f == "v2.0.0" for n, f, _ in w))
+
+    def test_no_change_yields_no_windows(self):
+        self.assertEqual(self._windows(old=self.NEW, new=self.NEW), [])
+
+    def test_templated_tags_are_skipped(self):
+        new = self.NEW.replace('ghcr.io/plausible/community-edition:v3.2.1', 'ghost:${IMAGE_VERSION}')
+        self.assertNotIn("ghost", [n for n, _, _ in self._windows(new=new)])
+
+
+class TestImageNameMatching(unittest.TestCase):
+    """An image name and its advisory source rarely spell each other exactly."""
+
+    def test_matches_when_the_image_name_is_LONGER_than_the_source(self):
+        # clickhouse/clickhouse-server vs source ClickHouse/ClickHouse — one-directional matching
+        # dropped this window silently when the key came from a compose file.
+        rep = run_scan([gh("ClickHouse/ClickHouse", [adv("CVE-1", patched="23.10.2.13")])],
+                       images=[("clickhouse-server", "23.4.2.11", "24.12")],
+                       urls=["https://github.com/ClickHouse/ClickHouse"])
+        self.assertIn("github-advisories:ClickHouse/ClickHouse", rep["windows"])
+        self.assertEqual(rep["cve_count_fixed"], 1)
+
+    def test_matches_when_the_image_name_is_SHORTER_than_the_source(self):
+        rep = run_scan([gh("redis/redis", [adv("CVE-2", patched="7.4.1")])],
+                       images=[("redis", "7.4", "8.10")], urls=["https://github.com/redis/redis"])
+        self.assertEqual(rep["cve_count_fixed"], 1)
+
+
 class TestAdjudicationEvidenceAssembly(unittest.TestCase):
    """Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""

@@ -18,6 +18,11 @@
 - AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed.
 - Minio tag uses a date-based RELEASE.YYYY-MM-DDTHH-MM-SSZ format — abra cannot parse it for upgrades;
  check manually on https://github.com/minio/minio/releases.
+- **2026-08-14: Minio stopped publishing Docker images after RELEASE.2025-09-07T16-13-09Z.**
+  GitHub has a newer release (`RELEASE.2025-10-15T17-29-55Z`, published 2025-10-16, with CVE fix
+  GHSA-jjjj-jwhf-8rgr), but the Docker image was never pushed to Docker Hub (returns 404; release
+  notes say "clone the source and build the latest container"). quay.io checked — only 2022-era
+  tags. As of this date, `RELEASE.2025-09-07T16-13-09Z` IS the newest available Docker image.
 - v5.2.0 adds two optional new env vars: DOCUMENT_ALL_ENDPOINT_ENABLED and OIDC_OP_USER_ENDPOINT_FORMAT.
  Both are backward-compatible (no action required for existing deployments).
 - Recipe version label convention: 0.X.Y+vA.B.C where A.B.C is the impress version.
@@ -6,6 +6,14 @@
 | postgres | postgres                                  | https://github.com/postgres/postgres              | https://www.postgresql.org/docs/release/                          |

 ## Standing notes
+  - **2026-08-28 re-check** (endoflife.date/api/mattermost.json 2026-08-28; Docker Hub; GitHub
+    releases): **11.7.10** (released 2026-08-26, "various bug fixes", not prerelease) is the newest
+    11.7.x ESR patch — the ESR/LTS line has NOT moved, still **11.7** (EOL **2027-05-15**). This run
+    bumps open PR #2 **11.7.9 → 11.7.10** (ESR patch within the same line). 11.8/11.9/11.10 remain
+    innovation releases (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.12 expired.
+    postgres **15-alpine HELD** (DB-major 16/17/18 out of scope, plain-postgres recipe w/o
+    pgautoupgrade: operator dump/restore needed). git.autonomic.zone mirror main reconciled to
+    coopcloud (86e0d0b5 = 10.11.22, upstream still pins the EXPIRED 10.11 ESR).
 - mattermost-lts tracks the **11.7 ESR (Extended Support Release)** = Mattermost's LTS line. The "LTS"
  concept in Mattermost is **ESR**; there is NO "10.x LTS line". Release calendar (endoflife.date/mattermost):
  - **10.11** = previous ESR, security support **ends 2026-08-15** (~7 weeks) — migrated away 2026-06-19
@@ -75,3 +83,32 @@
  recreate DB, reimport dump. `DROP DATABASE WITH (FORCE)` requires PostgreSQL 13+ — safe on postgres:15-alpine.
  The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
  was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
+
+  - **2026-08-21 re-check** (endoflife.date/api/mattermost.json 2026-08-17; Mattermost docs; GitHub):
+    **11.7 ESR is the only current ESR/LTS** — latest patch **11.7.9** (2026-08-13, "Low to High
+    security fixes"), EOL 2027-05-15. **10.11 ESR ENDED 2026-08-15** — upstream main still pins
+    10.11.22 (now on an EXPIRED ESR). 11.10/11.9/11.8 are innovation (EOL 2026-11-15/10-15/09-15),
+    NOT LTS. The `-lts` recipe must track 11.7. This run **redirects open PR #2 away from the
+    innovation 11.10.0 onto 11.7.9** (the correct LTS/ESR; ESR→ESR 10.11→11.7 is "fully supported
+    and tested" per Mattermost docs — no staging hop). Refer to the 2026-08-15 report where PR #2
+    was on 11.10.0 (operator chose innovation line that week); operator should now decide the
+    `-lts` line. postgres 15-alpine still HELD (DB-major out of scope).
+
+- **2026-08-31 re-check** (endoflife.date/api/mattermost.json 2026-08-31; Docker Hub tag list;
+    GitHub release `v11.7.10`): **11.7 ESR line UNCHANGED** — latest patch still **11.7.10**
+    (released 2026-08-26, "various bug fixes"), EOL 2027-05-15. PR #2 already carries 11.7.9 →
+    11.7.10; the wildcard cert is RENEWED (valid to 2026-11-29), so this run just RE-VERIFIES PR #2
+    via `!testme` (the 2026-08-28 run's `!testme` was infra-blocked). 11.8.5 / 11.9.1 / 11.10.1
+    remain innovation (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.11 ESR ended
+    2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries
+    remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine
+    still HELD (DB-major out of scope, operator dump/pg_upgrade).
+
+## NVD CPE fallback
+This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
+no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
+CPE-indexed and carries structured version ranges, so it can answer where the vendor
+cannot. It LAGS the vendor, so it is a fallback, never the primary source.
+
+- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
+- nvd-cpe: postgres = cpe:2.3:a:postgresql:postgresql:*:*:*:*:*:*:*:*
@@ -19,3 +19,11 @@
 - The server image tag is `v<version>-<build>` (e.g. `v1.6.870-4`); the trailing number is the image
  build, not an app version, and moves independently of upstream releases — `abra recipe upgrade`
  reports "no new versions" for it, so use `resolve-images.py` to see those bumps.
+
+## NVD CPE fallback
+This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
+no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
+CPE-indexed and carries structured version ranges, so it can answer where the vendor
+cannot. It LAGS the vendor, so it is a fallback, never the primary source.
+
+- nvd-cpe: mumble-server = cpe:2.3:a:mumble:mumble:*:*:*:*:*:*:*:*
@@ -50,3 +50,90 @@
 - 2026-08-07 run: operator directed 2.33.3 -> 2.34.2 (the newest). The whole 2.34.x line is still
  marked Pre-release on GitHub (2.33.5 holds the Latest badge); flagged in the PR body. No breaking
  changes across 2.33.3 -> 2.34.2; rolling upgrade safe (TypeORM migrations auto-run on boot).
+- 2.35.0 (2026-08-11, Pre-release): major feature release — self-hosted AI Assistant onboarding,
+  Simplified Custom Auth credentials, Agent Builder test runs + HITL, Discord agent chat channel,
+  local agent token counting, **VM expression engine now the default** (was opt-in), MCP SDK v2
+  migration + MCP 2026-07-28 discovery handshake, Kafka Node v2, Salesforce OAuth2 JWT, GitHub
+  dispatch timeout, X/Twitter Node OAuth2/API migrated to x.com, Azure Key Vault configurable
+  endpoints, Postgres-version startup warning, workflow review improvements (diffs, metadata, version
+  descriptions), and numerous core/editor bugfixes. No breaking compose/config/migration changes.
+- 2.35.1 (2026-08-12, Pre-release): 2 core bugfixes — data-tables resume scope, TLS options per hop
+  through a proxy.
+- 2.35.2 (2026-08-13, Pre-release): 1 core bugfix — report real activation mode for triggers via
+  publication outbox. **Deployed on cc-ci 2026-08-15**: 2.34.4→2.35.2, TypeORM migrations clean,
+  editor served HTTP 200. No breaking changes, no N8N_* env renames, no required operator action.
+- 2.35.3 (2026-08-14, Pre-release): bugfixes (Google Ads v21→v25 API migration, MS Teams OAuth scope
+  restore, workflow publication outbox abort deadline) + feature (skip update approval for workflows
+  from same Instance AI session). Not deployed (2.35.2 was the survey target).
+- 2.36.0 (2026-08-18, Pre-release; 2.36.1 was briefly tagged then withdrawn — 2.36.2 compares
+  2.36.0...2.36.2): a large feature minor. Core/ai-builder/editor bugfixes + new features (Agent
+  sandboxes + writable workspaces, MCP folder/move/read-data-table tools, Schedule Trigger missed-
+  execution grace period + catch-up coalescing, Confluence Cloud OAuth2 + page get sub-tree, MiniMax
+  M3/H3, LangSmith session debug export, role-mapping/roles API endpoints, n8n User Auth GA, etc.).
+  No breaking compose/env/migration changes, no `N8N_*` env renames; the deprecation notices
+  (Array.merge→Array.mergeIntoObject in expressions, workflow tags API migration) are API-level only.
+  2.36.2 (2026-08-19, patch): 3 fixes (decorator-route request schema in /discover, end-user
+  credential resolution in node parameters, editor agent-preview alignment). 2.36.3 (2026-08-20,
+  patch): 4 core/editor fixes (task-runners only restart when slow, expression engine init on
+  expression commands, test-webhook isolate release after teardown, Edit Fields type selector styles).
+- 2026-08-21 run: planning 2.34.4 → 2.36.3 (newest tag abra lists). Whole 2.36.x line still
+  Pre-release on GitHub (2.33.5/Latest was superseded by 2.35.x line). Rolling upgrade safe, sqlite
+  mode, TypeORM migrations auto-run on boot. No operator action required.
+- 2.36.4 (2026-08-21, Pre-release): 3 core fixes (env normalization before schema parsing; chat and
+  MCP trigger auth-mode validation for end-user credentials; wrap trigger closeFunction in an
+  expression isolate).
+- 2.36.5 (2026-08-21, Pre-release): 1 core fix (raise AI Assistant model verification token limit).
+- 2.36.6 (2026-08-24): 1 editor fix (personalization survey / community registration modal on
+  Instance AI landing page).
+- 2.36.7 (2026-08-25, **current Stable / Latest badge**; `stable` tag points here): API fix (adapt
+  binary data endpoint), API cross-backport (require a JSON content type on decorator routes that
+  take a body), core fixes (reconcile license auto-renewal on startup leader takeover; route OAuth2
+  token requests through environment proxies), feature (remove offset query param from list workflow
+  history endpoint). This is the newest stable-line release; the 2.37.x line is still pre-release.
+- 2.37.0 (2026-08-25, Pre-release): a large feature minor. Agent Builder / AIA / core/editor
+  bugfixes + features (Slack Agent view, prompt caching for Anthropic, SharePoint node v2 = default,
+  form-trigger OAuth2 (added then reverted), role-mapping rule CRUD endpoints, execution
+  startedAfter/startedBefore filters, Agent/community-package hardening: serialize community
+  package installs + non-destructive downloads + cold-boot storm reduction, MCP tool schemas as JSON
+  Schema 2020-12, distroless n8n image variant). **Deprecation: the "Any workflow" caller policy**
+  (API/config-level — flag for operators using execution-caller policies). Two enforced API behavior
+  changes: **require JSON content type on decorator body routes** and **"adapt binary data endpoint"**
+  (HTTP clients sending a body to those routes must send `Content-Type: application/json`). No
+  breaking compose/env/migration changes, no `N8N_*` env renames.
+- 2.37.1 (2026-08-25, Pre-release): 1 core fix (route OAuth2 token requests through env proxies in
+  every process).
+- 2.37.3 (2026-08-27, Pre-release; 2.37.2 was withdrawn): 2 runner-image fixes (copy glibc into
+  distroless runners image as a matched pair; stage libatomic into the distroless runners image) —
+  relevant if using n8n's community/distroless runner image.
+- 2026-08-28 run: PR #7 extended 2.34.4 → **2.37.3** (newest tag abra lists; 2.37.2 withdrawn, that
+  tag does not exist). 2.36.7 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub,
+  consistent with the tracking-the-newest-tag precedent (2.34.2/2.35.2/2.36.3 were pre-release when
+  taken). Rolling upgrade safe (sqlite, TypeORM auto-migrate on boot). Operator flag: the two 2.37.0
+  API behavior changes (JSON content-type on decorator body routes; binary-data endpoint adapt) only
+  affect HTTP API callers. Recommended release: `-y` (feature minor).
+  NOTE: PR #7's `!testme` on this date was BLOCKED — `*.ci.commoninternet.net` wildcard cert expired
+  (bridge: `CERTIFICATE_VERIFY_FAILED certificate has expired`). UNVERIFIED; operator renewed the cert
+  (now valid to 2026-11-29). The 2.37.3 work must be re-verified by a fresh `!testme`.
+- 2.34.5 (2026-08-12, patch): 1 core fix (apply TLS options per hop through a proxy).
+- 2.34.6 (2026-08-14, patch): 3 fixes (core `continueErrorOutput` `details` field; Google Ads sunset
+  v21→v25 API migration; MS Teams Restore `Group.ReadWrite.All` OAuth2 scope — matches the 2.35.3
+  in-progress backports).
+- 2.35.4/2.35.5/2.35.6/2.35.7 (2026-08-19..21, patches): API schema for decorator routes in /discover;
+  Google Ads v25 view metric rename; task-runners not restarted when only slow; expression engine
+  init on expression commands; test-webhook isolate release after teardown; env normalization before
+  schema parsing; end-user credential resolution; trigger closeFunction isolate; AI Assistant token
+  limit raise. All patch bugfixes, no breaking changes.
+- 2.36.8 (2026-08-28, patch): 1 core fix (domain-restricted credential usable in its own node) —
+  cross-backport of the 2.37.4 fix into the stable line.
+- 2.36.9 (2026-08-31, patch; **now the Stable/Latest badge** — `stable` tag): 1 core fix (apply proxy
+  environment variables consistently across packages and processes, cross-backport of 2.37.1/2.37.3).
+- 2.37.4 (2026-08-28, Pre-release): 1 core fix (allow a domain-restricted credential to work in its
+  own node).
+- 2.37.5 (2026-08-31): **withdrawn** — no plain `2.37.5` tag on Docker Hub (only partial
+  `2.37.5-amd64/-arm64/-pc` artifact tags, no manifest). Skip entirely, like 2.37.2/2.36.1.
+- 2.37.6 (2026-08-31, Pre-release; **newest 2.37.x tag**): 1 core fix (thread execution id through
+  dynamic-credential storage).
+- 2026-08-31 run: PR #7 extended 2.34.4 → **2.37.6** (newest tag abra lists = 2.37.6/2.37.4/2.37.3/…;
+  2.37.5 withdrawn). 2.36.9 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub
+  (consistent precedent). Re-verified 2.37.3→2.37.6 (pure core bugfixes), no breaking changes beyond
+  the already-flagged 2.37.0 API behavior pair. Rolling upgrade safe. Recommended release: `-y`.
@@ -4,6 +4,7 @@
 |---------|-------|-------------|----------------------|
 | app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
 | db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
+| ftp (optional overlay) | atmoz/sftp | https://github.com/atmoz/sftp | https://github.com/atmoz/sftp/releases (image: https://hub.docker.com/r/atmoz/sftp) |

 ## Standing notes
 - Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
@@ -22,3 +23,7 @@
  image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
  recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
  bumps in place.
+- `atmoz/sftp` is pinned as the floating `alpine` tag (updated 2026-07-28, current as of
+  2026-08-21) — not a semver abra can evaluate, so it never contributes an upgrade candidate;
+  verified current by direct Docker Hub tag check. Only bump if a real newer supported tag
+  exists and the app needs it.
@@ -31,7 +31,7 @@ matrix-synapse    weekly
 mattermost-lts    weekly
 mumble            weekly
 n8n               weekly
-plausible         weekly
+plausible         external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 uptime-kuma       external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 wordpress         weekly
 ```
@@ -1,5 +1,28 @@
 {
  "nodes": {
+    "cc-ci": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs"
+        ],
+        "sops-nix": [
+          "sops-nix"
+        ]
+      },
+      "locked": {
+        "lastModified": 1788812004,
+        "narHash": "sha256-Vc7RSeqFHCwlVRhnEEjavuJoyIID+RQdJSygHNA8s8Y=",
+        "ref": "refs/heads/main",
+        "rev": "f6dbfa368995f4d45de09f4052631fd433c87d5b",
+        "revCount": 1533,
+        "type": "git",
+        "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
+      }
+    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1785734586,
@@ -18,6 +41,7 @@
    },
    "root": {
      "inputs": {
+        "cc-ci": "cc-ci",
        "nixpkgs": "nixpkgs",
        "sops-nix": "sops-nix"
      }
@@ -1,32 +1,52 @@
 {
-  description = "cc-ci-orchestrator — NixOS host for the cc-ci loops runtime (Builder/Adversary/Watchdog)";
+  description = "cc-ci-orchestrator — the cc-ci orchestrator (loops, steering session, weekly upgrader) and the NixOS host it shares with the cc-ci CI server";

  inputs = {
-    # Follow the current stable release channel (operator 2026-08-01), was a hard rev pin at
-    # nixpkgs 24.11 (50ab7937, 2025-06-30) kept "the same as the cc-ci server". This host runs
-    # agents/tmux/nginx/docker, not recipe CI, so it does not need to match that server — and a
-    # frozen rev only accrues unpatched CVEs. `nix flake update` now actually moves.
+    # Stable release channel (operator 2026-08-01). `nix flake update` moves it; the cc-ci input
+    # below FOLLOWS it, so one nixpkgs builds the whole combined host and CVEs get patched once.
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

-    # sops-nix follows nixpkgs below, so it no longer needs its own matching pin.
    sops-nix.url = "github:Mic92/sops-nix";
    sops-nix.inputs.nixpkgs.follows = "nixpkgs";
+
+    # The cc-ci CI server, as a NixOS module (`nixosModules.cc-ci-server`). HTTPS, anonymous read:
+    # nix evaluates every input for every output, so the input must be fetchable without
+    # credentials. The private secrets submodule is deliberately NOT fetched through this input —
+    # the host reads the deployed --recursive checkout's secrets.yaml at activation instead
+    # (`cc-ci.sopsFile`). Both `follows` are REQUIRED: without them cc-ci's own nixpkgs/sops-nix
+    # pins would produce a second sops-nix module tree and a second nixpkgs in one system.
+    cc-ci.url = "git+https://git.autonomic.zone/recipe-maintainers/cc-ci.git";
+    cc-ci.inputs.nixpkgs.follows = "nixpkgs";
+    cc-ci.inputs.sops-nix.follows = "sops-nix";
  };

-  outputs = { nixpkgs, sops-nix, ... }:
+  outputs = { self, nixpkgs, sops-nix, cc-ci, ... }:
    let
      system = "x86_64-linux";
    in
    {
-      # Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
-      # Provision with terraform/ then run Stage 2 per terraform/README.md.
-      nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {
-        inherit system;
-        modules = [
-          sops-nix.nixosModules.sops
-          ./nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix
-          ./nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix
-        ];
+      nixosModules = {
+        # The orchestrator itself: loops supervisor, steering session, weekly/hourly timers.
+        cc-ci-orchestrator = ./nix/modules/cc-ci.nix;
+        # The host contract those units assume: loops user, claude/opencode CLIs, opencode web
+        # server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config.
+        orchestrator-host = ./nix/modules/orchestrator-host.nix;
+        # Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09).
+        cc-ci = ./nix/modules/cc-ci.nix;
+      };
+
+      nixosConfigurations = {
+        # THE live host: cc-ci CI server + cc-ci orchestrator on one Hetzner cpx32-class box
+        # (195.201.88.249, since 2026-09). README.md is the deploy guide.
+        cc-ci = nixpkgs.lib.nixosSystem {
+          inherit system;
+          modules = [
+            cc-ci.nixosModules.cc-ci-server
+            self.nixosModules.cc-ci-orchestrator
+            self.nixosModules.orchestrator-host
+            ./nix/hosts/cc-ci/configuration.nix
+          ];
+        };
      };
    };
 }
@@ -10,9 +10,10 @@ metadata:
 The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx22** as of
 2026-05-31, replacing the Incus VM (100.116.55.106).

- Hetzner server **134487234**, public **168.119.126.100**, tailnet **cc-ci-orchestrator-1** @
-  **100.84.190.30**. Flake host **cc-ci-orchestrator-hetzner**.
- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch`
+- Since 2026-09-07: ONE Hetzner host for CI server + orchestrator, public **195.201.88.249**,
+  tailnet **cc-ci**, flake host **`.#cc-ci`** (this repo). Before: orchestrator on Hetzner
+  134487234 (168.119.126.100 / 100.84.190.30, `cc-ci-orchestrator-hetzner`), shared with notplants.
+- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch`
  (`/srv/cc-ci` is a symlink to it). The Bash tool runs as user **loops** (uid 1000, passwordless
  sudo) — plain `nixos-rebuild switch` fails on the profile symlink; use `sudo`.
 - Reboot-resilience: `cc-ci-loops.service` is **enabled** (wantedBy multi-user.target); ExecStartPre
@@ -23,4 +24,4 @@ The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx
  identity unknown". Set per-repo to match prior commits: `autonomic-bot
  <autonomic-bot@git.autonomic.zone>`.

-Full record: `cc-ci-plan/plan-orchestrator-hetzner-migration.md`.
+Full record: `archive/plans/plan-orchestrator-hetzner-migration.md`.
@@ -0,0 +1,68 @@
+# cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
+#
+# This file is only what is physical or identity about the machine: hardware, networking, the
+# tailscale node, root SSH keys, swap, stateVersion. Everything functional comes from modules:
+#   cc-ci.nixosModules.cc-ci-server        recipe-maintainers/cc-ci — swarm, traefik, drone,
+#                                          runner, bridge, dashboard, reports, acme-dns, harness
+#   self.nixosModules.cc-ci-orchestrator   nix/modules/cc-ci.nix — loops, orchestrator, timers
+#   self.nixosModules.orchestrator-host    nix/modules/orchestrator-host.nix — loops user, CLIs
+# See README.md for provisioning (Hetzner Debian → nixos-infect → this flake) and staging.
+{ lib, pkgs, ... }:
+{
+  imports = [
+    ./hardware.nix
+    ./networking.nix
+  ];
+
+  networking.hostName = "cc-ci";
+
+  # ---- cc-ci server identity --------------------------------------------------------------
+  # Public address: acme-dns binds to it and publishes it as the `ns-acme` glue record; the
+  # Gandi A records for ci / *.ci / ns-acme .commoninternet.net point here.
+  cc-ci.publicIPv4 = "195.201.88.249";
+  # cc-ci is a plain flake input here (no private submodule), so the sops file is the one in
+  # the deployed --recursive checkout the weekly sweep runs from (README "Stage the workspace").
+  cc-ci.sopsFile = "/etc/cc-ci/secrets/secrets.yaml";
+
+  # ---- orchestrator identity --------------------------------------------------------------
+  # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
+  cc-ci-orchestrator.ciSshHost = "127.0.0.1";
+
+  # ---- tailscale — auth key staged out of band at /etc/ts-auth-key -----------------------
+  services.tailscale = {
+    enable = true;
+    authKeyFile = "/etc/ts-auth-key";
+    extraUpFlags = [ "--hostname=cc-ci" ];
+  };
+
+  # ---- ssh ----------------------------------------------------------------------------------
+  services.openssh = {
+    enable = true;
+    settings.PermitRootLogin = "yes";
+  };
+  # Root keys: PUBLIC keys, tracked deliberately in ./ssh-keys (one per line, blank lines ok).
+  users.users.root.openssh.authorizedKeys.keys =
+    builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+  # The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
+  users.users.loops.openssh.authorizedKeys.keys =
+    builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+
+  # ---- firewall -------------------------------------------------------------------------------
+  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. The tailscale
+  # interface is trusted, which is what makes the opencode UI on 8443 tailnet-only.
+  networking.firewall = {
+    enable = true;
+    trustedInterfaces = [ "tailscale0" ];
+    allowedTCPPorts = [ 22 ];
+  };
+  networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
+
+  # ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------
+  swapDevices = [ { device = "/swapfile"; size = 8192; } ];
+
+  # ssh client for root (the orchestrator's `ssh cc-ci` goes through the loops user's own config).
+  environment.systemPackages = [ pkgs.openssh ];
+
+  # Fresh NixOS 26.05 install (nixos-infect, 2026-09-07). Never change this on an existing host.
+  system.stateVersion = "26.05";
+}
@@ -0,0 +1,19 @@
+# Generated by nixos-infect on this machine (2026-09-07), captured verbatim per README §3.
+# The ESP UUID is specific to THIS server; a new server gets a new file.
+{ modulesPath, ... }:
+{
+  imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
+  boot.loader = {
+    efi.efiSysMountPoint = "/boot/efi";
+    grub = {
+      efiSupport = true;
+      efiInstallAsRemovable = true;
+      device = "nodev";
+    };
+  };
+  fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/E079-7D41"; fsType = "vfat"; };
+  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
+  boot.initrd.kernelModules = [ "nvme" ];
+  fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
+  
+}
@@ -0,0 +1,38 @@
+# Generated by nixos-infect on this machine (2026-09-07), captured per README §3, with ONE edit:
+# `defaultGateway` as an attrset WITH `interface = "eth0"`. The generated bare-string form leaves
+# NixOS ≥25.05 without a default route (the host boots and is unreachable) — see README §2.
+{ lib, ... }: {
+  # This file was populated at runtime with the networking
+  # details gathered from the active system.
+  networking = {
+    nameservers = [ "2a01:4ff:ff00::add:2"
+ "2a01:4ff:ff00::add:1"
+ "185.12.64.2"
+ ];
+    defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };
+    defaultGateway6 = {
+      address = "fe80::1";
+      interface = "eth0";
+    };
+    dhcpcd.enable = false;
+    usePredictableInterfaceNames = lib.mkForce false;
+    interfaces = {
+      eth0 = {
+        ipv4.addresses = [
+          { address="195.201.88.249"; prefixLength=32; }
+        ];
+        ipv6.addresses = [
+          { address="2a01:4f8:1c1c:a9b::1"; prefixLength=64; }
+{ address="fe80::2ff8:e3ea:bbb8:aa39"; prefixLength=64; }
+        ];
+        ipv4.routes = [ { address = "172.31.1.1"; prefixLength = 32; } ];
+        ipv6.routes = [ { address = "fe80::1"; prefixLength = 128; } ];
+      };
+      
+    };
+  };
+  services.udev.extraRules = ''
+    ATTR{address}=="92:00:09:d5:ec:0d", NAME="eth0"
+    
+  '';
+}
@@ -0,0 +1,10 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZGp/DQTFuD1GvsyTzCVBUTmoWqcb5T+Z7zZo5nYLXO
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhgo41nt8/L+Cr0PKd8jQK45mw/A+h041j6LQ8JWZisEVaQOzr6s9rxPL8VT5ML4P3/4bMblzdDiXWlJxymcb+yk5S5TnVrMavzHEDhWHwEvTRMe6xNTmsU6cmmhRw7PJqqQ+0GTlQalu3I4jkC0kTF7kuPwduUOgUuSpJqxvDTwYiXoyVnOQHAIygh+BmQvYUz0PBfQgIhgcbYmGZ++T0DnMzdGFzW2UB/iy5mymnpmbaZCgLy0w8AoDE+0YLtUc4gwTXc183nvqO1i7LQr+3jBYkv5ZthCCc52vXFHDSw9xZ5ohsOrBvoi5foRbqinmU5/t0aTK7SSrat7xXm/odIOyS+S7PJyeEcsXN6d5zdxbabAy5vLfodEaKGZd4rqQeDCxOTPAS/BlrBV/EV714n4E+fSOAllAuMBO4IibJM/gLJrh2Dql3co50QW9HEDeSC7iqp2lxRBDxvUs3rIEzy7o4HSN8chqBUK1bbBY6B17fuNHIpBAw4akRVVvPnVM= trav@trav480sweet
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6jrKj7iZUNRLBTZG0vZM1D/BXtARhhB4+GrvpyuqmPb9iw2ifT9YqRUwgyGrOW9U6nIAR9yFnfp9+FkyhEKWByqEBbe/zYKlGLRGjfsIdDdW29QQ3hvmqNyboCkXLxZGat93poYhnoomqicmGD/xST4s0OUhcK9E494lUmenlD9dcMZW1aKpJ+9O4Dq6A7nk2z1e4KFcZdrZDI2Hgg+gfEdsKZQqd/R3Mls/eVKpzhfv3Y8BiNoHssUChVf8IGESqTOBOR7Dk7FsU5Z2ZcnQ1coxY7VlBn4fPjTWmz/Ac0jLqgcpCLpNyQzFPDVMYZKYrPVoqBeKVhN5YnfwR5OVP8YsakT/obLwC43sx/esXfjhVGcsRoGpiLOfazzNw/eC8s6FlS8cesOubEM37a7F25z4UEG3d487oM7EjQ39gBCCj/KRgUimCKMWsm6yIas4OSctBWEAo/NhZp0gwulSRxleW6eJCNNwzOmWjdzYIVWoVP0EIeM95Tq8PVUN7gpc= aadil@t480
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMyHSi12R0+HCVBz7+d9fyOBnoJi8Nsj5D7vQ9UQO8a5
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQFuqUB2qNZSDNjDsjjhVA/WnnQNVAMmsUscW6OgMDN
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS notplants-orchestrator
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
@@ -0,0 +1,157 @@
+# cc-ci.nix — the cc-ci ORCHESTRATOR: the Builder/Adversary loops supervisor, the operator's
+# steering session, and the weekly-upgrade + hourly-supervisor timers. Nothing else.
+#
+# Exported from this repo's flake as `nixosModules.cc-ci-orchestrator` (and, for the host that
+# used to import it under the old name, `nixosModules.cc-ci`). Split out of the shared agent
+# host config on 2026-08-20; since 2026-09 it runs on the same Hetzner host as the CI server
+# itself (`#cc-ci` in flake.nix), next to recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
+#
+# All of it assumes the cc-ci workspaces exist on the host:
+#   /srv/cc-ci        the loops workspace (+ .cc-ci-logs, upgrader.env) — a symlink to
+#   /srv/cc-ci-orch   this repo (the orchestrator's own working dir), with cc-ci/ checked out
+# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
+# those are host concerns, provided by nix/modules/orchestrator-host.nix, not by this module.
+{ config, pkgs, lib, ... }:
+{
+  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
+  systemd.services.cc-ci-loops = {
+    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
+    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
+      # default (control-group) systemd kills every leftover process in the cgroup when the unit
+      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
+      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
+      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci/cc-ci";
+      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
+      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
+    };
+    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
+    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
+    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
+    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
+      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
+      export PATH="/home/loops/.local/bin:$PATH"
+      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
+        echo "workspace not staged yet — skipping loop start"
+    '';
+  };
+
+  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
+  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
+  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
+  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
+  # (backend/model chosen there — Claude Code under Remote Control since 2026-09-07; before that
+  # opencode/glm-5.2 attached to the shared opencode web server, opencode-web.service in
+  # orchestrator-host.nix, which the upgrader still uses). The harness watchdog (started by
+  # `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
+  # be killed just for idling). Added 2026-08-03 for reboot-resilience.
+  systemd.services.cc-ci-orchestrator = {
+    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
+      # must not tear down the (shared) tmux server and every agent session with it.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci-orch";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      proj="/srv/cc-ci-orch"
+      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
+        >> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
+      cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
+    '';
+  };
+
+  # Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
+  # verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
+  # phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
+  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
+  systemd.services.cc-ci-upgrade-all = {
+    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci";
+      # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
+      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
+      # Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
+      # (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
+      # (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
+      # here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
+      # so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
+      # the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
+      # rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
+      # timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
+      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
+    };
+    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
+    '';
+  };
+
+  systemd.timers.cc-ci-upgrade-all = {
+    description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      # 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
+      OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
+      Persistent = true;  # if the box was down at the scheduled time, run once on next boot
+    };
+  };
+
+  # Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
+  # log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
+  # disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
+  # CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
+  # (zero model tokens). Only when a run has stalled/died before completing does it launch a
+  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
+  systemd.services.cc-ci-upgrade-supervisor = {
+    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
+    after = [ "network-online.target" "tailscaled.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci";
+      # Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
+      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
+    '';
+  };
+
+  systemd.timers.cc-ci-upgrade-supervisor = {
+    description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      OnCalendar = "*-*-* *:07:00";  # every hour at :07 (offset from the weekly :00 fire)
+      Persistent = false;            # a missed hourly check is moot — the next hour re-checks
+    };
+  };
+}
@@ -0,0 +1,203 @@
+# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
+# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
+# the standalone claude/opencode CLIs, the shared opencode web server and its tailnet-only UI,
+# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
+#
+# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
+# `nixosModules.cc-ci-orchestrator`; the combined CI-server + orchestrator host (`#cc-ci`) also
+# imports recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
+#
+# History: until 2026-09 this lived (twice, drifting) in nix/hosts/cc-ci-orchestrator-hetzner/
+# configuration.nix here and in notplants-nix's hosts/notplants-orchestrator/configuration.nix,
+# the shared agent box that also ran lichen + project-orchestrator. The cc-ci half moved to its
+# own host; this file is that half.
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.cc-ci-orchestrator;
+in
+{
+  options.cc-ci-orchestrator = {
+    ciSshHost = lib.mkOption {
+      type = lib.types.str;
+      default = "127.0.0.1";
+      example = "100.95.31.88";
+      description = ''
+        Where `ssh cc-ci` (used by every skill and script that drives the CI server) connects to,
+        as root with ~loops/.ssh/cc-ci-root-ed25519. On the combined host the CI server IS this
+        machine, so the default is loopback; a standalone orchestrator points it at the CI
+        server's tailnet address.
+      '';
+    };
+
+    opencodeUiPort = lib.mkOption {
+      type = lib.types.port;
+      default = 8443;
+      description = ''
+        TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host
+        Traefik (docker swarm) owns 80/443. The port is not opened in the firewall, so it is
+        reachable only over the trusted tailscale interface.
+      '';
+    };
+
+    opencodeUiHost = lib.mkOption {
+      type = lib.types.str;
+      default = "oc.commoninternet.net";
+      description = "nginx server_name for the opencode web UI (self-signed, basic auth).";
+    };
+  };
+
+  config = {
+    # ---- the loops user -------------------------------------------------------------------
+    # claude sessions run as non-root (--dangerously-skip-permissions is refused for root).
+    users.users.loops = {
+      isNormalUser = true;
+      uid = 1000;                 # fixed: workspace files are rsynced between hosts by uid
+      home = "/home/loops";
+      shell = pkgs.bash;
+      extraGroups = [ "wheel" "docker" ];
+    };
+    security.sudo.wheelNeedsPassword = false;
+    security.sudo.extraRules = [{
+      users = [ "loops" ];
+      commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
+    }];
+
+    # /home/loops/.local/bin holds the standalone claude + opencode binaries; it must be first on
+    # every PATH (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
+    environment.variables.PATH = lib.mkForce
+      "/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
+
+    # ---- nix-ld: the standalone Claude Code / opencode CLIs are foreign dynamic ELF binaries ---
+    programs.nix-ld.enable = true;
+    programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
+
+    # ---- the toolbox every agent on this box gets ----------------------------------------
+    # Bar for adding something: an agent doing ordinary work would otherwise waste a turn
+    # discovering it is absent.
+    environment.systemPackages = with pkgs; [
+      git tmux python3 jq curl cacert
+      gnused gawk coreutils gnugrep findutils util-linux nettools openssh
+      age sops ssh-to-age
+      wget gnutar gzip unzip zip xz
+      ripgrep fd tree file less which
+      procps psmisc htop lsof strace ncdu
+      dnsutils socat netcat-gnu iproute2 iputils
+      openssl gnumake gcc pkg-config
+      yq-go diffutils patch rsync bubblewrap
+    ];
+
+    # ---- ssh config for the loops user: `ssh cc-ci` = the CI server (root) -----------------
+    # Written only if absent so a manual customisation survives rebuilds.
+    system.activationScripts.loopsSshConfig = ''
+      mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
+      if [ ! -f /home/loops/.ssh/config ]; then
+        cat > /home/loops/.ssh/config <<'SSHCFG'
+Host cc-ci
+    HostName ${cfg.ciSshHost}
+    User root
+    IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
+    IdentitiesOnly yes
+    StrictHostKeyChecking accept-new
+    ServerAliveInterval 30
+
+Host git.autonomic.zone
+    HostName git.autonomic.zone
+    Port 2222
+    User git
+    IdentityFile /home/loops/.ssh/autonomic-bot-gitea-ed25519
+    IdentitiesOnly yes
+
+Host tangled.org
+    IdentityFile /home/loops/.ssh/tangled-ed25519
+    IdentitiesOnly yes
+SSHCFG
+        chmod 600 /home/loops/.ssh/config
+        chown loops:users /home/loops/.ssh/config
+      fi
+    '';
+
+    # ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
+    systemd.services.claude-install = {
+      description = "Install Claude Code CLI for loops user (idempotent)";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
+      environment = { HOME = "/home/loops"; };
+      path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
+      script = ''
+        if [ ! -x "$HOME/.local/bin/claude" ]; then
+          echo "installing Claude Code CLI for loops user..."
+          curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
+        fi
+      '';
+    };
+
+    systemd.services.opencode-install = {
+      description = "Install opencode CLI for loops user (idempotent)";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
+      environment = { HOME = "/home/loops"; };
+      path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
+      script = ''
+        if [ ! -x "$HOME/.local/bin/opencode" ]; then
+          echo "installing opencode CLI for loops user..."
+          curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation"
+          # The installer puts the binary in ~/.opencode/bin; every unit here expects ~/.local/bin.
+          if [ -x "$HOME/.opencode/bin/opencode" ]; then
+            mkdir -p "$HOME/.local/bin" && ln -sfn "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode"
+          fi
+        fi
+      '';
+    };
+
+    # ---- opencode web server: one shared instance the opencode-backed agents attach to -------
+    # Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
+    systemd.services.opencode-web = {
+      description = "opencode web server for cc-ci agents";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" "tailscaled.service" "opencode-install.service" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = {
+        Type = "simple";
+        User = "loops"; Group = "users";
+        WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
+        EnvironmentFile = [ "-/srv/cc-ci/cc-ci/.env.public" "/srv/cc-ci/.testenv" ];
+        ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
+        ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
+        Restart = "on-failure";
+        RestartSec = "5s";
+      };
+      environment = {
+        HOME = "/home/loops";
+        PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
+      };
+      path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
+    };
+
+    # ---- tailnet-only nginx front door for the opencode UI -------------------------------
+    # Self-signed cert + basic auth, both created out of band (a store path would be world
+    # readable) — see README "Secrets to stage". nginx FAILS TO START if they are missing.
+    #   /etc/nginx/oc-selfsigned.crt  root:nginx 0644
+    #   /etc/nginx/oc-selfsigned.key  root:nginx 0640
+    #   /etc/nginx/oc-htpasswd        root:nginx 0640   (`oc:<bcrypt>`; plaintext in /secrets)
+    services.nginx = {
+      enable = true;
+      recommendedProxySettings = true;
+      virtualHosts.${cfg.opencodeUiHost} = {
+        listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ];
+        # onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener.
+        onlySSL = true;
+        sslCertificate = "/etc/nginx/oc-selfsigned.crt";
+        sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
+        basicAuthFile = "/etc/nginx/oc-htpasswd";
+        locations."/" = {
+          proxyPass = "http://127.0.0.1:4096";
+          proxyWebsockets = true;
+        };
+      };
+    };
+  };
+}
Author	SHA1	Message	Date
notplantsandClaude Fable 5.1	4bc483326b	orchestrator-host: symlink the opencode installer's binary into ~/.local/bin; README: nginx file perms, first-rebuild wrapper, pre-cutover checks Found on the first activation of #cc-ci on 195.201.88.249: the opencode installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and nginx's pre-start config check runs as the nginx user, so the staged cert/htpasswd must be root:nginx 0640, not root:root 0600. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:24:47 +00:00
notplantsandClaude Fable 5.1	f105808bed	nix/hosts/cc-ci: real hardware + networking from the box; document the infect fixes hardware.nix / networking.nix are now the nixos-infect output of 195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that matters: defaultGateway as an attrset with interface = "eth0". README §2a records what it actually took to get NixOS running on the Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's unit files shadowed every NixOS service), and the gateway fix applied via a chroot rebuild with the nix sandbox off. flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned to 1000 so workspace rsyncs by uid line up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:15:16 +00:00
notplantsandClaude Fable 5.1	31af820079	nix: one Hetzner host for the CI server + the orchestrator (#cc-ci) The orchestrator's flake now builds the machine it shares with the cc-ci CI server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server (new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix — the host contract those units always assumed (loops user, claude/opencode CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns 80/443, nix-ld, tool set, `ssh cc-ci` → loopback). nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old server's layout so the flake evaluates; they get replaced by the nixos-infect output of 195.201.88.249. README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake → staging → data restore → cutover). archive/ holds the retired Incus/Hetzner orchestrator host configs, the old terraform and the migration plans; references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working plan for the move. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 19:58:33 +00:00
autonomic-bot	5bebd13e49	docs: record weekly upgrade report Completed DeepSeek Flash run and published report verified.	2026-08-31 20:47:10 +00:00
autonomic-bot	82d3127c2a	docs: record weekly upgrade report	2026-08-31 20:46:55 +00:00
autonomic-bot	d824800f8a	upstream(n8n): 2026-08-31 release-notes (2.34.5/2.34.6, 2.35.4-7, 2.36.8/2.36.9, 2.37.4, 2.37.5 withdrawn, 2.37.6)	2026-08-31 20:31:59 +00:00
autonomic-bot	e7f861ec9e	upstream(mattermost-lts): 2026-08-31 re-check note (ESR still 11.7.10, PR #2 re-verify)	2026-08-31 20:20:09 +00:00
autonomic-bot	1b75d986b2	fix: default testme Gitea URL Verified with bash -n; public default only.	2026-08-31 19:29:59 +00:00
autonomic-bot	e72fbdb79c	fix: default testme Gitea URL	2026-08-31 19:29:31 +00:00
autonomic-bot	f6c6b39b18	docs: record acme-dns production issuance	2026-08-31 19:04:54 +00:00
autonomic-bot	02ff90777f	docs: record acme-dns production issuance	2026-08-31 19:04:41 +00:00
autonomic-bot	012212498d	docs: record acme-dns handoff	2026-08-31 17:17:09 +00:00
autonomic-bot	521f6e6934	docs: record acme-dns handoff	2026-08-31 17:16:51 +00:00
autonomic-bot	07bc0a15d5	Merge pull request 'config: load public cc-ci runtime environment' (#14 ) from chore/public-runtime-config into main	2026-08-31 16:50:25 +00:00
autonomic-bot	00a4ee6752	config: load public cc-ci runtime environment	2026-08-31 16:49:15 +00:00
autonomic-bot	9c619abd6c	upstream(n8n): release-notes sources 2.36.4..2.37.3	2026-08-28 04:15:13 +00:00
autonomic-bot	980530f505	upstream(mattermost-lts): 2026-08-28 re-check — 11.7.10 newest ESR patch	2026-08-28 04:00:02 +00:00
autonomic-bot	3047ee7724	config: GITEA_URL is public config, not a credential The Gitea host was stored in the credentials env file, so a literal secret scanner treated it as a secret and refused two READMEs in another repo that link to git.autonomic.zone. It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py already default it to git.autonomic.zone, and it is this repo's own remote host. It is a bare hostname with no scheme, userinfo or path. Removed from the env file and defaulted here the same way the Python already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard -- those are real credentials and must still fail loudly if unset.	2026-08-26 17:46:58 +00:00
autonomic-bot	c472aa3642	upstream(wordpress): release-notes sources (sftp/atmoz + standing note)	2026-08-21 03:45:57 +00:00
autonomic-bot	7252f7ee35	upstream(mattermost-lts): 2026-08-21 re-check — 11.7.9 ESR is the only current LTS; PR #2 redirected to ESR	2026-08-21 03:34:36 +00:00
autonomic-bot	75386fc23b	upstream(n8n): 2.36.x release-notes coverage (2.36.0 feature minor, 2.36.2/2.36.3 patches)	2026-08-21 03:28:47 +00:00
autonomic-botandClaude Opus 4.8	19abebb312	nix: split the cc-ci-specific config into its own module + flake output The orchestrator host is a general agent box that also serves several unrelated projects (atproto-likes, the lichen and project orchestrators, the shared opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci orchestrator session, the weekly recipe-upgrade run and its hourly supervisor — now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so the two can evolve and be reviewed independently and another host can consume just the cc-ci half. No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the split: the unit set is identical to the running system, and all six cc-ci units are byte-identical modulo nix store hashes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet	2026-08-20 16:29:02 +00:00
autonomic-bot	565e021185	Merge pull request 'config: drop plausible from weekly upgrade set (-> external)' (#13 ) from config/drop-plausible-weekly into main	2026-08-17 16:01:38 +00:00
autonomic-bot	929faeb055	config: drop plausible from weekly upgrade set (-> external) Move plausible from weekly to external in used-recipes.md. cc-ci still deploys/tests it (the recipe + tests stay enrolled, !testme bridge unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all run — maintained elsewhere, like uptime-kuma. The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's run stays open for the operator; no future weekly PRs will be opened for it.	2026-08-17 16:01:26 +00:00
autonomic-bot	a29b993160	Merge pull request 'config: switch upgrader + report to deepseek, keep supervisor on glm' (#12 ) from config/deepseek-upgrader-model into main	2026-08-16 02:29:20 +00:00
autonomic-bot	a0d6fc9417	config: switch upgrader + report to deepseek, keep supervisor on glm The weekly /upgrade-all parent session and the /recipe-report session now run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env). The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default in launch-supervisor.py, not overridden). Subagents already bind deepseek via the cc-ci repo's opencode config (fix from 2026-08-10, verified this week: all 16 subagents across both waves ran deepseek-v4-pro). LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to the zen endpoint, which returns 200 (not 429) → resume immediately — correct, since tinfoil has no rolling usage limit to wait out. Verified the probe behaviour with a direct curl. Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance' (opencode zen workspace balance exhausted), then sat unfinished for 40h while the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use API key) has no rolling balance limit, so this can't recur. Also documents the session recovery in JOURNAL.md (the stalled run was completed via a fresh scoped upgrader — the original 2.58M-token session was unresumable: the inference endpoint silently drops the oversized request).	2026-08-16 02:28:38 +00:00
autonomic-bot	9409adffb8	upstream(n8n): 2.35.x release notes	2026-08-15 21:04:58 +00:00
autonomic-bot	69d1840ea5	upstream(lasuite-docs): note minio Docker images stopped at 2025-09-07	2026-08-14 03:06:49 +00:00
autonomic-bot	ab5d11ea2a	Merge pull request 'advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?'' (#11 ) from feat/nvd-cpe-source into main	2026-08-11 22:17:13 +00:00
autonomic-bot	74117c2260	cve-check: record the remedy for a blind recipe, not just the symptom The skill said to render a sourceless recipe as '?'. It now says how to stop it being sourceless: declare an NVD CPE in the registry. That is what took the fleet from two blind recipes to zero, and it is the first thing to try when the sweep flags another.	2026-08-11 22:16:52 +00:00
autonomic-bot	985dc06e47	advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?' Two recipes could not see CVEs at all. mattermost-lts has an empty GitHub advisory feed and renders its security bulletins client-side, so a text sweep finds nothing; mumble publishes nothing anywhere the registry points. Both returned '?' - nothing measured - which is honest but useless. NVD is CPE-indexed and carries structured version ranges, so it answers where the vendor does not. Declared per recipe as 'nvd-cpe: <image> = <cpe:2.3:...>'. mattermost-lts 10.5.0 -> 10.12.4 165 CVEs mattermost-lts 10.11.22 -> 10.12.4 0 CVEs (measured, not unknown) mumble 1.3.0 -> 1.6.870 2 CVEs Both NVD range forms are used: versionEndExcluding is a patched version; versionEndIncluding means the fix version is unpublished but the upgrade delivers it whenever it crosses X. That 0 for the actual mattermost upgrade is the interesting one, and it needed a new rule to be correct: a fix on the line you upgrade FROM was already yours. mattermost patches every maintained line at once, so 10.11.22 -> 10.12.4 crosses 10.12.1 while 10.11.22 already had the 10.11.4 backport. Without the rule the scan claimed 12 CVEs the upgrade did not deliver. The rule is skipped for placeholders: '7.4.X' parses to a bare 7.4 and would read as 'already fixed at 7.4', which silently dropped redis CVE-2024-46981 and took discourse 140 -> 139 before I caught it. 79 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 / plausible 6 unchanged. Fleet sweep: 0 recipes with no usable CVE source, down from 2.	2026-08-11 22:16:37 +00:00
autonomic-bot	4b9978ac02	Merge pull request 'audit-sources --security-sources: find the recipes that cannot see CVEs at all' (#10 ) from feat/audit-security-sources into main	2026-08-11 20:02:24 +00:00
autonomic-bot	46ace30b4d	audit-sources --security-sources: find the recipes that cannot see CVEs at all Follow-up to the nginx blind spot. Sweeping all 22 recipes for sources whose CVEs are USABLE (structured advisory feed, or a changelog attributable to releases) rather than merely visible. Before the changelog-attribution fix: 20 unusable sources. After: 5, and all five are redundant - the same project also publishes an advisory feed (redis, gitea, minio, clickhouse), so nothing is actually lost. One real find, same shape as nginx: ONLYOFFICE/DocumentServer publishes NO GitHub advisories, and the registry pointed its CHANGELOG.md at the GitHub blob page - 636KB of markup in which the release headings do not survive HTML-stripping, so 24 CVEs were visible and NONE attributable. The raw URL attributes all 24. Rather than fix one registry line, advisory-scan now normalises github.com/../blob/.. to raw.githubusercontent.com, which fixes every entry present and future. lasuite-drive bumps documentserver, so this was live. Genuinely blind after all that: mattermost-lts and mumble - no advisory feed, no attributable changelog, no CVE data anywhere the registry points. mattermost is the notable one: its bulletins are client-side rendered, so a regex sweep sees nothing. Their scans can report 0 while nothing was measured, so /cve-check now renders those recipes as ? and says why. The audit output distinguishes a blind RECIPE from an unparseable PAGE, because conflating them made 5 harmless redundancies look like 5 gaps.	2026-08-11 20:02:06 +00:00
autonomic-bot	dab3edf3c2	Merge pull request 'advisory-scan: attribute vendor-changelog CVEs to the release that fixed them' (#9 ) from feat/changelog-version-attribution into main	2026-08-11 19:50:00 +00:00
autonomic-bot	db37f1618b	advisory-scan: attribute vendor-changelog CVEs to the release that fixed them nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes from nginx.org/en/CHANGES, and the scan scraped ids out of it without attributing them to a release - so they had no patched version, could never be classified, and every nginx bump in the fleet reported 0 CVEs. nginx is a sidecar in most recipes, so this was a fleet-wide blind spot. Measured on the two PRs that prompted the question: lasuite-docs#7 nginx 1.31.1 -> 1.31.3 0 -> 6 CVEs lasuite-drive#6 nginx 1.31.2 -> 1.31.3 0 -> 3 CVEs matching a hand count of the changelog exactly (three fixed in 1.31.2, three in 1.31.3; the narrower window correctly counts only the latter). How: when a vendor page is organised by release, each CVE is attributed to the nearest preceding release heading ('Changes with nginx 1.31.3', '## v1.31.3'), and that becomes its fixed-in version. The CVE is tied to a window by the image name appearing in the page URL (window 'nginx' <-> nginx.org/...). A changelog lists the project's whole history, so only releases the window actually crosses count - asserted by a test that the 2013 entries stay out. 76 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 unchanged.	2026-08-11 19:49:43 +00:00
autonomic-bot	4bad1ea6db	Merge pull request 'advisory-scan: derive the scan windows from a compose diff (--compose-to)' (#8 ) from feat/advisory-scan-compose into main	2026-08-11 19:24:46 +00:00
autonomic-bot	ef58e33102	advisory-scan: derive windows from a compose diff (--compose-to) Typing --from/--to/--image by hand means someone has to remember the recipe also bumped its redis. That is how sidecar CVEs went uncounted for months. Point this at a PR's compose.yml and it reads the windows off the diff instead. advisory-scan.py plausible --compose-to <.../branch/<pr>/compose.yml> -> community-edition: v2.0.0 -> v3.2.1 -> clickhouse-server: 23.4.2.11-alpine -> 24.12-alpine -> 6 CVEs, identical to the hand-specified args Details that mattered: - keyed by SERVICE, not image repo. plausible moved plausible/analytics -> ghcr.io/plausible/community-edition; keyed by repo that reads as one image vanishing and an unrelated one appearing, and the app window - the one carrying the critical - is lost entirely. - the baseline is the repo's DEFAULT BRANCH resolved from the API, never assumed to be main, because several recipes keep a stale main beside a live master. - image names are matched against advisory sources BOTH ways: an image name is often longer than its source repo (clickhouse/clickhouse-server vs ClickHouse/ClickHouse) and sometimes shorter (redis vs redis/redis). One direction silently dropped the clickhouse window. - credentials go in an Authorization header, never the URL: in-URL creds leak into shell history and process lists, and urllib mis-parses a password containing a colon. --from/--to/--image remain for finer-grained checks (scanning a window that is not a literal compose diff). 71 tests; discourse 140 / gitea 2 / mailu 2 unchanged.	2026-08-11 19:24:24 +00:00
autonomic-bot	c352ea9058	Merge pull request 'add /cc-ci-cleanup — reconcile, close dead PRs, report what blocks the rest' (#7 ) from skill/cc-ci-cleanup into main	2026-08-11 19:17:19 +00:00
autonomic-bot	fae2fbe21b	add /cc-ci-cleanup — reconcile, close dead PRs, report what actually blocks the rest 25 open recipe PRs had accumulated, and the list had stopped being readable: CI sweep artifacts that were never meant to merge sat next to genuine CVE fixes, and three PRs the operator had been told were outstanding were in fact already merged upstream (discourse #6 with 140 CVEs, keycloak #6 with 12, n8n #5) — visible only once the mirrors were reconciled. The skill: reconcile every mirror from true upstream FIRST (that step alone closed those three), survey every open PR deterministically, close the ones that cannot merge or were never meant to, and report prioritised action items — CVE-carrying first — for the ones that should land. It never merges a recipe PR. pr-survey.py gathers the facts and decides nothing: behind_main, mergeable, diff_files, which images the PR adds vs which are already pinned in main, the newest !testme verdict, branch kind, age. One correctness detail worth the extra state: a FAILED diff fetch is reported as unknown, never as an empty diff. gitea #4 reads that way (force-pushed branch) while being a verified green fix, and 'empty diff' is a close signal — so the tool says DIFF-UNREADABLE(do not close on this) instead.	2026-08-11 19:17:00 +00:00
autonomic-bot	6ebc35bc18	Merge pull request 'AGENTS.md: ship infra work as PRs, self-merge, operator reviews retrospectively' (#6 ) from policy/pr-then-merge into main	2026-08-11 19:08:55 +00:00
autonomic-bot	96c536f543	AGENTS.md: ship infra work as PRs, self-merge, operator reviews retrospectively Operator policy (2026-08-11). For cc-ci-orchestrator and cc-ci: branch, open a PR whose description is written to be read AFTER the fact, merge it yourself once verified, and let the operator review retrospectively. The PR is not a gate — it is how the work stays legible — so a description that says 'fix scanner' has failed at its only job. Explicitly does NOT extend to recipe repos: those are created and verified but never agent-merged, because they change what deploys on other people's infrastructure. Also records what to do when work has already landed on main without a PR: pin a branch at the pre-work commit and PR against that, rather than rewriting published history.	2026-08-11 19:08:39 +00:00