fix: verify GHCR release manifests when tag listing is huge

Only this event entry committed; pre-existing uncommitted orchestrator WIP left in the worktree untouched.

.claude/skills/cve-check/SKILL.md

@@ -95,8 +95,9 @@ For each recipe produce **one window per image**: `current pinned tag → newest
 the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
 CRITICAL came from, and an image with no window is not counted at all.
 
-- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
-  nothing an upgrade could fix. Record it and move on.
+- **No upgrade available** → the recipe is `UPTODATE`; its **CVEs fixed by an available upgrade**
+  count is `0`. This does not assert that the running image has no known CVEs. If advisory sources
+  failed or are blind, use `?` and say why.
 - **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
   leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
   Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
@@ -111,9 +112,17 @@ python3 cc-ci-plan/audit-sources.py --security-sources
 A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
 measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
 
-**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory
-feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by
-declaring an NVD CPE in their registry:
+Do not assume that the fleet has zero blind recipes. Run the audit every sweep and inspect source
+statuses; an HTTP error or rate limit is not evidence of no CVEs. `rustdesk-server` is currently
+blind because its upstream GitHub repo publishes no security advisories and no other usable source
+has been verified; mark it `?` until that changes.
+
+Check `test -s /srv/cc-ci/.github-token` before a fleet scan. Without that read-only token,
+GitHub's anonymous API limit can be exhausted partway through the fleet; HTTP 403 results are
+failed sources, so the affected CVE counts remain unknown.
+
+Two older blind spots — `mattermost-lts` and `mumble` — were addressed by declaring NVD CPEs in
+their registries:
 ```
 - nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
 ```

.claude/skills/recipe-report/SKILL.md

@@ -30,7 +30,9 @@ keeps every weekly edition looking the same regardless of which model writes the
 2. **Survey.** `python3 /srv/cc-ci/cc-ci-plan/recipe-report.py survey <DATE> > /tmp/survey.json`. This
    gives you: the raw `/upgrade-all` summary; for every recipe mirror its open PRs + the `cc-ci/testme`
    verdict; and each recipe's **per-recipe upgrade notes** (`upgrade_notes_md` — the breaking-change /
-   **CVE / security** analysis the upgrade did). Read it all.
+   **CVE / security** analysis the upgrade did). Read it all. Also run
+   `python3 /srv/cc-ci/cc-ci-plan/audit-sources.py --security-sources`; a recipe with no verified
+   security source or a failed source must be called out, not presented as a clean CVE scan.
 
 3. **Review & classify.** The page order is: **short lead → the full wire table (priority-sorted) →
    Addendum → Security Bulletin → per-recipe "What changed".** Same newspaper style; this order.
@@ -46,9 +48,13 @@ keeps every weekly edition looking the same regardless of which model writes the
        lists **failed sources**, the count is **not** authoritative: render the cve cell as `?`
        (unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea
        RCEs were reported as "none" on 2026-08-07.
-     - **`?` must stay RARE — it means "we tried and could not tell", not "we didn't look".** Use it
-       ONLY when a scan ran and reported genuinely failed sources, **or when the scan block says
-       COUNT UNKNOWN**. In that case the scan's `0` means *not determined*: publish `?` and say so in
+     - The table's **CVEs fixed** column counts known CVEs addressed by the available recipe upgrade.
+       A zero for an up-to-date recipe means there is no upgrade window to credit; it does not
+       certify that the running images are free of vulnerabilities. If an upgrade exists but its
+       advisory scan is absent, a source failed, or an image window was unmatched, use `?`.
+     - **`?` means the count is not verified.** Use it when an upgrade has no scan, a source failed,
+       an image window was unmatched, or the scan block says COUNT UNKNOWN. In that case publish
+       `?` and say why in
        the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
        **version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
        (discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.

.claude/skills/upgrade-all/SKILL.md

@@ -143,12 +143,31 @@ ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<r>; \
   script -qec "abra recipe upgrade $R -m -n" /dev/null; \
   git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
 ```
-Build `RECIPES_TO_UPGRADE` = recipes with **≥1 available upgrade** (after the stash, so the untracked
-overlay doesn't count as dirty). Others go to `SKIPPED_UPFRONT` with a reason (`dirty-worktree` only for
-real TRACKED edits, `up-to-date`, `not-fetchable`). **A `tag and digest … not supported` parse FATA is
-NOT `not-fetchable`** — per box item 4, catch it and resolve availability via the direct upstream-tag
-check; only `not-fetchable` when the mirror genuinely can't be fetched (network/auth), never for a
-digest-pinned image.
+**Mandatory registry cross-check for EVERY recipe and EVERY image, even if abra says current:** after
+reconciling the mirror and refreshing the recipe checkout, run
+`python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --recipe-dir /root/.abra/recipes`
+on cc-ci. Read the JSON `images`, `upgrades_available`, `unresolved`, and `all_resolved` fields.
+The candidate list is the **union** of abra's upgrades and the resolver's upgrades. CryptPad's
+`version-YYYY.M.D` app tag is a known counterexample: abra only reports a parse warning, while the
+resolver finds the newer image. A warning is never evidence that an image is current. Inspect both
+`newest_same_shape` and `newest_within_major`; a newer major is still a candidate for planning,
+though it must not be applied without a compatibility check.
+
+Build `RECIPES_TO_UPGRADE` from every recipe with at least one candidate. Only label a recipe
+`up-to-date` if abra completed successfully **and** the resolver checked every relevant image
+(`all_resolved: true`) **and** neither found a candidate. If an image is unresolved, templated,
+floating, or lacks comparable tags, investigate it directly and record the evidence. Otherwise
+record `unverified` with the image and reason; never fold it into `up-to-date`. `dirty-worktree`
+applies only to real tracked edits. A tag+digest parse failure is not `not-fetchable`.
+
+Run `python3 /srv/cc-ci/cc-ci-plan/audit-sources.py --security-sources` once for the full inventory
+and include any `NO VERIFIED CVE SOURCE` recipes or failed advisory sources in the weekly summary.
+Do not describe their CVE count as clean. This audit is a coverage check, not a substitute for
+`advisory-scan.py` on each actual upgrade window.
+Before the audit, check `test -s /srv/cc-ci/.github-token`. If it is missing, record that GitHub
+advisory checks are subject to the anonymous 60-request/hour limit; any resulting HTTP 403 is a
+failed source and must become `?` in the report. Never replace it with `0` or silently retry from
+model memory. The file is a read-only GitHub token location; never print its contents.
 
 **Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
 PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new

.opencode/skills/cctest-recipe-upstream/SKILL.md

@@ -24,3 +24,68 @@ run `/cctest-intro` / `/cctest-setup-sandbox` first.
   timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
 - The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
   a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
+
+**Sandboxed vs non-sandboxed mode.** This skill needs only git + (optionally) the Gitea API —
+not the test server — so it can run either way. Probe, then follow that branch:
+
+- **Sandboxed** (ARM env configured: `test-ssh/.testenv` with `GITEA_USERNAME`/`GITEA_PASSWORD`/
+  `GITEA_URL`, sandbox/test instances): the canonical `recipe-upstream` script in
+  `references/recipe-maintainer/.claude/commands/recipe-upstream.md` runs as written. Note its
+  WORKSPACE probing expects `/workspace` or `~/Documents/recipe-maintainer`; on a bare host pass
+  the submodule dir explicitly instead.
+- **Non-sandboxed** (no ARM env on the host — e.g. the orchestrator, where the recipe-maintainer
+  checkout is only a pinned submodule): no sandbox/test instances are needed and NONE of the
+  setup skills are. Recipe-maintainer mirrors on `git.autonomic.zone` are publicly readable, so:
+  1. Check out the recipe if missing: `abra recipe fetch <recipe>` (lands in `~/.abra/recipes/<recipe>`)
+     — or a plain anonymous `git clone https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     if abra is unavailable.
+  2. Fetch the PR head branch from the mirror **anonymously** — no credentials in the remote URL:
+     `git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     (remote update rather than re-add if it exists), then
+     `git fetch gitea +refs/pull/<N>/head:refs/heads/<head_ref>`.
+  3. Fetch PR metadata (head/base refs, merged flag, release bump line) from
+     `https://git.autonomic.zone/api/v1/repos/recipe-maintainers/<recipe>/pulls/<N>` —
+     unauthenticated; use bot creds only if the repo turns out to be private (orchestrator hosts
+     can read them from `/srv/cc-ci-orch/.testenv` — never written anywhere else).
+  4. Everything else in the canonical script (origin/dev remote setup, release recommendation,
+     emitted next-steps) is identical.
+
+Every time the branch was prepared **here**, remember it exists only on this host — the
+operator's machine must fetch it first. Always emit this **step 0** before the push step
+(anonymous public fetch, no credentials needed):
+
+```
+  # 0. On a machine WITHOUT the branch pre-fetched, get it from the autonomic mirror
+  #    (fetch by URL — works regardless of what the local remotes are named):
+  cd <local checkout of the recipe>
+  git fetch https://git.autonomic.zone/recipe-maintainers/<recipe>.git +refs/pull/<PR_NUM>/head:refs/heads/<HEAD_REF>
+  git checkout <HEAD_REF>
+```
+
+If the operator's checkout does NOT yet have the mirror remote, emit once before the fetch:
+
+```
+  git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git
+```
+
+In both modes the final output is a set of commands for the operator to run on a machine **with
+push access to `git.coopcloud.tech`** — always print them, even when everything local is
+already prepared.
+
+**Verify the merge-base BEFORE rebasing — bases can be traps.** Upstream repos can
+carry a stray, divergent `main` ALONGSIDE the real `master` base branch. Rebase targets must be
+chosen by evidence: `git ls-remote origin | grep -E 'refs/heads/(main|master)$'`, then
+`git merge-base <head_ref> origin/<candidate>` — the correct base is the one where the upgrade
+branch's merge-base is its tip's parent (i.e. `<head_ref>` is already directly on that line —
+then NO cherry-pick is needed, the raw branch compares clean). Cherry-picking onto the wrong
+base produces the very messy 3-dot compare this guard exists to prevent (real case: gitea —
+first "rebase" landed on a stray `main` and had to be redone onto `master`).
+
+**Branch-name mismatch mirror vs upstream.** The mirror and upstream can use different base-branch
+names, and a mirror can even carry a stray, divergent `main` alongside the real base (gitea: mirror
+PR against `main`, upstream `master`; keycloak PR #9: base `main`, both mirror and upstream on
+`master`). The canonical script now detects the upstream default branch by evidence (`git
+ls-remote --symref origin HEAD`, then the git.coopcloud.tech API, then the PR base with a warning)
+and uses it for the compare URL AND the post-merge release steps — never hardcode `main`, and when
+it prints the base-mismatch warning, confirm the upgrade branch actually sits on that line
+(`git merge-base` against both candidates) before emitting the push step.

.opencode/skills/recipe-enroll/SKILL.md

@@ -43,7 +43,10 @@ On cc-ci, after `abra recipe fetch <recipe>`, read `~/.abra/recipes/<recipe>/`:
 ### 2. Create + sync the mirror
 
 Create `recipe-maintainers/<recipe>` (Gitea API: `POST /orgs/recipe-maintainers/repos`,
-`{"name":…,"private":true,"default_branch":"main","auto_init":false}`), then force-sync from
+`{"name":…,"private":false,"default_branch":"main","auto_init":false}`) — a recipe mirror MUST be
+created **public**: the cc-ci recipe mirrors are public (fleet invariant; the report's live PR-STATUS
+proxy `report./pr/<recipe>/<n>` is tokenless and only sees public repos — private-from-birth mirrors
+like gitea/wordpress left its status cells dark until flipped). Then force-sync from
 coopcloud with the existing helper **run on cc-ci with creds injected via stdin** (the host has
 no `.testenv`):
 ```

cc-ci-plan/JOURNAL.md

@@ -31,8 +31,44 @@ handoff).
 
 ---
 
-## Session 2026-05-31 ~18:30 UTC — Claude Sonnet 4.6
+## Session 2026-09-14 ~16:45 UTC — opencode glm-5.3-flash (orchestrator) — round 2: Anubis UA
 
+**Left off:** The real root cause turned out to be TWO independent layers; the mirror-privacy
+fix (earlier session entry today) was necessary but not sufficient. Operator's browser console
+showed CORS failures redirecting to `anubis.swarm.autonomic.zone/.within.website/?redir=…`.
+Reproduced exactly: the `/pr/` proxy forwards the END browser's User-Agent to Gitea; Gitea sits
+behind **Anubis**, which 307-challenges browser-like UAs to `anubis.swarm.autonomic.zone`
+(no CORS headers) → every fetch throws in the browser → all cells "?" (curl passed clean, which
+is why server-side checks and my earlier headless test never saw it — intermittent/rate-dependent
+for my playwright run). Fix: cc-ci **PR #38** adds
+`proxy_set_header User-Agent "ccci-reports-proxy/1.0";` to the reports.nix `/pr/` location.
+Hot-verified on the host by mount-swapping a fixed conf into the running task (one mis-step:
+`--mount-rm`+`--mount-add` same-target order wiped the mount; re-added), scoped live, all 16
+cells rendering with a real Chromium. Merged PR #38, `nix flake update cc-ci`,
+`nixos-rebuild test` → healthy (reports 200, no failed units) → `switch` (flake.lock commit
+9e7770f). Final verify: browser-UA curl 200 both gitea/9 + full headless-Chromium sweep 16/16
+OPEN, zero non-200 /pr fetches. Also flipped memory: `memory/gitea-anubis-ua-challenge.md` +
+MEMORY.md index.
+**Open:** nothing blocking; next weekly /recipe-report and STATUS live-checks carry the fix.
+
+## Session 2026-09-14 ~15:00 UTC — opencode glm-5.3-flash (orchestrator)
+
+**Left off:** Report STATUS column fix. Operator reported the week-2026-09-11 report's live
+PR-STATUS column all "?" — root cause: the tokenless same-origin proxy
+`report./pr/<recipe>/<n>` (cc-ci `nix/modules/reports.nix`) 404s on **private** mirrors; two
+late-enrolled mirrors, `recipe-maintainers/gitea` (2026-06-11) and `wordpress` (2026-08-03),
+had been created `"private":true` from birth — by the stale instruction in
+`/recipe-enroll`'s mirror step (the other 21 mirrors were flipped public on 2026-06-09, and
+the old 'org is private' blocker is long resolved). Fixed: secret-scanned both repos, flipped
+`private=false` (PATCH with bot creds), patched `.opencode/skills/recipe-enroll/SKILL.md` to
+create mirrors `private:false`, updated memory/recipe-mirrors-public-org-blocker.md +
+MEMORY.md index. **Verified in a real headless Chromium (nixpkgs chromium + playwright)**:
+all 16 STATUS rows render `open`, every `/pr/` fetch 200 JSON. Commit 6e93922 pushed. The
+STATUS column refreshes live every 30s; cells go ✓ when a PR merges. No reports.nix change
+was needed (proxy itself was healthy).
+**Open:** nothing on this; the report index regenerates next weekly run.
+
+## Session 2026-05-31 ~18:30 UTC — Claude Sonnet 4.6
 **Left off:** Got opencode/deepseek-v4-pro working as the loop backend. Both builder and
 adversary are actively running on `tinfoil/deepseek-v4-pro` (via `inference.tinfoil.sh`).
 Phase 5 [11/11] in progress. The operator is debugging the opencode web UI visibility and
@@ -1202,3 +1238,175 @@ the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer
   `upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the
   tier default). Next fire Fri 2026-09-11 02:00 UTC.
 - The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change).
+
+## Session 2026-09-21 — domain cutover to ci.autonomic.zone (orchestrator)
+
+**What happened.** Morning: hourly supervisor resumed the stalled 09-18 weekly run on the GO tier
+(ZEN endpoint dead server-side — `UnknownError`; run completed 13 green PRs, 0 failed). Published
+the missing week-2026-09-18 report (PR-finding #4; launcher defaults flipped to `go` in
+cc-ci-orchestrator PR #23). Then executed the full domain cutover per
+`cc-ci-plan/plan-domain-migration-ci-autonomic-zone.md` (PR #25).
+
+**Plan deviation (simplification).** No dual-cert SNI: ONE Let's Encrypt cert carries SANs for
+BOTH zones (`ci` + `*.ci` of autonomic.zone AND commoninternet.net) — the unchanged single-pair
+`ssl_cert/ssl_key` traefik reconciler keeps working; Phase 4 reissues without the legacy SANs.
+The new zone's DNS-01 challenge reuses the SAME acme-dns account: storage re-keyed by
+`cc-ci-acme-storage-seed.service` (jq clone of the legacy entry under `ci.autonomic.zone`),
+CNAME already delegated. Proven by a hand lego **staging** run before any production change.
+
+**Merged:** cc-ci #39 (front doors dual Host rules, bridge/dashboard env URLs, drone abra rename,
+runner RPC, naming.py → `*.ci.autonomic.zone`, dual-zone name regexes in lifecycle/warm/prune,
+recipe-report URLs) · cc-ci #40 (seed-unit nesting fix) · cc-ci #41 (have_secret stack-scope —
+caught live: the old stack's `*_rpc_secret_v1` satisfied the check post-rename) · cc-ci #42 (nix
+interpolation escape) · orchestrator #27 (oc.ci host + `opencodeUiExtraHosts`, host self-pins,
+flake bump).
+
+**Deployed** via `nixos-rebuild test` → verify → `switch` (generation `nn1vwiv7v1k…`, running ==
+boot). Cert SANs confirmed 4-name; all 5 front doors answer on BOTH zones
+(200/200/303/401 + traefik 200), TLS verify=0 from outside; zero failed units; disk dropped
+88%→45% after prune.
+
+**Drone migration.** New abra app `drone.ci.autonomic.zone`, FRESH DB (module's
+`DRONE_USER_CREATE` re-injected the sops bridge token). The Gitea OAuth app redirect now has both
+URIs; the client secret was rotated (each Gitea PATCH regenerates it) and synced through
+sops → `sops-install-secrets` → swarm secret v1 → drone. Bootstrapped OAuth
+(`drone login ok (admin=true)`), re-enabled cc-ci + discourse repos, build timeout 60m.
+Webhooks: ghost + discourse repointed (secrets preserved); cc-ci repo's bridge webhook →
+`ci.autonomic.zone/hook`, stale drone hook deleted, Drone's auto-created new-zone hook active.
+Old stack removed + orphaned secrets reaped.
+
+**E2E proof.** `!testme` on keycloak PR #9 → bridge → drone build #1 (new DB numbering) → runner →
+harness → `results.json` + PR card `✅ passed` linking `ci.autonomic.zone/runs/1/summary.png`.
+
+**Deferred / open.**
+- Warm stacks + backupbot stay on the legacy zone (data-warm volumes / restic password tied to
+  abra app names) — post-bake migration; harness regexes accept both zones meanwhile.
+- Docs sweep (~60 references: AGENTS.md, README, skills incl. cc-ci-status front-door list,
+  launcher printed URLs).
+- Phase 4 (after ≥7 clean days): drop legacy SANs (reissue), remove legacy Host arms + host
+  self-pins + Gandi records (`ci`, `*.ci`, `_acme-challenge.ci`), TTLs back to 3600.
+- Host auto-update was `failed` 2026-09-15 (health check) — `/cc-ci-orchestrator-update` still
+  pending; next auto-attempt Tue 09-22.
+
+## Session 2026-09-28 20:00 UTC — operator-broken cc-ci recovered by plain hard reset
+
+- Operator reported ci.autonomic.zone down after their own change, supplied a Hetzner API token
+  in chat (token is now in the transcript — SHOULD BE ROTATED). Staged at /tmp/opencode/hcloud-token
+  (0600) instead of echoing it.
+- Triage: SSH (port 22) timed out, ICMP 100% loss, tailscale 100.95.31.88 no reply — yet Hetzner
+  reported "running". Old recovery note's server id 134485294 is GONE; current cc-ci is id
+  165014541, public 195.201.88.249 (token project also holds 114514766 autonomic-cc-testing).
+  Last Hetzner action was 2026-09-07 (rescue cycles during the rebuild), so the outage was
+  OS-internal, not API-driven.
+- Fix: single hard reset via `POST /servers/165014541/actions/reset`. ICMP after ~60s, SSH after
+  ~90s. Box booted the default profile nixos-system-cc-ci-26.05.20260906.c257840 — no rescue/
+  GRUB generation-picking needed this time.
+- Post-checks: nginx + gitea active, drone-runner-exec active (NOT drone-runner-docker — wrong
+  guess), disk 41%, https://ci.autonomic.zone → 200. One failed unit:
+  acme-order-renew-ci.autonomic.zone.service — renewal itself fine (cert valid to 2026-12-20),
+  it died on `chmod: out/acme-dns-accounts.json: Operation not permitted` because the file was
+  root:root (touched today 19:54, likely by whatever the operator did) while the unit runs as
+  acme. chown acme:acme (matching the healthy ci.commoninternet.net dir) + restart → unit green,
+  zero failed units.
+- NOTE: no tailscale on this host (`tailscale: command not found`) — the AGENTS.md "ssh cc-ci"
+  alias + 100.90.116.4 peer notes are stale post-rebuild; public-IP SSH is the access path.
+  Recovery scripts in scripts/recovery/ still reference old server id 134485294 — worth updating.
+
+---
+## 2026-09-28 — upgrader session (weekly /upgrade-all)
+
+Weekly run 2026-09-28 complete: 19 considered · 7 GREEN PRs · 1 externally-CI-blocked (lasuite-docs, quay minio 401) · 0 failed · 11 skipped. Summary (PR list): `<logs>/upgrades/upgrade-all-2026-09-28.md` — repo ignores .cc-ci-logs, so the PR list lives in the file + the weekly report.
+
+**Operational config change (no PR):** subagent `general` model `deepseek-v4-flash` was killed upstream ("Model access is disabled", zen endpoint; glm-5.2 also dead) — pointed to `deepseek-v4.1-flash` in `/srv/cc-ci-orch/cc-ci/opencode.json`, `/srv/cc-ci-orch/opencode.json` (uncommitted) + live `/etc/cc-ci/opencode.json`. Prior committed setting still in the cc-ci repo's main. Operator: confirm + PR the bump (config change precedent #37/#42).
+
+**minio watch:** docker.io removal (2026-09-18) → now quay.io/minio/minio 401s since ~09-24; lasuite-docs CI-blocked on it; lasuite-drive passed (image cached).
+
+**abandoned worktree note:** `/srv/cc-ci-orch/cc-ci-conc/` (worktree of cc-ci, untracked in orch repo) has a stale worktree at 2f6787f; no writes this week; recorded so the next sweep notices.
+
+Key logs: .cc-ci-logs/upgrades/*-2026-09-28.md (8 per-recipe + summary).
+
+## 2026-10-02 — upgrader session (weekly /upgrade-all)
+
+Weekly run 2026-10-02 complete: 20 considered · 9 upgraded (PRs opened/extended) · 8 GREEN (!testme) · 1 live-verified but CI-blocked externally (lasuite-docs base canonical, commented) + 1 RED on a genuinely stale gitea LFS test (commented; operator re-runs --with-tests) · 0 failed · 9 skipped. Summary (PR list): <logs>/upgrades/upgrade-all-2026-10-02.md — repo ignores .cc-ci-logs, so the PR list lives in the file + the weekly report.
+Key highlights: gitea upstream shipped its first 2-digit stable (28.0.0-rootless, 2026-09-29 — BREAKING git-egress/actions-retention notes documented in PR #10); ghost 6.67.0 fixes HIGH CVE-2026-84383; nulls held at mysql 8.4 / mariadb 12.3 / postgres 13/15 lines per precedent; matrix-synapse extended to synapse 1.162.0 + MAS 1.26.0; immich bumped v3.2.4 (rc.3.3 declined). Operational: recipe-maintainers/gitea mirror had LOST its master branch (abra whole-recipe FATA) — repaired both refs to upstream HEAD f167b73; reconcile-upstream.sh should learn to recreate a missing upstream-default branch ref, not only sync main. 2 targets (gitea, n8n 2.42.2) were caught in follow-up passes after the initial worklist mis-derivation — nothing left unprocessed. Sweep start: 4 leaked volumes + 46.3 GB images reclaimed (disk 70%->40%); end-of-run reap 0 leaked dev deploys; disk 65% (51G free).
+Per-recipe logs: .cc-ci-logs/upgrades/<recipe>-upgrade-2026-10-02.md (10 files incl. survey).
+NEVER MERGED anything; all PRs await operator review + the operator's `abra recipe release <recipe> -x/y/z` after each merge.
+
+### Event 2026-10-05 ~17:45 UTC — gitea PR #10 stale test fixed (cc-ci PR #44, verified GREEN, NOT merged)
+Operator invoked --with-tests for gitea PR #10 (28.0.0-rootless, RED on test_lfs_roundtrip run #31).
+Diagnosis: stale test — `_api()` caught only HTTPError so one urlopen(timeout=20) socket timeout
+during the 28.0.0 post-restart migration window escaped the poll loop; poll also hard-coded 120s
+deadline vs recipe_meta HTTP_TIMEOUT=600 (STYLE.md §5). Fix: never-raising _api (mirrors
+lifecycle.http_fetch) + deadline from recipe_meta.HTTP_TIMEOUT; assertions untouched.
+cc-ci PR #44 (branch fix/gitea-lfs-restart-wait) also carries the two PRE-EXISTING main lint
+failures (warm.py ruff format; acme-dns.nix statix W201, eval-identical restructure) that had made
+push build #33 RED and would have blocked merge — build #34 now success, lint.sh PASS.
+Verified: full cold suite on host worktree (e83cd46) vs recipe head 91e42c8 → 5/5 tiers pass,
+test_lfs_roundtrip PASSED; unit sample 315 pass / 4 fail = identical pre-existing failures on main.
+Pair note posted on both PRs: merge #44, then re-!testme #10. Nothing merged.
+
+### Event 2026-10-05 ~18:00 UTC — wrong-recipe card screenshot root-caused; runner fix #45 merged; gitea !testme re-run GREEN
+Operator's !testme on gitea PR #10 (drone build 35) showed a custom-html-tiny screenshot in the card.
+Root cause: drone build numbers restarted when its DB was re-created (2026-09-27) while
+/var/lib/cc-ci-runs still holds old-era dirs (numbered to 1377) — build 35 collided with a May-31
+run 35 whose fossil screenshot.png (12950 B = exactly custom-html-tiny's real shot) was embedded
+because screenshot.capture()'s hook branch SKIPPED the snap when out_path existed (no hook saves
+itself). Run itself was genuinely gitea level 5. Fix (cosmetics-only, R7): results.fresh_artifact_dir()
+wipes owned artifacts at run start + capture() always snaps; 4 unit tests (non-vacuous: capture test
+fails on main). Adversary verification PASS (independent fresh clones + own E2E collision test).
+cc-ci PR #45 merged (squash, 35ba9976); push self-tests 38/39 green; lint gate now green on main
+(#44 had already fixed the two pre-existing lint failures).
+Merged #44 first (operator informed; verified earlier via full cold harness run).
+Re-ran !testme on gitea PR #10 → drone build 40: GREEN, level 5, card links /runs/40/, screenshot
+is a fresh PNG of the run's own gitea (footer 1.24.2 = the upgrade-tier BASE deploy, the last-green
+warm canonical — pre-existing capture timing, correct behavior, only freshness was broken before).
+Gitea PR #10 now fully green incl. the fixed LFS test (uses merged #44 test). NOTHING else merged;
+recipe PR #10 + unrelated PR #4 still await operator merge + `abra recipe release gitea -x`.
+Cleanup: host /tmp/v /tmp/r, local worktrees, test-file mis-commit in orch cc-ci clone reverted.
+### Event 2026-10-23 04:15 UTC — weekly /upgrade-all run complete (DEFAULT mode, nothing merged)
+Run 2026-10-23: 19 considered · 9 upgraded with !testme GREEN · 2 PRs open pending the warm-canonical
+minio-prepull fix (lasuite-docs PR #11, lasuite-drive PR #8 — !testme RED at the upgrade BASE prepull:
+warm canonicals 0.4.0+v5.4.1 / 0.12.0+v0.21.0 pin the REMOVED docker.io/minio/minio; PR comments
+explain; operator: advance those canonicals then re-post !testme) · 0 failed · 8 skipped up-to-date
+(cryptpad discourse drone hedgedoc keycloak mailu mumble wordpress — several had stale survey data;
+each re-verified against true upstream before skipping). GREEN PRs: bluesky-pds #6 · custom-html #10 ·
+custom-html-tiny #9 · ghost #8 · gitea #11 · immich #6 · lasuite-meet #12 · matrix-synapse #8 · n8n #9.
+Summary: .cc-ci-logs/upgrades/upgrade-all-2026-10-23.md (per-recipe logs alongside). Step-0 sweep
+reclaimed 19GB images (disk 61%->49%); end-of-run reap 0 leaked dev deploys.
+NEVER MERGED anything; PRs await operator review + `abra recipe release` after each merge.
+### Event 2026-10-30 02:00 UTC — weekly /upgrade-all run complete (DEFAULT mode, nothing merged)
+Run 2026-10-30: 19 considered · 12 upgraded with !testme GREEN · 1 PR open pending its stale warm-canonical
+base (lasuite-docs PR #11 — !testme RED at the base INSTALL rung: canonical 0.4.0+v5.4.1 references the
+removed collab_ss secret; last week's minio blocker is already fixed via #47; head linted + live 12/12;
+operator: seed a v6-cold canonical, re-post !testme) · 1 failed (mailu PR #9 — restore tier RED 3x on a
+PRE-EXISTING upstream defect: main's new backup support lacks a restart post-hook so /data/main.db is
+never restored; PR + operator comment with options) · 5 skipped (gitea + lasuite-meet were already merged
+upstream by their turn; custom-html-tiny drone mumble up-to-date). GREEN PRs: bluesky-pds #6 · cryptpad #6
+(2026.5.1→2026.10.0 breaking server rewrite handled) · custom-html #10→11 (stale nginx digest re-pin) ·
+discourse #12 · ghost #8 · hedgedoc #3 (pgautoupgrade 16→18 in-place) · immich #7 (pgvectors 0.3.0
+same-scheme re-tag) · keycloak #13 (mariadb 12.3→13.0 + AUTO_UPGRADE) · lasuite-drive #8 (+collabora +
+onlyoffice direct checks) · matrix-synapse #8 · n8n #10 · wordpress #3.
+Summary: .cc-ci-logs/upgrades/upgrade-all-2026-10-30.md (per-recipe logs alongside). Step-0 sweep
+reclaimed 66.76GB images + orphan stacks (disk 92%->49%); end-of-run reap 0 leaked dev deploys.
+NEVER MERGED anything; PRs await operator review + `abra recipe release` after each merge.
+### Event 2026-10-30 19:00–21:30 UTC — rustdesk-server one-off upgrade + FULL ENROLLMENT (orchestrator)
+One-off upgrade requested from scratch: public mirror `recipe-maintainers/rustdesk-server` created +
+synced; recipe PR https://git.autonomic.zone/recipe-maintainers/rustdesk-server/pulls/1 (1.1.12→1.1.16,
+branch upgrade-2e65135, unmerged — operator merges). Upstream checked (low-churn, stray 1.4.5 docker
+tags = aborted push; CVE fix POV: UDP punch-hole amplification GH #670 + mio bump; scan 0 CVE-ids,
+client-side CVEs fixed in client 1.4.9). Dev-deploy converged clean twice, then !== enrollment: tests
+authored (status-key health sidecar overlay compose.ccci.yml serving the private-tool-unreachable
+real hbbs public key; ports 21115-21119 + UDP; ws 101 on 21118/19; identity-key cross-source assert;
+punch-hole-amplification regression test w/ version-gate), committed d71a53c, cc-ci PR #48.
+Verify: run 1 RED (custom tier) — hand-rolled version read returned None so the punch-hole gate
+didn't skip on the 1.1.12 head; fixed via harness lifecycle.deployed_identity + loud fail on
+unparsable tag (1653582, f5f11cd); runs 2+3 GREEN level 5/5 (recipe main with gate skipping, AND
+the upgrade head with the punch-hole test passing actively). PR #48 evidence-commented + MERGED
+(200); --cc-ci-only deploy: flake.lock cc-ci 2f6787f→a8b836e, lock commit 4e286ba pushed,
+/etc/cc-ci ff to a8b836e, detached nixos-rebuild test → health (0 failed, swarm 1/1) → switch
+(21:17 UTC). Bridge env confirms POLL_REPOS += recipe-maintainers/rustdesk-server (1/1, no auth
+errors). !testme posted on recipe PR #1 via testme-on-pr.sh → build 119 VERDICT=GREEN (level 5,
+lint pass, clean_teardown, no_secret_leak; card embedded in PR by bridge). Card saved:
+.cc-ci-logs/upgrades/rustdesk-server-testme-20261030.png. Next weekly /upgrade-all picks
+rustdesk-server up automatically (used-recipes.md row committed cd1ec01). /root/cc-ci-test-verify
+removed. Operator merges PR #1 then `abra recipe release`.

cc-ci-plan/REBOOTS.md

@@ -23,3 +23,4 @@ restarts the loops on boot. Count the lines below to see how often it's happenin
 - 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
 - 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
 - 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e
+- 2026-09-28 19:54:59 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=66dadc4e-7f44-493a-aee9-fa76952d78c3

cc-ci-plan/advisory-scan.py

@@ -231,6 +231,8 @@ def registry_urls(recipe: str, registry_dir: str) -> tuple[list[str], str | None
         # recipe render '?' for no real reason — that is what put immich and n8n in the unknown
         # column on 2026-08-07 (https://docs.n8n.io/release-notes/` ← note the backtick).
         u = u.rstrip("`'\"*.,;:>)")
+        if "…" in u or "..." in u:
+            continue  # prose placeholder, not a fetchable security source
         if u and u not in urls:
             urls.append(u)
     return urls, path
@@ -826,6 +828,7 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
                     windows[src] = (v_from, v_to)
                     window_key[src] = k
     for key, wf, wt in (images or []):
+        matched = False
         for src in gh_sources:
             if src in windows:
                 continue
@@ -837,6 +840,9 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
             if k in src.lower() or name in k:
                 windows[src] = (wf, wt)
                 window_key[src] = key
+                matched = True
+        if not matched:
+            report.setdefault("unmatched_image_windows", []).append(key)
     report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()}
 
     def _classify_window(src, wf, wt):
@@ -1020,24 +1026,22 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
     # A scan with NO usable source has not measured anything, so it must not report a number —
     # least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and
     # still produced "0 identified", which was then published as a clean 0 in a CVE report.
-    usable_sources = [
-        s for s in report["sources"]
-        if s["status"] == "ok" or s["status"].startswith("no-advisories-published")
-    ]
+    usable_sources = [s for s in report["sources"] if s["status"] == "ok"]
     no_sources = not usable_sources
     if no_sources:
         report["no_usable_sources"] = True
-    report["count_known"] = not unresolved_any and not no_sources
-    report["cve_count_fixed"] = (len(fixed_set)
-                                 if (not unresolved_any and not no_sources) else None)
-    report["cve_count_total_seen"] = len(report["cves"])
-    # Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
-    # no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
     report["sources_failed"] = [
         s["source"]
         for s in report["sources"]
         if not (s["status"] == "ok" or s["status"].startswith(("no-advisories-published", "skipped:")))
     ]
+    coverage_failed = bool(report.get("unmatched_image_windows")) or bool(v_from and not primary)
+    report["count_known"] = not unresolved_any and not no_sources and not coverage_failed and not report["sources_failed"]
+    report["cve_count_fixed"] = (len(fixed_set)
+                                 if report["count_known"] else None)
+    report["cve_count_total_seen"] = len(report["cves"])
+    # Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
+    # no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
     report["sources_benign"] = [
         s["source"]
         for s in report["sources"]
@@ -1052,16 +1056,20 @@ def markdown(rep: dict) -> str:
          f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
     if not rep.get("count_known", True):
         L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
+        if rep.get("fixed_by_this_upgrade"):
+            L.append(f"\nAt least {len(rep['fixed_by_this_upgrade'])} fixed CVE(s) were identified: "
+                     + ", ".join(rep["fixed_by_this_upgrade"][:15]))
         if rep.get("no_usable_sources"):
             L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry "
                      "file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so "
                      "nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add "
                      "the registry file.")
         else:
-            L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes "
-                     "numeric ordering meaningless across this jump, so no advisory could be "
-                     "classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's "
-                     "release notes for the jump and count by hand.")
+            L.append("\n⚠ This is NOT zero. At least one version window or advisory source could "
+                     "not be checked completely. Render this recipe's cve cell as `?`, never `0`.")
+        if rep.get("unmatched_image_windows"):
+            L.append("\n⚠ image windows with no matching advisory source: "
+                     + ", ".join(rep["unmatched_image_windows"]))
         if rep["unclassified"]:
             L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
                      f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))

cc-ci-plan/audit-sources.py

@@ -234,6 +234,8 @@ def _registry_urls(recipe: str):
     urls = []
     for u in re.findall(r"https?://[^\s)|\]]+", text):
         u = u.rstrip("`'\"*.,;:>)")
+        if "…" in u or "..." in u:
+            continue
         if u and u not in urls:
             urls.append(u)
     return urls, path
@@ -272,21 +274,23 @@ def main() -> int:
         blind_recipes, noisy = [], 0
         for r in recipes:
             rows = security_source_audit(r)
-            feeds = [x for x in rows if x["kind"] == "advisory-feed" and x["cves"] > 0]
-            logs = [x for x in rows if x["kind"] == "changelog"]
+            feeds = [x for x in rows if x["kind"] == "advisory-feed" and x["cves"] > 0
+                     and x["status"] == "ok"]
+            logs = [x for x in rows if x["kind"] == "changelog" and x["status"] == "ok"]
             unattr = [x for x in rows if x["kind"] == "unattributable"]
+            failed = [x for x in rows if x["status"].startswith("error:")]
             noisy += len(unattr)
             usable = len(feeds) + len(logs)
             if usable == 0:
                 blind_recipes.append(r)
-                print(f"!! {r}: NO USABLE CVE SOURCE — {len(unattr)} unparseable page(s), "
-                      f"0 advisory feeds, 0 attributable changelogs")
+                print(f"!! {r}: NO VERIFIED CVE SOURCE — {len(unattr)} unparseable page(s), "
+                      f"0 populated advisory feeds, 0 attributable changelogs, {len(failed)} failed source(s)")
                 for x in rows:
-                    print(f"      {x['kind']:15} {x['source'][:64]} ({x['cves']} CVEs)")
+                    print(f"      {x['kind']:15} {x['source'][:64]} ({x['cves']} CVEs; {x['status']})")
             else:
                 print(f"OK {r}: {len(feeds)} advisory-feed(s), {len(logs)} changelog(s)"
-                      + (f", {len(unattr)} unparseable page(s) (redundant — covered by a feed)"
-                         if unattr else ""))
+                      + (f", {len(unattr)} unparseable page(s)" if unattr else "")
+                      + (f", {len(failed)} failed source(s) — counts incomplete" if failed else ""))
                 for x in logs:
                     print(f"      changelog  {x['source'][:62]} ({x['usable']}/{x['cves']})")
         print(f"\n{len(recipes)} recipes · {len(blind_recipes)} with NO usable CVE source"

cc-ci-plan/launch-report.py

@@ -13,7 +13,8 @@ Usage:
   launch-report.py status         show session state
 
 Env: REPORT_BACKEND (default opencode), REPORT_MODEL (default tracks tier),
-     LOOP_TIER / REPORT_TIER (default 'zen'; 'go' or 'zen'; only for opencode),
+     LOOP_TIER / REPORT_TIER (default 'go' — the only tier with a key on this host; ZEN
+     models are dead here; 'go' or 'zen'; only for opencode),
      OPENCODE_SHARE (1=attach web server + public --share link), REPORT_SESSION, REPORT_DIR.
 """
 import os, subprocess, sys, time
@@ -24,9 +25,9 @@ SESSION = os.environ.get("REPORT_SESSION", "cc-ci-report")
 WORKDIR = os.environ.get("REPORT_DIR", "/srv/cc-ci")
 LOG_DIR = os.environ.get("LOG_DIR", "/srv/cc-ci/.cc-ci-logs")
 BACKEND = os.environ.get("REPORT_BACKEND", "opencode")
-TIER = os.environ.get("LOOP_TIER", os.environ.get("REPORT_TIER", "zen"))
+TIER = os.environ.get("LOOP_TIER", os.environ.get("REPORT_TIER", "go"))
 _TIER_MODEL = {"go": "opencode-go/glm-5.2", "zen": "opencode/glm-5.2"}
-_DEFAULT_MODEL = _TIER_MODEL.get(TIER, _TIER_MODEL["zen"]) if BACKEND == "opencode" else "opus"
+_DEFAULT_MODEL = _TIER_MODEL.get(TIER, _TIER_MODEL["go"]) if BACKEND == "opencode" else "opus"
 MODEL   = os.environ.get("REPORT_MODEL", _DEFAULT_MODEL)
 CLAUDE_BIN   = os.environ.get("CLAUDE_BIN", "claude")
 CLAUDE_FLAGS = os.environ.get("CLAUDE_FLAGS", "--dangerously-skip-permissions")

cc-ci-plan/launch-upgrader.py

@@ -44,11 +44,13 @@ WORKDIR = os.environ.get("UPGRADER_DIR",     "/srv/cc-ci")
 LOG_DIR = os.environ.get("LOG_DIR",          "/srv/cc-ci/.cc-ci-logs")
 
 # LOOP_BACKEND / LOOP_MODEL take precedence (unified control from the operator).
-# LOOP_TIER selects the OpenCode subscription: "zen" (default) or "go". Go hit a monthly
-# usage limit; ZEN is the working alternative. Set LOOP_TIER=go when the Go limit resets.
+# LOOP_TIER selects the OpenCode subscription: "go" (DEFAULT — the only tier with a key on
+# this host) or "zen" (opt-in on a host that has a ZEN key). ZEN models are DEAD here — the
+# endpoint errors server-side, and env loss in the tmux→agent→launch-report chain is exactly
+# how it silently ate the 2026-09-18 weekly report. Never default back to zen.
 # Only affects the opencode backend; claude ignores the tier.
 BACKEND = os.environ.get("LOOP_BACKEND", os.environ.get("UPGRADER_BACKEND", "opencode"))
-TIER = os.environ.get("LOOP_TIER", os.environ.get("UPGRADER_TIER", "zen"))
+TIER = os.environ.get("LOOP_TIER", os.environ.get("UPGRADER_TIER", "go"))
 _TIER_CFG = {
     "go":  {"model": "opencode-go/glm-5.2", "endpoint": "https://opencode.ai/zen/go/v1/chat/completions",
             "auth": "opencode-go", "label": "OpenCode Go"},
@@ -58,7 +60,7 @@ _TIER_CFG = {
 if BACKEND == "opencode" and TIER not in _TIER_CFG:
     print(f"[upgrader] ERROR: unknown LOOP_TIER '{TIER}' — use 'go' or 'zen'", flush=True)
     sys.exit(1)
-_tier = _TIER_CFG.get(TIER, _TIER_CFG["zen"])
+_tier = _TIER_CFG.get(TIER, _TIER_CFG["go"])
 _DEFAULT_MODEL = _tier["model"] if BACKEND == "opencode" else "sonnet"
 MODEL   = os.environ.get("LOOP_MODEL",   os.environ.get("UPGRADER_MODEL",   _DEFAULT_MODEL))
 

cc-ci-plan/plan-domain-migration-ci-autonomic-zone.md

@@ -0,0 +1,172 @@
+# Plan: migrate cc-ci domains from `*.commoninternet.net` to `ci.autonomic.zone`
+
+**Date:** 2026-09-21 · **Status:** DRAFT — blocked only on operator DNS access (Gandi zone `autonomic.zone`)
+**Principle:** same host, same IP (195.201.88.249), same services — only names change. Dual-run new
+alongside old; old names retire last; rollback at any point is "change nothing further" because the
+old DNS zone keeps answering until Phase 4.
+
+---
+
+## 1. Inventory — what answers on what today
+
+| Name | Served by | Router/config source |
+|---|---|---|
+| `ci.commoninternet.net` | results dashboard + bridge `/hook` | `cc-ci/nix/modules/dashboard.nix`, `bridge.nix` (traefik Host rules) |
+| `report.ci.commoninternet.net` | weekly report static site | `cc-ci/nix/modules/reports.nix` |
+| `drone.ci.commoninternet.net` | Drone server | `cc-ci/nix/modules/drone.nix` (abra app; env `DOMAIN`, `DRONE_SERVER_HOST`); runner RPC: `drone-runner.nix` |
+| `oc.ci.commoninternet.net` | opencode UI (traefik → host nginx basic auth) | `cc-ci-orch/nix/modules/orchestrator-host.nix` (`opencodeUiHost`), set in `nix/hosts/cc-ci/configuration.nix` |
+| `traefik.ci.commoninternet.net` | traefik stack identity + deploy-proxy health gate | `cc-ci/nix/modules/proxy.nix` |
+| `warm-*.ci.commoninternet.net` | warm canonical recipe stacks | `warm-keycloak.nix`, warm-bluesky-pds (caddy labels) |
+| `<tag>-<6hex>.ci.commoninternet.net` | per-run recipe test deploys | `cc-ci/runner/harness/naming.py` `app_domain()` — **resolves via public DNS** (containers use 1.1.1.1/8.8.8.8) |
+| `backups.ci.commoninternet.net` | backup-bot-two — stack name only, no web route | `backupbot.nix` |
+| `acme.commoninternet.net` (NS `ns-acme`) | local acme-dns (systemd, port 53) | `cc-ci/nix/modules/acme-dns.nix` |
+
+**Cert chain today:** nixos `security.acme` (lego) issues `ci.commoninternet.net` + `*.ci.commoninternet.net`
+via DNS-01 through the local acme-dns; Gandi delegates `_acme-challenge.ci.commoninternet.net` CNAME →
+`9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net`. `postRun` fires
+`cc-ci-acme-traefik-handoff.service` → installs cert into `/var/lib/ci-certs/live` → restarts
+`deploy-proxy.service`, whose reconciler re-derives the `traefik_ci_commoninternet_net_ssl_{cert,key}_v<hash>`
+swarm secrets (traefik runs co-op-cloud wildcard/no-ACME mode off those secrets).
+**DNS:** both zones on Gandi. IPv4-only (no AAAA records anywhere; none needed).
+
+**External references:** Gitea OAuth app for Drone (redirect URI → drone.ci…); per-repo webhooks →
+`/hook` — **only `recipe-maintainers/ghost` and `recipe-maintainers/discourse` have any** (everything
+else is bridge-driven via the Drone API); ~60 doc references across cc-ci-orch (AGENTS.md, README,
+plan.md, skills).
+
+## 2. Target naming
+
+| Old | New |
+|---|---|
+| ci.commoninternet.net | ci.autonomic.zone |
+| report.ci.commoninternet.net | report.ci.autonomic.zone |
+| drone.ci.commoninternet.net | drone.ci.autonomic.zone |
+| oc.ci.commoninternet.net | oc.ci.autonomic.zone |
+| traefik.ci.commoninternet.net | traefik.ci.autonomic.zone |
+| *.ci.commoninternet.net (warm-\*, per-run apps, backups identity) | *.ci.autonomic.zone |
+| acme.commoninternet.net | **stays** through Phases 1–2 (infra, not a front door); optional Phase 3 |
+
+Swarm **stack names** (`traefik_ci_commoninternet_net`, `ccci-*`, warm stacks) are identifiers, not
+domains — keep them where possible to avoid churn in keep-lists/prune filters, EXCEPT abra-managed
+apps whose name **is** the domain (drone, backup-bot, per-run apps).
+
+## 3. DNS records to set (zone `autonomic.zone`, Gandi — operator)
+
+| # | Name | Type | Value | TTL | Purpose |
+|---|---|---|---|---|---|
+| 1 | `ci.autonomic.zone` | A | `195.201.88.249` | 300 | apex front door (wildcard does NOT cover the apex) |
+| 2 | `*.ci.autonomic.zone` | A | `195.201.88.249` | 300 | report/drone/oc/traefik/warm-\*/`<tag>-<6hex>` run apps |
+| 3 | `_acme-challenge.ci.autonomic.zone` | CNAME | `9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` | 300 | DNS-01 for the new wildcard — **reuses the existing restricted acme-dns account** (`disable_registration=true`; the lego *account* authenticates, not the FQDN, so one acme-dns subdomain serves both zones) |
+
+Also: lower TTL to 300 on the old `ci`/`*.ci` records a few days ahead (fast rollback later).
+Phase 3 (optional) adds: `acme.ci.autonomic.zone` + `ns-acme.ci.autonomic.zone` A records.
+
+**DNS verification (before touching the host):**
+```bash
+dig +short ci.autonomic.zone A @ns-187-a.gandi.net   # 195.201.88.249
+dig +short report.ci.autonomic.zone @1.1.1.1          # 195.201.88.249 (via wildcard)
+dig +short drone.ci.autonomic.zone @8.8.8.8           # 195.201.88.249
+dig +short _acme-challenge.ci.autonomic.zone CNAME    # 9d3f0428-….acme.commoninternet.net.
+```
+
+## 4. Phase 1a — certificate (issue BEFORE any routing change)
+
+1. `cc-ci/nix/modules/acme-dns.nix`: add a second `security.acme.certs."ci.autonomic.zone"` entry
+   (domain + `*.ci.autonomic.zone` SAN, `dnsProvider = "acmedns"`, `dnsResolver = "1.1.1.1:53"`).
+   Split the lego env per cert — today `ACME_DNS_STORAGE_PATH` is baked to
+   `/var/lib/acme/ci.commoninternet.net/…`; generate one env file per cert so each keeps its own
+   acme-dns account file.
+2. Extend `cc-ci-acme-traefik-handoff` to stage the new cert alongside the old
+   (`/var/lib/ci-certs/live` holds both, e.g. `fullchain.pem` = new + `fullchain-old.pem`).
+3. Extend the **deploy-proxy reconciler** (`proxy.nix`) for **dual-cert SNI**: new wildcard pair as
+   `ssl_cert/ssl_key` (default store) + old pair as a second secret pair wired through a small
+   file-provider `[[tls.certificates]]` so `*.ci.commoninternet.net` keeps serving during the bake
+   window. This is the one fiddly bit — validate against the **Let's Encrypt staging** endpoint
+   first (the module already has the staging-marker pattern).
+4. `nixos-rebuild test` → confirm both certs issued and served (`openssl s_client -servername` for a
+   name in each zone, check SANs) → `switch`.
+
+## 5. Phase 1b — routing + service config (one stack at a time, health-gated)
+
+During the bake window every router rule carries **both** hostnames
+(`Host(`ci.autonomic.zone`) || Host(`ci.commoninternet.net`)`); old names drop out only in Phase 4.
+
+- `dashboard.nix` + `bridge.nix` (share the `ci` host: dashboard `/` + bridge `/hook`), `reports.nix`,
+  and the traefik health-gate name in `proxy.nix`.
+- `bridge.py` / `dashboard.py` code defaults for `DRONE_URL` / `DASH_URL`.
+- `recipe-report.py` — footer, publish URL text and its verification curl → `report.ci.autonomic.zone`
+  (regenerate the index after, so the home page stops linking the old name).
+- Drone **abra app rename** (name = domain, so this is a new stack):
+  1. Export enabled-repo list from the old Drone API.
+  2. Deploy new abra app `drone.ci.autonomic.zone` (same module; secrets `rpc_secret`,
+     `client_secret` re-insert from `/run/secrets` as today). **Fresh DB** — Drone data volume is
+     stack-name-prefixed; repos re-sync on Gitea login, build history is lost (accepted; export the
+     old repo list and re-enable via API).
+  3. Point `drone-runner.nix` `DRONE_RPC_HOST` at the new name; restart runner.
+  4. Re-run `scripts/bootstrap-drone-oauth.sh` with `DRONE=https://drone.ci.autonomic.zone`
+     (updates the Gitea OAuth redirect URI; script already uses `--resolve` to hit loopback).
+  5. Update the two webhooks (ghost, discourse) to `https://drone.ci.autonomic.zone/hook?secret=<same>`
+     — read the secret off the existing hook config before deleting it.
+  6. `abra app undeploy` + remove the old drone app once green.
+- `configuration.nix`: `opencodeUiHost = "oc.ci.autonomic.zone"`; extend the `networking.hosts`
+  self-pin list with the new names (keep old pins during bake).
+- Gitea `GITEA_SERVER` stays `git.autonomic.zone` — unchanged.
+
+## 6. Phase 2 — harness / warm / per-run naming (the `_ci_commoninternet_net` blast radius)
+
+- `cc-ci/runner/harness/naming.py` `app_domain()` → `.ci.autonomic.zone` (single source of truth).
+- Regexes/patterns that match the derived names: `docker-prune.nix` run-app filter,
+  `tests/concurrency/concutil.py` (`DOMAIN` const + `RUN_APP_RE`), any separate `RUN_APP_RE` in
+  `lifecycle.py` / `warm.py` / `warm_reconcile.py` / `canonical.py` / `generic.py` / `http.py`,
+  `nightly-sweep` keep-list patterns, `tests/gitea/recipe_meta.py` comment.
+- Redeploy warm stacks under new names (abra `warm-<recipe>.ci.autonomic.zone`), reconcile, drain old.
+- `backupbot.nix` `DOMAIN` (stack-name-only change).
+- Verify: full deterministic test suite + one live `!testme` run end-to-end.
+
+## 7. Phase 3 (optional) — move acme-dns off commoninternet.net
+
+Only if `commoninternet.net` is being dropped entirely: new `domain`/`nsname` in acme-dns config
+(`acme.ci.autonomic.zone` / `ns-acme.ci.autonomic.zone` + Gandi records), briefly re-enable
+registration or hand-migrate the account DB, re-point `_acme-challenge` CNAMEs, re-issue both certs.
+
+## 8. Phase 4 — retirement (after ≥7 clean days)
+
+Remove old names from the dual router rules → remove `security.acme.certs."ci.commoninternet.net"`
++ its env file + old cert secret pair → drop old `networking.hosts` pins → delete old Gandi records
+(`ci`, `*.ci`, `_acme-challenge.ci`) → restore TTLs to 3600. Keep `acme.commoninternet.net` unless
+Phase 3 was done.
+
+## 9. Verification checklist
+
+- **DNS:** the dig table in §3 against a Gandi NS and a public resolver.
+- **Cert:** `openssl s_client -connect 195.201.88.249:443 -servername <name>` for one name per zone —
+  SANs correct for both, `ssl_verify_result=0`.
+- **Front doors:** `ci` → 200 · `report.ci` → 200 · `drone.ci` → 200/303 · `oc.ci` → **401**
+  (basic auth intact) · `traefik.ci/api/version` → 200.
+- **SSO:** Drone login through Gitea on the new redirect URI.
+- **Webhook:** empty commit to `recipe-maintainers/ghost` main → build fires on the new host.
+- **Bridge/!testme:** `!testme` on a scratch PR → build runs, `results.json` written, PR comment
+  contains **new** domain links.
+- **Report:** publish one report page and verify it at `https://report.ci.autonomic.zone`.
+- **Host:** `systemctl --failed` empty; `deploy-proxy` reconcile logs clean.
+
+## 10. Risks / gotchas
+
+- **Dual-cert SNI in the traefik reconciler** is the highest-risk change — stage-first, verify SNI
+  selection for both zones before switching.
+- **Drone DB** does not follow the abra rename — accept fresh build history; export/re-apply the
+  enabled-repo list; re-insert secrets from `/run/secrets` (module already does).
+- **Per-run recipe domains resolve via public DNS** — the `*.ci.autonomic.zone` wildcard A record is
+  load-bearing for every test deploy; never merge Phase 2 before it propagates.
+- Old PR comments/badges link old names — they keep working until Phase 4 removes the old records.
+- ~60 doc references (AGENTS.md, README, plan.md, skills, status skill) — one cosmetic doc PR at the
+  end so the docs don't lie.
+- No domain-bearing secret values were found (webhook secrets are per-repo and copied as-is).
+
+## 11. Sequencing
+
+1. **Operator:** TTLs down + 3 Gandi records (§3) — 5 minutes.
+2. **Agent:** cert PR (§4) → `test` → verify dual SNI → `switch`.
+3. **Agent:** routing/env PRs (§5) stack-by-stack, health-gated; OAuth + webhooks + drone rename.
+4. **Agent:** end-to-end verification (§9) + report republish under the new name.
+5. **Bake ≥7 days** → Phase 2 PR → full suite → bake → Phase 4 retirement.

cc-ci-plan/recipe-report.py

@@ -217,7 +217,8 @@ def _table(rows, repo_url=None):
         return ""
     # TESTS = the CI/test verdict (GREEN/FAILED/STALE/…); STATUS = the PR's LIVE state, fetched
     # client-side (open vs ✓ for any not-open state) by the inline script in _page().
-    head = ("<tr><th>Recipe</th><th>Change</th><th>TESTS</th><th>CVEs</th>"
+    head = ("<tr><th>Recipe</th><th>Change</th><th>TESTS</th>"
+            '<th title="Known CVEs fixed by the available recipe upgrade; zero does not certify the current image as vulnerability-free">CVEs fixed</th>'
             "<th>CI</th><th>PR</th><th>STATUS</th><th>Notes</th></tr>")
     trs = []
     for r in rows:
@@ -303,7 +304,7 @@ def render(spec_path, out_path):
                        for p in lead.split("\n\n") if p.strip())
     body = (_mast() +
             f'<div class="dateline"><span>{_esc(sub)}</span>'
-            f'<span>report.ci.commoninternet.net</span><span>{gen}</span></div>'
+            f'<span>report.ci.autonomic.zone</span><span>{gen}</span></div>'
             f'<div class="lead">{lead}</div>')
     # 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column.
     wire = ("The full wire — every recipe, in priority order" if kind == "week"
@@ -322,7 +323,7 @@ def render(spec_path, out_path):
     if s.get("changes"):
         body += f'<h2>What changed</h2>{_changes(s.get("changes"), repo_url)}'
     body += (f'<footer>{title} · generated {gen} · '
-             f'<a href="https://ci.commoninternet.net/">dashboard</a> · <a href="./">archive</a></footer>')
+             f'<a href="https://ci.autonomic.zone/">dashboard</a> · <a href="./">archive</a></footer>')
     open(out_path, "w").write(_page(f"{title} · " + s["date"], body))
     print("wrote", out_path)
 
@@ -347,10 +348,10 @@ def publish(html_path, date, kind="week"):
         for d, k in sorted(set(entries), reverse=True))
     idx = _page("The Recipe Report — Archive", _mast() +
                 '<div class="dateline"><span>Weekly review of Co-op Cloud recipe upgrades &amp; CI</span>'
-                '<span>report.ci.commoninternet.net</span></div>'
+                '<span>report.ci.autonomic.zone</span></div>'
                 f'<ul class="idx">{lis or "<li><em>No reports yet.</em></li>"}</ul>')
     subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/index.html"], input=idx.encode(), check=True)
-    print(f"published https://report.ci.commoninternet.net/{page}  (+ index)")
+    print(f"published https://report.ci.autonomic.zone/{page}  (+ index)")
 
 
 def main():

cc-ci-plan/resolve-images.py

@@ -163,7 +163,8 @@ def parse_ref(ref: str) -> dict:
     return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
 
 
-HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10"))
+HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "40"))
+_HUB_TRUNCATED: set[str] = set()
 
 
 def _hub_tag_exists(repo: str, tag: str) -> bool:
@@ -177,10 +178,10 @@ def _hub_tag_exists(repo: str, tag: str) -> bool:
 def _hub_tags(repo: str) -> list[str]:
     """Recently-pushed tags, newest first.
 
-    Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical —
-    but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so
-    ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade.
-    (ghcr offers no ordering, which is why that path needs a different strategy.)
+    Popular Docker Hub repos carry many thousands of tags, so enumerate a bounded recent window.
+    A rebuilt old tag can displace a newer version from that window, so callers MUST treat a
+    truncated listing with no detected upgrade as unverified, not current. (ghcr offers no recency
+    ordering, which is why that path needs a different strategy.)
     """
     tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
                      f"?page_size=100&ordering=last_updated")
@@ -191,6 +192,8 @@ def _hub_tags(repo: str) -> list[str]:
         url = d.get("next")
         if not url:
             break
+    if url:
+        _HUB_TRUNCATED.add(repo)
     return tags
 
 
@@ -236,15 +239,26 @@ def _oci_tags(host: str, repo: str) -> list[str]:
     hdrs = _oci_bearer(host, repo)
     tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
     for _ in range(MAX_TAG_PAGES):
-        req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
-        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
-            tags += (json.load(r) or {}).get("tags") or []
-            link = r.headers.get("Link", "") or ""
+        for attempt in range(RETRIES):
+            req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
+            try:
+                with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
+                    tags += (json.load(r) or {}).get("tags") or []
+                    link = r.headers.get("Link", "") or ""
+                break
+            except urllib.error.HTTPError as e:
+                if e.code not in (429, 503) or attempt == RETRIES - 1:
+                    raise
+                retry_after = e.headers.get("Retry-After") or ""
+                delay = min(int(retry_after) if retry_after.isdigit() else 2 ** (attempt + 1), 60)
+                time.sleep(delay)
         m = re.search(r'<([^>]+)>;\s*rel="next"', link)
         if not m:
             break
         nxt = m.group(1)
         url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
+    else:
+        raise RuntimeError(f"tag listing truncated after {MAX_TAG_PAGES} pages")
     return tags
 
 
@@ -299,6 +313,39 @@ def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
     return cands
 
 
+def _oci_tag_exists(host: str, repo: str, tag: str) -> bool:
+    """Verify a release candidate is actually published as an image manifest."""
+    hdrs = {"User-Agent": UA,
+            "Accept": "application/vnd.oci.image.index.v1+json, "
+                      "application/vnd.docker.distribution.manifest.list.v2+json, "
+                      "application/vnd.oci.image.manifest.v1+json",
+            **_oci_bearer(host, repo)}
+    url = f"https://{host}/v2/{repo}/manifests/{urllib.parse.quote(tag, safe='')}"
+    try:
+        with urllib.request.urlopen(urllib.request.Request(url, headers=hdrs, method="HEAD"),
+                                    timeout=TIMEOUT) as response:
+            return response.status == 200
+    except urllib.error.HTTPError as e:
+        if e.code == 404:
+            return False
+        raise
+
+
+def _release_manifest_fallback(registry: str, repo: str, current: str) -> tuple[list[str], str] | None:
+    """For huge GHCR registries, use release tags only after checking actual manifests."""
+    for owner, name in _release_fallback_repos(registry, repo):
+        releases = github_release_tags(owner, name)
+        if current not in releases:
+            continue  # incomplete release window; it cannot establish a current baseline
+        candidates = sorted((tag for tag in set(releases)
+                             if shape(tag) == shape(current) and vkey(tag) >= vkey(current)),
+                            key=vkey, reverse=True)
+        for tag in candidates:
+            if _oci_tag_exists(registry, repo, tag):
+                return [current, tag], f"github-releases+registry-manifest:{owner}/{name}"
+    return None
+
+
 _TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
 
 
@@ -335,8 +382,16 @@ def resolve(ref: str) -> dict:
            "upgrade_available": False}
     tags, err = list_tags(info["registry"], info["repo"])
     if err:
-        out["status"] = f"error: {err}"
-        return out
+        try:
+            fallback = (_release_manifest_fallback(info["registry"], info["repo"], info["tag"])
+                        if "tag listing truncated" in err else None)
+        except Exception as e:  # noqa: BLE001 — a failed fallback must stay unresolved
+            out["status"] = f"error: registry {err}; release fallback {type(e).__name__}: {e}"
+            return out
+        if not fallback:
+            out["status"] = f"error: {err}"
+            return out
+        tags, out["source"] = fallback
     out["tags_seen"] = len(set(tags))
     # INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
     # the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
@@ -367,8 +422,9 @@ def resolve(ref: str) -> dict:
     same = [t for t in set(tags) if shape(t) == want and vkey(t)]
     out["candidates"] = len(same)
     if not same:
-        # Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings.
-        out["status"] = "no comparable tags (shape has no numeric siblings)"
+        # A floating or unusual tag cannot prove that an image is current. Keep it visible to
+        # callers rather than letting an empty candidate set become a false UPTODATE verdict.
+        out["status"] = "unverifiable: no comparable numeric tags"
         return out
     newest = max(same, key=vkey)
     out["newest_same_shape"] = newest
@@ -377,6 +433,10 @@ def resolve(ref: str) -> dict:
         if within:
             out["newest_within_major"] = max(within, key=vkey)
     out["upgrade_available"] = bool(cur and vkey(newest) > cur)
+    if (info["registry"] in ("docker.io", "registry-1.docker.io")
+            and info["repo"] in _HUB_TRUNCATED and not out["upgrade_available"]):
+        out["status"] = (f"unverifiable: Docker Hub listing truncated after "
+                         f"{HUB_RECENT_PAGES * 100} recent tags")
     return out
 
 
@@ -436,16 +496,16 @@ def main() -> int:
         "recipe": a.recipe,
         "images": results,
         "upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
-        "unresolved": [r["ref"] for r in results if r["status"].startswith("error")],
+        "unresolved": [r["ref"] for r in results if r["status"].startswith(("error", "skipped", "unverifiable"))],
         # The whole point: distinguish "checked, current" from "could not check".
-        "all_resolved": not any(r["status"].startswith("error") for r in results),
+        "all_resolved": bool(results) and all(r["status"] == "ok" for r in results),
     }
     if not a.table:
         print(json.dumps(report, indent=2))
         return 0
     print(f"{a.recipe} — {len(results)} images")
     for r in results:
-        flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current")
+        flag = "UPGRADE" if r["upgrade_available"] else ("UNKNOWN" if r["status"] != "ok" else "current")
         print(f"  [{flag:7}] {r['repo']}:{r['tag']}" + ("  (digest-pinned)" if r["digest"] else ""))
         print(f"            shape={r['shape']}  candidates={r['candidates']}"
               f"  newest_same_shape={r['newest_same_shape']}"

cc-ci-plan/test-advisory-scan.py

@@ -22,6 +22,7 @@ import io
 import json
 import os
 import pathlib
+import tempfile
 import sys
 import unittest
 import unittest.mock
@@ -153,6 +154,12 @@ class TestRegistryUrls(unittest.TestCase):
         p.write_text(text)
         return A.registry_urls("r", tmp)
 
+    def test_prose_ellipsis_is_not_a_security_source(self):
+        with tempfile.TemporaryDirectory() as directory:
+            urls, _ = self._write(directory,
+                                  "https://github.com/cryptpad/cryptpad and example `https://…` URLs")
+        self.assertEqual(urls, ["https://github.com/cryptpad/cryptpad"])
+
     def test_strips_trailing_markdown_punctuation(self):
         # Production defect: a captured backtick 404'd the fetch and rendered n8n/immich as '?'.
         import tempfile
@@ -290,12 +297,14 @@ class TestPerImageWindows(unittest.TestCase):
         self.assertEqual(rep["windows"]["github-advisories:discourse/discourse"],
                          {"from": "3.5.3", "to": "3.5.4"})
 
-    def test_unmatched_image_name_is_silently_ignored(self):
-        # Documents CURRENT behaviour: a typo'd name costs coverage without warning.
+    def test_unmatched_image_name_makes_count_unknown(self):
+        # A typo'd sidecar name must not silently turn its CVEs into a clean zero.
         rep = run_scan([self.APP], v_from="3.5.3", v_to="3.5.4",
                        images=[("nosuchimage", "1", "2")], urls=self.URLS[:1])
         self.assertEqual(list(rep["windows"]), ["github-advisories:discourse/discourse"])
-        self.assertTrue(rep["count_known"])
+        self.assertFalse(rep["count_known"])
+        self.assertEqual(rep["unmatched_image_windows"], ["nosuchimage"])
+        self.assertIsNone(rep["cve_count_fixed"])
 
 
 class TestSchemeChangeDateFallback(unittest.TestCase):
@@ -377,6 +386,13 @@ class TestCountGuarantees(unittest.TestCase):
                        v_from="1.0", v_to="1.1")
         self.assertIn("github-advisories:app/app", rep["sources_failed"])
         self.assertIn("FAILED", A.markdown(rep))
+        self.assertFalse(rep["count_known"])
+
+    def test_only_404_feed_is_not_evidence_of_clean_scan(self):
+        rep = run_scan([gh("app/app", [], status="no-advisories-published")],
+                       v_from="1.0", v_to="1.1")
+        self.assertTrue(rep["no_usable_sources"])
+        self.assertFalse(rep["count_known"])
 
     def test_no_window_given_classifies_nothing(self):
         rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")])])

cc-ci-plan/test-resolve-images.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""Offline regressions for the weekly image survey's false-current cases."""
+
+import importlib.util
+import pathlib
+import unittest
+from unittest import mock
+
+HERE = pathlib.Path(__file__).resolve().parent
+spec = importlib.util.spec_from_file_location("resolve_images", HERE / "resolve-images.py")
+resolver = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(resolver)
+
+
+class TestResolverCoverage(unittest.TestCase):
+    def test_cryptpad_calver_tag_is_found_independently_of_abra(self):
+        with mock.patch.object(resolver, "list_tags", return_value=(
+            ["version-2026.5.1", "version-2026.10.0", "opendesk-2026.08.04"], None)):
+            row = resolver.resolve("cryptpad/cryptpad:version-2026.5.1")
+        self.assertTrue(row["upgrade_available"])
+        self.assertEqual(row["newest_same_shape"], "version-2026.10.0")
+
+    def test_floating_and_templated_tags_cannot_prove_current(self):
+        with mock.patch.object(resolver, "list_tags", return_value=(["alpine", "1.2.3"], None)):
+            floating = resolver.resolve("atmoz/sftp:alpine")
+        templated = resolver.resolve("ghost:${IMAGE_VERSION}-alpine")
+        self.assertTrue(floating["status"].startswith("unverifiable:"))
+        self.assertTrue(templated["status"].startswith("skipped:"))
+
+    def test_truncated_registry_listing_cannot_prove_current(self):
+        with mock.patch.object(resolver, "list_tags", return_value=(["1.2.3"], None)):
+            with mock.patch.object(resolver, "_HUB_TRUNCATED", {"library/example"}):
+                row = resolver.resolve("example:1.2.3")
+        self.assertTrue(row["status"].startswith("unverifiable:"))
+
+    def test_huge_ghcr_listing_uses_only_published_release_manifests(self):
+        ref = "ghcr.io/immich-app/immich-machine-learning:v3.3.5"
+        with mock.patch.object(resolver, "list_tags", return_value=([], "RuntimeError: tag listing truncated")), \
+             mock.patch.object(resolver, "github_release_tags",
+                               return_value=["v3.3.6", "v3.3.5"]), \
+             mock.patch.object(resolver, "_oci_tag_exists", side_effect=lambda h, r, t: t == "v3.3.5"):
+            row = resolver.resolve(ref)
+        self.assertEqual(row["status"], "ok")
+        self.assertFalse(row["upgrade_available"])  # release exists; image does not yet
+        self.assertIn("registry-manifest", row["source"])
+        with mock.patch.object(resolver, "list_tags", return_value=([], "RuntimeError: tag listing truncated")), \
+             mock.patch.object(resolver, "github_release_tags",
+                               return_value=["v3.3.6", "v3.3.5"]), \
+             mock.patch.object(resolver, "_oci_tag_exists", return_value=True):
+            upgraded = resolver.resolve(ref)
+        self.assertTrue(upgraded["upgrade_available"])
+
+
+if __name__ == "__main__":
+    unittest.main()

cc-ci-plan/upstream/cryptpad.md

@@ -2,11 +2,26 @@
 
 | service | image | source repo | releases / changelog |
 |---------|-------|-------------|----------------------|
-| app     | cryptpad/cryptpad | https://github.com/cryptpad/cryptpad | https://github.com/cryptpad/cryptpad/releases |
+| app     | cryptpad/cryptpad | https://github.com/cryptpad/cryptpad | https://github.com/cryptpad/cryptpad/releases · https://hub.docker.com/r/cryptpad/cryptpad/tags |
 | web     | nginx             | https://github.com/nginx/nginx       | https://nginx.org/en/CHANGES |
 
 ## Standing notes
 - `abra recipe upgrade` cannot parse the `version-YYYY.M.D` tag format of the cryptpad/cryptpad image,
   so the app image is bumped by hand; the nginx sidecar is what `abra recipe upgrade` actually moves.
+- Only use tags in the canonical `version-YYYY.M.x` scheme. The `opendesk-YYYY.MM.DD` tags on
+  Docker Hub are openDesk-specific builds, not newer CryptPad releases — e.g. `opendesk-2026.08.04`
+  (built 2026-08-04) still contains CryptPad `2026.5.1` in `/cryptpad/package.json`.
+- CryptPad's GitHub releases and Docker Hub tags are not simultaneous: `2026.10.0` was released
+  2026-10-21; `version-2026.10.0` appeared on Docker Hub on 2026-10-23 (first seen by the weekly
+  upgrade run on 2026-10-30). Re-check the Docker Hub tags page before treating a GitHub release as
+  available for the recipe.
+- The 2026.10.0 release is a breaking one for the recipe: full server rewrite (separate
+  `cryptpad-server` package), split `config.js` → `config/infra.js` (public origin/ports move),
+  one proxy port for HTTP + websocket (no separate `3003` websocket port), SSO plugin v1.0.0
+  required, and Euro-Office 9.3 replacing the OnlyOffice editors. Recipe handling in the
+  `upgrade-2026.10.0` PR: `CPAD_MAIN_DOMAIN`/`CPAD_SANDBOX_DOMAIN` must be full `https://…` URLs
+  (the new server assigns them verbatim to `infra.public.origin`/`sandboxOrigin`; the image's
+  entrypoint auto-creates `config/infra.js`), nginx `/cryptpad_websocket` proxies to `:3000`,
+  `abra.sh` config/nginx/SSO-entrypoint version labels bumped, SSO default 1.0.0.
 - nginx is HTTP/1.1-only here (sidecar on :8083, `proxy_http_version 1.1`), so HTTP/2/3 CVE changes in
   nginx releases generally don't affect this recipe — but still note them.

cc-ci-plan/upstream/custom-html.md

@@ -7,8 +7,10 @@
 
 ## Standing notes
 - nginx even-numbered minor versions (1.28.x, 1.30.x) are mainline; odd (1.27.x, 1.29.x) are stable. The recipe tracks mainline.
+- nginx 1.31.7 (14 Oct 2026) is a security patch release with FIVE security fixes: CVE-2026-15176 (buffer overflow in a worker using `alias` when the URI was changed by `rewrite`), CVE-2026-93479 (buffer overflow evaluating string expressions with variables — affects document-root calculation and request logging in http/stream), CVE-2026-95101 (HTTP/3: part of an unknown frame treated as request body — bypasses `client_max_body_size` / injects into proxied request), CVE-2026-97629 (TLS session-ticket key rotation left one key uninitialized — session forgery), CVE-2026-97205 (use-after-free with async file I/O when an SSI page has `include virtual`). Features: geoip MMDB support, backend SSL cert verification by IP, QUIC reconfiguration + binary upgrade with `quic_bpf`. Change: `ngx_http_v2_module` now uses RFC 9218 prioritisation (RFC 7540 priority signals ignored). Bugfixes (X-Accel-Redirect cached response segfault, dav COPY/MOVE with alias segfault, QUIC flood false positives, SSL close_notify handling). NONE affect the custom-html static-file-serving use case (no HTTP/3/QUIC, no AIO/SSI, no `rewrite`+`alias` path mapping, no session-ticket rotation config) — but the 5 CVEs are why this bump is warranted. No breaking config changes; no operator action beyond the image pull.
 - compose.git-pull.yml is an optional overlay for git-pull functionality; alpine/git version lives there.
 - compose.sftp.yml and compose.sso.yml are other optional overlays; linuxserver/openssh-server uses `latest` tag and is not version-pinned.
+- nginx 1.31.6 (15 Sep 2026) is a security patch release: CVE-2026-90439 (heap buffer overflow in a worker under certain configurations when using HTTP/3 with OpenSSL 3.5.0 and earlier). Change: QUIC transport parameters extension received in an SSL connection is now always ignored. Bugfixes: binary upgrade with control API socket + perl module; predicate-location evaluation error ignored; nested location lookup with regex/predicate locations; segfault reading config with `geo` … `ranges` + corrupted binary base file. None affect the custom-html static-file-serving use case (no HTTP/3/QUIC, no predicate locations, no perl, no `geo`).
 - Breaking change in nginx 1.31.0: HTTP/2 and HTTP/3 requests with Connection/Proxy-Connection/Keep-Alive/Transfer-Encoding/Upgrade/TE headers are now rejected.
 - Breaking change in nginx 1.29.7: keepalive in upstream block is enabled by default; proxy_http_version changed to 1.1; Connection proxy header no longer sent by default.
 - nginx 1.31.3 (15 Jul 2026) is a security patch release: CVE-2026-42533 (heap buffer overflow in `map` directive with regex matching / non-cacheable variable in string expression), CVE-2026-60005 (uninitialized memory access with unnamed regex captures + `slice` directive / background cache update → memory disclosure or worker crash), CVE-2026-56434 (use-after-free in `ngx_http_ssi_filter_module` processing crafted proxied backend response). Two non-security `Change`s: HTTP/2 response header/trailer size now limited by `proxy_buffer_size`/`grpc_buffer_size` in `ngx_http_proxy_v2_module`+`ngx_http_grpc_module`; external-entity loading disabled by default in `ngx_http_xslt_filter_module` (new `xml_external_entities` directive re-enables it). Plus bugfixes (HTTP/2 out-of-order pseudo-headers, flow control, Upgrade header in HTTP/2/3 responses). NONE of these changes affect the custom-html static-file-serving use case (no `map`, no `slice`/cache, no SSI, no xslt, no proxy_v2/grpc). No breaking config changes — no operator action required beyond the image pull.

cc-ci-plan/upstream/gitea.md

@@ -41,3 +41,7 @@ in this file, so keep vendor security pages listed here.
 - **1.26.3**: carries a regression (#38177 "context deadline exceeded" opening repo code pages) — upstream says upgrade straight to 1.26.4, skip 1.26.3.
 - **1.27.0** (released 2026-07-13, MINOR with BREAKING changes): `Feat(actions)!: improve support for reusable workflows` (#37478) and `Use Content-Security-Policy: script nonce` (#37232, may break custom inline JS / reverse-proxy CSP). Many security + feature additions. A dedicated minor-bump run should evaluate CSP / reusable-workflow impact before adopting.
 - **1.27.1** (released 2026-07-27, PATCH): security fix (oauth2 mandatory 2FA enforcement on authorize/grant endpoints, #38606), API swagger alignment, diff-contrast enhancement, and many bugfixes (actions reusable-workflow/job-stranding, OIDC end-session, repo-deletion cleanup, webhook/mail). No breaking changes; safe patch bump from 1.27.0. Release notes: https://github.com/go-gitea/gitea/releases/tag/v1.27.1
+- **28.0.0** (released 2026-09-29, NEW MAJOR, BREAKING): git network operations now route through an internal proxy and egress settings changed (`Fix(git)!`, #39426) — outbound Git may need `[egress]` allow-listing; Actions adds `RUN_RETENTION_DAYS` (#38855). Also a large SECURITY batch (SSH key fingerprint identification, rejected invalid/duplicate Git objects, repo-scoped team authz, x/crypto DoS). GitHub notes: https://github.com/go-gitea/gitea/releases/tag/v28.0.0 — vendor post: https://blog.gitea.com/release-of-28.0.0/
+- **28.1.0** (released 2026-10-06, MINOR): 9 security fixes (push-mirror permission/`DISABLE_NEW_PUSH`, local-path import authz, repository media stored XSS, OAuth2 refresh-token kind, issue-attachment authz, `FORCE_PRIVATE` push-to-create, private RSS/Atom feed leak, tag-delete deleting releases, migration page-size DoS), plus MySQL/MariaDB `READ COMMITTED` and go-git repack fixes. No breaking changes. GitHub notes: https://github.com/go-gitea/gitea/releases/tag/v28.1.0 — vendor post (CVE list): https://blog.gitea.com/release-of-28.1.0/
+- **28.2.0** (released 2026-10-18, MINOR): 9 newly-fixed Gitea CVEs (SSH cert principal confusion, OpenID linking `state`, locked-issue reactions authz, release-attachment metadata leak, `/{username}/-/repositories` panic, draft-release attachment disclosure, Actions artifact v4 body limit, CSRF on repo topics, cross-org label deletion) plus dependency CVE-2026-78659 (`golang.org/x/net` v0.60.0). Three more CVEs are disclosed here but were already fixed in ≤28.1 (CVE-2026-110324, CVE-2026-109844, CVE-2026-97859). No breaking changes and no Postgres requirement. GitHub notes: https://github.com/go-gitea/gitea/releases/tag/v28.2.0 — vendor post (CVE list): https://blog.gitea.com/release-of-28.2.0/
+- **app.ini writability**: Gitea 1.24+ writes runtime config (e.g. `WORK_PATH`) back to `app.ini`; mounting it directly from a read-only swarm config makes that write fail (non-fatal but warned). The unmerged mirror PR #4 (`fix-app-ini-writable`) seeds a writable copy in the `config` volume. Image bumps must not change the config mount / `docker-setup.sh` handling on their own.

cc-ci-plan/upstream/hedgedoc.md

@@ -7,7 +7,7 @@
 
 ## Standing notes
 - hedgedoc 1.11.0 (2026): 4 security CVEs fixed (HTML injection, YAML DoS, CSRF via Gist export, rate-limit bypass). No breaking changes, no migrations, no schema changes. Optional new env var `CMD_RATE_LIMIT_USING_CLOUDFLARE` only needed if running behind Cloudflare — not required for standard deployments.
-- pgautoupgrade: handles Postgres major-version upgrades automatically on container start. Bump ONE major at a time (16→17, then 17→18 on next cycle). The image tag is `<pg-major>-alpine`.
-- pgautoupgrade 18+ (deferred, learned 2026-09-11): PG 18 changes the data-directory layout (docker-library/postgres#1259; PGDATA → `/var/lib/postgresql/<MAJOR>/docker`) — this recipe mounts `postgres:/var/lib/postgresql/data` with no PGDATA, and per the pgautoupgrade README ("Error message when mounting data to /var/lib/postgresql/data on Postgres v18+") that needs a dedicated PGDATA/volume-mount change before bumping to 18, or you can get an empty instance instead of an upgraded one. 16→17 verified live on cc-ci 2026-09-11 (`Upgrade to PostgreSQL 17.11 complete.`, data survived).
+- pgautoupgrade: handles Postgres major-version upgrades automatically on container start; multi-major in ONE deploy works (16.15→18.6 verified live 2026-09-18 — pg_upgrade ran directly on the v16 files to 18.6, data survived). The image tag is `<pg-major>-alpine`.
+- pgautoupgrade 18+ (RESOLVED on PR #3, 2026-09-18): PG 18 changes the data-directory layout (docker-library/postgres#1259; PGDATA → `/var/lib/postgresql/<MAJOR>/docker`, `/var/lib/postgresql/data` becomes a symlink → mount error or EMPTY instance if the mount stays at .../data). The fix carried by PR #3: move the volume mount to `postgres:/var/lib/postgresql` (NO PGDATA env pin needed) — the entrypoint's `MOVING_TO_NEW_STRUCTURE` detects `/var/lib/postgresql/PG_VERSION`, upgrades, and moves data into `<MAJOR>/docker` automatically. Verified live on cc-ci 2026-09-18: `Upgrade to PostgreSQL 18.6 complete.`, pad data byte-identical, app green; 16→17 separately verified 2026-09-11 (`Upgrade to PostgreSQL 17.11 complete.`). Upstream coopcloud renovate PR #22 bumps the tag WITHOUT this mount fix — do not merge that alone.
 - hedgedoc 1.12.0 (2026-08): requires Node 20.17+ (app image bundles its own node — not recipe-relevant); webp uploads; some highlight.js languages removed/changed. No breaking changes or migrations for the recipe.
 - cc-ci tests use the sqlite backend (default compose.yml), not the postgresql compose override — so pgautoupgrade bumps do not affect CI test coverage (verify pg bumps via a direct --chaos deploy with the postgresql override instead).

cc-ci-plan/upstream/immich.md

@@ -68,6 +68,72 @@
   reject). Done in the 2026-09-11 upgrade (v3.1.0→v3.2.0, extending PR #4 from upstream main
   794560f; PR #4's pgvectors0.3.0/valkey-8e8d64b4 divergences superseded — DB realigned to
   immich's official combo, which upstream main itself chose 2026-08-07).
+- **immich-server v3.2.1 (2026-09-14) + v3.2.2 (2026-09-15) pins** (from `docker/docker-compose.yml`
+  @ the v3.2.2 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` and
+  `valkey:9@sha256:70739f85ad…` — **BOTH identical to the recipe's v3.2.0-era pins** (DB and
+  redis stay put; live valkey `9` tag has drifted to `c123e371…` = 9.1.2 but immich's tested
+  pin `70739f85…` = 9.1.1 is kept per the 2026-07-17 precedent; live postgres tag digest
+  re-verified = the pin, not stale). `example.env` @ v3.2.2 byte-identical to v3.2.0. Both
+  releases are bug-fix-only (v3.2.1: sync connection-pool exhaustion, search modal, partner
+  assets on people page, person merge, server auto-VACUUM after TypeORM migrations; v3.2.2:
+  reassign-faces fix) — no breaking changes, no config additions, no operator action. Done in
+  the 2026-09-18 upgrade (v3.2.0→v3.2.2, app+ML only; fresh PR — PR #4 was closed when its
+  v3.2.0 content merged upstream as #19 / `1.11.0+v3.2.0`).
+- **immich-server v3.2.4 (2026-09-28, latest stable) pins** (from `docker/docker-compose.yml` @ the
+  v3.2.4 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…` and
+  `valkey:9@sha256:70739f85ad…` — **BOTH identical to the recipe's v3.2.2-era pins** (DB + redis stay
+  put; live `valkey:9` tag has now moved to `ac9c858b…` = **9.1.2**, but immich's tested
+  `70739f85…` = 9.1.1 is kept per the 2026-07-17 precedent). `example.env` @ v3.2.4 byte-identical to
+  v3.2.2. Window v3.2.2→v3.2.4 is 4 commits: mobile sync-status fix (#31644), an internal `v3.2.3`
+  version commit (no Release object), a base-image dep bump to `202609281550` (#31854), and `v3.2.4`.
+  The v3.2.4 release body: "small patch that primarily fixes the memory leak people have observed
+  through a dependency update." **No breaking changes, no config additions, no operator action.**
+  The base-image bump (`202608300913`→`202609281550`, immich-app/base-images) adds a hardened
+  ImageMagick security policy (#390), reduces server image size (#388), and restores `curl` in the
+  prod image so `immich-healthcheck` works (#391) — all internal to the server image. `v3.3.0-rc.0/1`
+  are pre-releases (2026-09-29/30) — do NOT bump to rc. Done in the 2026-10-02 upgrade
+  (v3.2.2→v3.2.4, app+ML only, fresh PR). Direct registry check confirmed
+  `immich-server/ml:v3.2.4` exist; `abra recipe upgrade` still FATAs on the tag+digest DB pin.
+  Valkey CVE note: CVE-2026-56684 / CVE-2026-63639 (UAF→RCE) are fixed in 9.1.1 and are therefore
+  **  already closed** by the recipe's pinned 9.1.1 — not by this app bump; ignore the live-`9`-tag
+  drift. CVE-2026-25243 (valkey RESTORE zipmap) is vendor-page-only and undecided.
+- **immich-server v3.3.0 (2026-10-07) + v3.3.1 (2026-10-08, latest stable) pins** (from
+  `docker/docker-compose.yml` @ the v3.3.1 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…`
+  (UNCHANGED vs v3.2.4) and `valkey:9@sha256:c123e3715db63d06d4ad6964884037aa0d5d4d703939b9929954112889708e1d`
+  — the **valkey pin moved** 9.1.1 (`70739f85…`) → **9.1.2 (`c123e371…`)**, which is now immich's own
+  official tested pin (the live `9` tag has since drifted to `418652cf…`; keep immich's `c123e371…`
+  per the 2026-07-17 precedent). `example.env` @ v3.3.1 is **byte-identical** to v3.2.4. v3.3.0 is a
+  feature release (people sharing / cluster groups, birthday memories, auto-stacked mobile edits,
+  higher-quality thumbnails via linear-light resampling, OAuth claims synced on every login, PP-OCRv6
+  OCR models); v3.3.1 is a patch (people-sharing + mobile scroll fixes). **No breaking server-side
+  changes, no required config, no operator action.** New TypeORM migrations auto-run
+  (`ConvertUserOAuthIdEmptyStringToNull`, `RenameGeoNamesCountries`, `PersonSharing`,
+  `AddPersonUserTableSharedBySharedWithConstraint`) — all succeeded in a live v3.2.4→v3.3.1 deploy.
+  Opt-in only: `MACHINE_LEARNING_MODEL_REVISION=v2` (optimized models, numerically equivalent, no job
+  re-run) and the people-sharing feature (user-triggered button, not a deploy migration). Done in the
+  2026-10-09 upgrade (v3.2.4→v3.3.1, app+ML+valkey, extending PR #6; fresh `!testme` GREEN — build #45).
+- **immich-server v3.3.3 (2026-10-14) + v3.3.4 (2026-10-20) + v3.3.5 (2026-10-21, latest stable) pins** (from
+  `docker/docker-compose.yml` @ the v3.3.5 tag): `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357…`
+  (UNCHANGED since v3.3.0) and `valkey:9@sha256:c123e371…` (immich's tested 9.1.2, also unchanged). Upstream
+  coopcloud published v3.3.4 as `1.12.0+v3.3.4` (`6b294fce`) and moved the recipe's `redis` pin to the then-live
+  `418652cf…`; that divergence is left as upstream has it (no reason to churn a sidecar in an app bump).
+  `docker/docker-compose.yml` and `docker/example.env` are byte-identical between v3.3.4 and v3.3.5.
+  v3.3.3 = bug fixes (mobile memories date, people-query perf); v3.3.4 = further people-query perf + translation;
+  **v3.3.5 = security patch** (certifi CA-trust fix + October 2026 Node.js security patches). No schema
+  migration, no config/env change, no breaking server-side change. Done in the 2026-10-23 upgrade
+  (v3.3.4→v3.3.5, app+ML only, extending PR #6; live v3.3.4→v3.3.5 dev deploy healthy — migrations finished,
+  `vector`/`vchord` present, no pgvectors drop; `!testme` GREEN first run — build #78).
+- **2026-10-30: postgres sidecar pgvectors 0.2.0 → 0.3.0 (same PG14 + VectorChord 0.4.3 scheme).** With
+  app/ML already at v3.3.5, the only newer same-scheme tag is
+  `ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.3.0` (multi-arch OCI index digest
+  `sha256:87c050465fb969a68c7ac23e375e21f4c95cfacd0edce5fa1bc31e63b7891891`; re-verified via ghcr +
+  `docker buildx imagetools inspect`). immich v3.3.5's own compose still names 0.2.0, but immich only
+  supports `vchord`/`vector` now — `vectors` (pgvecto.rs) is not in its `VECTOR_EXTENSIONS`, so the
+  sidecar library is unused; 0.3.0 is backward-compatible (`ALTER EXTENSION vectors UPDATE`), immich
+  added 0.3.0 support in immich-app/immich#11624, and this recipe previously shipped 0.3.0 with
+  v3.0.1/v2.7.5. Direct dev deploy: 0.2.0-created volume reused by the 0.3.0 task, PG14 clean start,
+  `vchord 0.4.3`/`vector 0.8.1` installed, `/api/server/version` = 3.3.5, `!testme` GREEN (build #105).
+  Recipe PR extends upstream main `ac03d23`; publish with `abra recipe release immich -z`.
 - **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build
   #1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for
   'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified

cc-ci-plan/upstream/lasuite-docs.md

@@ -6,15 +6,30 @@
 | backend | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
 | celery | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
 | y-provider | lasuite/impress-y-provider | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
-| docspec | ghcr.io/docspecio/api | https://github.com/docspecIO/api | https://github.com/docspecIO/api/releases |
+| yhub | lasuite/impress-yhub | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
+| yhub-valkey | valkey/valkey | https://github.com/valkey-io/valkey | https://github.com/valkey-io/valkey/releases |
+| jwt-keys | lasuite/impress-yhub (Node crypto sidecar) | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/blob/main/UPGRADE.md |
+| docspec | ghcr.io/docspec/api | https://github.com/docspec/docspec | https://github.com/docspec/docspec/releases |
 | db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases |
 | redis | redis | https://github.com/redis/redis | https://hub.docker.com/_/redis/tags |
-| minio | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
+| minio | quay.io/minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
 | web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
 
 ## Standing notes
-- impress-frontend, impress-backend (used for both backend + celery services), impress-y-provider all
-  share the same version tag from the suitenumerique/docs monorepo. Upgrade all four in lockstep.
+- **v6.0.x replaces the collaboration server**: hocuspocus/y-provider collaboration is gone; a new
+  `yhub` service (port 3002) serves `/collaboration/*`, with its own `yhub` PostgreSQL database and a
+  Valkey store. `y-provider` is conversion-only. New RSA keys (`JWT_PRIVATE_KEY_FILE` backend,
+  `YHUB_JWT_PRIVATE_KEY_FILE` yhub), `COLLABORATION_SERVER_SECRET` removed. Docs must be migrated into
+  yhub (`SOFT_MIGRATION=true` + `backend migrate_documents`). See UPGRADE.md `[6.0.0]`.
+- **v6.0.2** moves yhub to `@y/hub 0.10.1`, adding the `yhub_ydoc_versions_v1` table/indexes: re-run the
+  yhub schema script (`init-db`) — the recipe does it on every yhub start.
+- **The recipe generates the two RSA keys at runtime** into a `jwt_keys` volume (`jwt-keys` sidecar)
+  because abra cannot generate PEM secrets (`generate=false`) and Swarm secrets must pre-exist.
+- **docspec**: since Docs v5.7.0 the Rust `ghcr.io/docspec/api` (port 3000) replaces the archived Elixir
+  `ghcr.io/docspecio/api` (port 4000); Docs v6.0.2 expects `DOCSPEC_API_URL=http://docspec:3000/conversion`.
+- impress-frontend, impress-backend (used for both backend + celery services), impress-y-provider,
+  impress-yhub (and the jwt-keys sidecar image) all share the same version tag from the
+  suitenumerique/docs monorepo. Upgrade all of them in lockstep.
 - AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed.
 - Minio tag uses a date-based RELEASE.YYYY-MM-DDTHH-MM-SSZ format — abra cannot parse it for upgrades;
   check manually on https://github.com/minio/minio/releases.
@@ -36,3 +51,17 @@
 - redis sidecar (cache/broker for celery+backend): prefer the patch line (8.8.1 over 8.10.0) unless a
   feature is needed — 8.8.1 is a security patch (RedisBloom/TDigest RESTORE RCE); 8.10.0 is a minor
   with many new features.
+- **2026-09-18: docspec image switched with impress v5.7.0** (upstream PR suitenumerique/docs#2553):
+  `ghcr.io/docspecio/api:3.0.x` (Elixir, repo ARCHIVED) → `ghcr.io/docspec/api:1.21.4` (Rust rewrite,
+  github.com/docspec/docspec). New image listens on **3000** (was 4000): recipe sets
+  `DOCSPEC_API_URL=http://docspec:3000/conversion` and healthcheck `http://127.0.0.1:3000/health`
+  (path `/conversion` and the raw-body request contract unchanged).
+- ⚠️ When checking GHCR tags by script: the `/tags/list` endpoint CAPS AT 100 tags — a missing tag
+  there means nothing. Verify the **manifest** endpoint instead (e.g. `ghcr.io/docspec/api:1.21.4`
+  is absent from tags/list but pulls fine).
+- **2026-09-13ish: upstream #25 moved the minio pin to `quay.io/minio/minio`** (same
+  RELEASE.2025-09-07T16-13-09Z tag — manifest-verified pullable on quay; quay's own tags/list is
+  paginated and misleading, use manifest checks there too).
+- **v5.7.0 moved the favorites API endpoint to `/documents/favorites/`** — breaking only for
+  external API consumers, not the recipe/cc-ci tests (which use /authenticate/, /users/me/,
+  /documents/).

cc-ci-plan/upstream/lasuite-drive.md

@@ -9,8 +9,8 @@
 | db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases |
 | redis | redis | https://github.com/redis/redis | https://github.com/redis/redis/releases |
 | mailcatcher | sj26/mailcatcher | https://github.com/sj26/mailcatcher | https://github.com/sj26/mailcatcher/releases |
-| minio | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
-| minio-createbuckets | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
+| minio | quay.io/minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
+| minio-createbuckets | quay.io/minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
 | collabora | collabora/code | https://github.com/CollaboraOnline/online | https://www.collaboraonline.com/release-notes/ (per-version: .../collabora-online-25-04-release-notes/) |
 | onlyoffice | onlyoffice/documentserver-de | https://github.com/ONLYOFFICE/DocumentServer | https://github.com/ONLYOFFICE/DocumentServer/blob/master/CHANGELOG.md |
 | web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
@@ -19,9 +19,12 @@
 - lasuite/drive-frontend and lasuite/drive-backend share the same version tag (drive monorepo).
 - minio and onlyoffice tags use non-semver formats; abra cannot auto-detect upgrades for them.
 - collabora/code uses a 5-part version scheme; abra cannot auto-detect upgrades for it.
-- minio: Docker Hub `minio/minio` `latest` is frozen at `RELEASE.2025-09-07T16-13-09Z` (the
-  `RELEASE.2025-10-15T17-29-55Z` GitHub security release is NOT on Docker Hub `minio/minio`).
-  Newer minio releases moved to the **AIStor** product line (`quay.io/minio/aistor/minio`).
-  A future minio bump requires a registry+product switch — operator decision, not a routine tag bump.
+- minio: Docker Hub `minio/minio` was **REMOVED** (verified 2026-09-18: Hub repo API 404, registry
+  manifest 401 on anonymous pull; last successful docker.io pull was the 2026-09-11 run). The same
+  release tag lives on **`quay.io/minio/minio`** (MinIO's official distribution channel) — recipe
+  pins switched there in lasuite-drive PR #7 (2026-09-18), same release `RELEASE.2025-09-07T16-13-09Z`,
+  mirroring the identical fix merged upstream for lasuite-docs ("fix: pull MinIO image from Quay",
+  PR #25, 2026-09-15). A future minio **version** bump still requires the AIStor product-line
+  decision — quay.io only re-hosts the frozen community line.
 - nginx 1.31.x (1.31.0, 1.31.1) contains multiple security CVE fixes; upgrade is recommended.
 - nginx 1.31.0 breaking change: HTTP/2 and HTTP/3 requests with Connection/Proxy-Connection/Keep-Alive/Transfer-Encoding/Upgrade headers are now rejected (affects proxied HTTP/2 backends).

cc-ci-plan/upstream/mattermost-lts.md

@@ -6,6 +6,23 @@
 | postgres | postgres                                  | https://github.com/postgres/postgres              | https://www.postgresql.org/docs/release/                          |
 
 ## Standing notes
+  - **2026-09-18 re-check** (Mattermost release-policy docs; GitHub release; Docker Hub; vendor
+    security feed): **11.7 ESR line UNCHANGED** — newest patch **11.7.11** (released 2026-09-15,
+    "Low to High severity level security fixes"; Docker Hub tag pushed 2026-09-15). v10.11 ESR
+    remains EXPIRED (2026-08-15); 10.11.24 sits on the dead line — do NOT target. This run
+    extended open PR #2 **11.7.10 → 11.7.11** (branch tip 5696f0b4). Direct `--chaos` deploy on
+    cc-ci exercised the full operator path (deploy at 10.11.22 → seed admin user via API →
+    upgrade to 11.7.11): schema migrations ran on boot, **seeded user data survived** (same
+    id/roles, login 200), ping 200 with x-version-id 11.7.11; `!testme` **GREEN**, drone 1373
+    (level 5: install/upgrade/backup/restore/custom all pass, clean teardown). Vendor feed: **6
+    server advisories fixed in 11.7.11** (MMSA-2026-00711 Low; -00761/-00763/-00764/-00765/-00773
+    Medium, all affecting 11.7.x <= 11.7.10; CVE ids pending 30-day disclosure, due 2026-10-15).
+    **CVE-2026-13426 re-adjudicated FIXED** for the 10.11.22 → 11.7.11 window (fresh GitHub
+    compare: fix commit 3321db82 is an ancestor of v11.7.11 — ahead/behind 0 — and diverged from
+    v10.11.22) → window CVE count **14** (13 deterministic + 1 adjudicated). postgres **15-alpine
+    HELD** (DB-major out of scope). Gotchas learned: `abra app new` RESETS the recipe checkout to
+    main (re-checkout the branch before a `--chaos` deploy), and `abra app new --domain` does NOT
+    write DOMAIN into the app .env (edit the .env + redeploy to fix traefik routing).
   - **2026-09-11 re-check** (Mattermost Server Releases docs; endoflife.date/api/mattermost.json;
     GitHub releases + module tags): **11.7 ESR line UNCHANGED** — newest patch still **11.7.10**
     (2026-08-26; no 11.7.11; v11.11.0 is still a GitHub **prerelease**, innovation). **10.11 ESR
@@ -117,6 +134,18 @@
     2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries
     remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine
     still HELD (DB-major out of scope, operator dump/pg_upgrade).
+  - **2026-09-04 re-check** (endoflife.date/api/mattermost.json 2026-09-04; Mattermost release-policy
+    docs `https://docs.mattermost.com/product-overview/release-policy.html`; `mattermost-server-releases.html`;
+    GitHub releases `v11.7.10`): **11.7 ESR is STILL the current supported ESR/LTS line** — "v11.7 &
+    Desktop App v6.2 Extended Support: 2026-05-15 → 2027-05-15" (the chart on the release-policy page;
+    ESR cadence = every 9 months, supported 12 months). Latest 11.7.x patch **11.7.10** (2026-08-26,
+    "Mattermost Platform Extended Support Release 11.7.10 contains various bug fixes") — NOT a
+    prerelease; target confirmed. 11.8/11.9/11.10 remain Feature/innovation releases (EOL 2026-09-15 /
+    10-15 / 11-15, `lts:false`), NOT ESR — do NOT target; wait for the NEXT official ESR (expected
+    ~Feb 2027 on the 9-month cadence). No newer 11.7.x ESR patch exists as of this week, so PR #2's
+    head (`59e8c2c`, app image `11.7.10`) is still the correct target → this run RE-VERIFIES PR #2
+    (no new app bump). 11.11.0-rc1/rc2 seen on GitHub but innovation + pre-release — not a target.
+    postgres 15-alpine still HELD (DB-major out of scope, operator dump/pg_upgrade).
 
 ## NVD CPE fallback
 This project publishes nothing machine-readable we can reach — no GitHub advisory feed,

cc-ci-plan/upstream/n8n.md

@@ -205,3 +205,184 @@
   2.36.0 Array.merge→mergeIntoObject + workflow-tags API migration; 2.37.0 "Any workflow" caller-policy
   deprecation + JSON content-type on decorator body routes + binary-data endpoint adapt; 2.39.0
   workflow-version endpoint deprecation. Recommended release: `-y` (feature minor).
+- 2.38.7 (2026-09-11, patch): 3 core fixes (MCP toolkit members on workers; consent-grant reuse;
+  trust the CA from GIT_SSL_CAINFO for source-control HTTPS remotes).
+- 2.39.3 (2026-09-11, Pre-release): 1 core fix (GIT_SSL_CAINFO trust — 2.38.7 backport).
+- 2.39.4 (2026-09-11, Pre-release): 1 core fix (sandbox npm install flags by provider).
+- 2.39.5 (2026-09-14, stable line): editor fix (revoke end-user credentials) + feature
+  (instance-report log streaming events).
+- 2.39.6 (2026-09-16, stable line): core fix (Gateway credits eligibility) + features (**v3
+  breaking-change rule for the storage-directory rename**; **warn on deprecated N8N_DB_PING_TIMEOUT
+  env var** — recipe does not set it).
+- 2.39.7 (2026-09-17, stable line; **`stable` tag points here** — the stable badge moved to the
+  2.39.x line): core breaking-change-detection resilience; editor numeric credential inputs.
+- 2.40.0 (2026-09-15, Pre-release, feature minor — large): Microsoft Dataverse core node; AI Agent
+  forced first-iteration tool calls; Git-based promotion model (selective promote, timestamped
+  branches, promotion entities/DTOs); **workflow publication service enabled by default**;
+  ephemeral sandboxes; agent episodic memory; ai_preference table; HTTP Request
+  PROPFIND/MKCOL/MOVE/COPY/REPORT; Grist OAuth2; MS Teams online-meeting/channel-message ops; v3
+  migration-report rules (storage-directory rename, preExecute) + "chat hub goes away" warning;
+  N8N_DB_PING_TIMEOUT deprecation warning; secret-redaction consolidation; queue-mode execution
+  timeout for stuck jobs; ~150 core/editor/node bugfixes. **No breaking compose/env/migration
+  changes; no N8N_* renames.**
+- 2.40.1 (2026-09-16, Pre-release): GitHub release body EMPTY (maintenance republish, no listed
+  changes).
+- 2.40.2 (2026-09-17, Pre-release): core breaking-change-detection resilience + editor numeric
+  credential inputs (the 2.39.7 pair on the 2.40 line). Docker Hub manifest verified active.
+- 2.40.3 (2026-09-18, Pre-release): core "repair data-encryption keys stored as the raw instance
+  key" + editor blank-workflow-preview fix. Published the morning of the 2026-09-18 run — newer
+  than the operator's snapshot (which named 2.40.2 as newest); NOT taken that run (operator's
+  explicit target 2.40.2), flagged in the plan/PR as a trivial follow-up bump.
+- 2026-09-18 run: PR #8 (branch upgrade-ef0dd56, at 2.39.2 since 2026-09-11, !testme GREEN then)
+  extended **2.38.4 → 2.40.2** on upstream main tip 0b436ec (still 3.5.0+2.38.4 — upstream has not
+  started the bump). Stable badge on 2.39.7; 2.40.x pre-release per precedent. Rolling upgrade
+  safe (sqlite, TypeORM auto-migrate). Operator flags unchanged (API-caller-level deprecations,
+  encryption-key rework informational, N8N_DB_PING_TIMEOUT warn-only, recipe doesn't set it).
+  2.40.3 exists (see above) — flagged, not taken. Recommended release: `-y`.
+- 2.40.4 (2026-09-21, Pre-release): core fix (tear down workflow triggers when publication meets a
+  node type the instance cannot load) + perf (stop loading project members when listing
+  credentials).
+- 2.40.5 (2026-09-21, Pre-release): core fix (limit declarative routing during base URL ownership
+  checks). Docker Hub `2.40.5` manifest verified active multi-arch (amd64+arm64, digest
+  sha256:9f693fd5..., 2026-09-21) — not a withdrawn 2.37.5-style partial tag. NOTE: 2.40.x
+  `POST /rest/login` now takes `emailOrLdapLoginId` instead of `email` (request-shape change; the
+  cc-ci login-state test only GETs /rest/login so unaffected).
+- 2026-09-21 run: the flagged catch-up. PR #8 (branch `upgrade-ef0dd56`, at 2.40.2, !testme GREEN
+  2026-09-18) extended **2.40.2 → 2.40.5** on the same upstream main tip `0b436ec` (n8n NOT
+  merged upstream today, unlike discourse/keycloak/matrix-synapse). 2.40.2→2.40.5 is bugfix-only
+  (encryption-key raw-repair, trigger teardown, declarative-routing limit); no breaking
+  compose/env changes; no new migrations beyond the 15 already verified at 2.40.2. Stable badge
+  now on the 2.39.x line (2.39.9/2.39.10 released 2026-09-21); 2.40.x pre-release per precedent.
+  Recommended release: `-y`.
+- 2.40.6 (2026-09-24, patch): core fixes (keep OpenTelemetry export working after a restart when
+  Sentry is enabled; retry instance reports that cross the UTC midnight boundary) + feature
+  (authenticate instance reports with the license certificate).
+- 2.40.7 (2026-09-25, patch): 1 core fix (propagate project span attributes to node spans).
+- 2.41.0 (2026-09-22, Pre-release; the 2.41 feature minor — note the 2.40.x patch line continued
+  alongside with 2.40.6/2.40.7): a large feature/bugfix release. Features: standalone node
+  execution as a Workflow Builder / AI Assistant tool, admin permission for node execution in the
+  assistant, API credential creation with source IDs, push-based reload endpoint for custom-node
+  development, execution view/delete permissions in project roles, MCP-registry capability
+  filtering, Databricks Embeddings/Chat-Model node, MS Teams @mentions + a Chat resource, MiniMax
+  dynamic model list, email-change confirmation flow. Core bugfixes include repair of
+  data-encryption keys stored as the raw instance key, stalled-job success finalization,
+  multi-main workflow-publishing fix, stop an unreachable external-secrets provider blocking
+  startup, undici 7.29.1 + vm2 3.12.2 bumps, RFC 9068 MCP OAuth compliance. **No breaking
+  compose/env/migration changes; no `N8N_*` env renames.** (2.41.1/2.41.2 = patch bugfixes.)
+- 2.41.3 (2026-09-25, patch): 1 core fix (propagate project span attributes to node spans). Last
+  week's PR #9 target — !testme GREEN (run 21, 2026-09-28); TypeORM migrations clean live.
+- 2.41.4 (2026-09-30, patch): API/core/editor fixes (return an execution when its stored trace
+  context is incomplete; count a database ping as successful when its reply arrives during
+  event-loop lag; store queue job results only for executions this process enqueued) + features
+  (n8n Assistant onboarding thread for new Cloud signups; route assistant credit CTAs to top-up).
+- 2.41.5 (2026-10-01, patch): atom-feed release body empty (release republish / no listed changes).
+- 2.42.0 (2026-09-29, Pre-release; the 2.42 feature minor): a large feature/bugfix release.
+  Features: AI Agent Tool can use its own tools under a pre-v3 parent agent, workflow descriptions
+  on create, `extendsCredential` in workflow create/update, OpenAPI path-parameter validation,
+  credential descriptions behind an instance flag, Anthropic prompt caching in the Message op,
+  API projects with source IDs, pinned Apply/Apply-Continue CLI commands, MCP-registry feature-flag
+  removal, Agent Builder/editor improvements. ~60 core/editor/node bugfixes (chat-integration
+  shutdown, Redis pubsub reconnect + half-open detection, OTel-after-restart-with-Sentry,
+  instance-report unique index, expression-isolate release, async delete retries). **No breaking
+  compose/env/migration changes; no `N8N_*` env renames.**
+- 2.42.1 (2026-09-30, patch): API/editor fixes (emit valid schemas for untyped values in the Public
+  API spec; return an execution when its trace context is incomplete) + editor feature (show the
+  n8n logo on canvas in canvas-only mode).
+- 2.42.2 (2026-10-01, patch; **newest numeric tag on Docker Hub — verified 2026-10-02**): 1 core
+  fix (stop waiting on Bull `job.finished()` for queued executions). Docker Hub `2.42.2` manifest
+  active; no 2.43.x exists.
+- 2026-10-02 run: reconcile confirmed coopcloud upstream main still `9141e23` (3.5.1+2.40.5) and
+  PR #9 (branch `upgrade-a144baf`, at 2.41.3, !testme GREEN 2026-09-28) still open/unmerged — so
+  per the one-evolving-PR contract the run extends PR #9 with a fast-forward commit
+  **2.40.5 → 2.42.2**. abra lists 2.42.2 as the newest candidate and a direct Docker Hub tag
+  enumeration agrees (no 2.43.x). The delta spans two feature minors (2.41.0, 2.42.0) plus patches
+  on both lines; **no breaking compose/env/migration changes, no `N8N_*` renames**. Rolling
+  upgrade safe (sqlite default; TypeORM migrations auto-run on boot). Operator flags remain
+  HTTP-API-caller-level only (2.33.0 workflow activate/deactivate deprecation; 2.36.0
+  `Array.merge`→`mergeIntoObject`; 2.37.0 JSON content-type on decorator body routes + binary-data
+  endpoint adapt; 2.39.0 workflow-version endpoint deprecation; 2.40.x `POST /rest/login`
+  `emailOrLdapLoginId` rename). Recommended release: `-y`.
+- 2.42.3 (2026-10-05, patch): API fix (let global members grant variable scopes to API keys) + core
+  fix (keep the task runner running on an unhandled promise rejection) + editor feature (Gateway
+  credits promotions on community nodes).
+- 2.42.4 (2026-10-07, patch): release body empty (republish / no listed changes).
+- 2.42.5 (2026-10-08, patch): 1 editor fix (show the Agent artifact while the builder creates it).
+- 2.42.6 (2026-10-09, patch): core feature (block create/update of workflows with deprecated nodes) +
+  Gemini node feature (Nano Banana 2.1 + 2 Lite in the image-model list). Backport of the 2.43.3 pair
+  onto the stable 2.42 line.
+- 2.42.7 (2026-10-12, patch): 2 core fixes (concurrent credential tests no longer share one node-type
+  registry; resume tool calls for deferred tools a skill loaded).
+- 2.43.0 (2026-10-06, Pre-release; the 2.43 feature minor): a large feature/bugfix release. Features:
+  Agent Builder / Instance AI / editor improvements + ~many core/API/editor fixes (valid schemas for
+  untyped values in the Public API spec; global members can grant variable scopes to API keys; return
+  an execution when its stored trace context is incomplete; Azure OpenAI Chat Model; etc.). **No
+  breaking compose/env/migration changes; no `N8N_*` env renames.** (2.43.1/2.43.2 = patch bugfixes:
+  editor Assistant/Execute UI; Kimi K3 + OpenAI-compatible Assistant providers self-hosted; Agent
+  artifact preview; agent-builder prompt-caching/background-sandbox perf.)
+- 2.43.3 (2026-10-09, Pre-release): core feature (block create/update of workflows with deprecated
+  nodes) + Gemini node feature (Nano Banana 2.1 + 2 Lite in the image-model list).
+- 2.43.4 (2026-10-12, patch, **stable line**): 2 core fixes (handle each Chat Hub segment once in
+  queue mode; resume tool calls for deferred tools a skill loaded).
+- 2.43.5 (2026-10-14, patch, stable line): core fix (validate the `size` arg of the `chunk()`
+  expression extension) + editor feature (show Gemini models by name / prioritised first).
+- 2.43.6 (2026-10-15, patch, **stable line — `stable` floating Docker tag points here**, digest
+  `sha256:dd580fe1…`): 1 core fix (create an XML parser per webhook request body). Backport of the
+  2.44.2 fix onto the stable 2.43 line.
+- 2.44.0 (2026-10-13, **Pre-release / beta line — `beta` floating Docker tag points to 2.44.2**): the
+  2.44 feature minor. Large core/editor change set (credential/node-type policy enforcement across
+  agent tools, credential tests/probes, package import, folder/credential/delete-transfer moves;
+  block the assistant when AI credits hit zero; per-process crash-journal paths; configurable
+  healthcheck timeout; Microsoft SharePoint Trigger node; many bugfixes). **No breaking
+  compose/env/volume/label changes; no `N8N_*` renames.**
+- 2.44.1 (2026-10-14, Pre-release/beta): 1 editor feature (Gemini models by name / prioritised first).
+- 2.44.2 (2026-10-15, Pre-release/beta; **newest numeric tag abra lists**; Docker Hub `beta` floating
+  tag = `sha256:b57e7a9c…`): 1 core fix (create an XML parser per webhook request body).
+- 2026-10-09 run (not separately logged here): PR #9 (branch `upgrade-a144baf`) extended from the
+  2.42.2 of the 2026-10-02 run to **2.43.2** (newest numeric tag abra listed that day). 2.43.2 was
+  still Pre-release on GitHub at the time (tracking-the-newest-tag precedent).
+- 2026-10-16 run: reconcile confirmed coopcloud upstream main still `9141e23` (3.5.1+2.40.5) and
+  PR #9 (branch `upgrade-a144baf`, at 2.43.2) still open/unmerged, so per the one-evolving-PR
+  contract the run extends PR #9 **2.43.2 → 2.43.6**. Operator guidance this run: verify whether
+  2.44.x is the newest STABLE line and take the newest SAFE patch. **2.44.x is the beta/pre-release
+  line** — GitHub marks 2.44.0/2.44.1/2.44.2 `prerelease=true`, and Docker Hub's `beta` floating tag
+  points at 2.44.2 while `stable` points at 2.43.6 (verified live). So the safe target is **2.43.6**
+  (newest patch on the now-stable 2.43 line), NOT the beta 2.44.2. 2.43.2→2.43.6 is patch-only
+  (2.43.3 deprecated-node workflow guard + Gemini image list; 2.43.4 Chat-Hub queue-mode + deferred
+  tools; 2.43.5 `chunk()` size-arg validation + Gemini picker; 2.43.6 per-request XML parser). No
+  breaking compose/env/migration changes. Live-deployed on cc-ci `dev-n8n` (2.40.5 → 2.43.6): TypeORM
+  migrations completed cleanly, editor `/rest/login` 401 (auth-gated, as expected), `/healthz` 200 —
+  converged. Advisory scan 2.43.2→2.43.6: 0 CVEs fixed (9 sources, none failed). Recommended
+  release: `-y` (relative to the published label app tag 2.40.5 the delta spans feature minors
+  2.41/2.42/2.43).
+- 2.43.7 (2026-10-16, patch): core suspend in-flight production executions at worker shutdown.
+- 2.43.8 (2026-10-19, patch): core turn off expression spans by default (backport of 2.44.4).
+- 2.44.3 (2026-10-16, patch): core keep Observer detail lines instead of discarding the response.
+- 2.44.4 (2026-10-19, patch): core reuse connections in the log-streaming webhook destination; turn
+  off expression spans by default.
+- 2.44.5 (2026-10-21, patch): core key queue job outcomes by job on shutdown resume; editor load
+  credential types on startup.
+- 2.44.6 (2026-10-22, patch; **`stable` floating tag points here**): Postgres Node prevent global
+  pgtypes overwrite.
+- 2.45.0 (2026-10-20, Pre-release; the 2.45 feature minor): automated migrations for removed nodes;
+  node and credential type policies enabled by default; durable scheduler enabled by default;
+  Idempotency-Key on Public API POSTs; workflow migration-findings endpoints/notifications; Kafka v2
+  default for new nodes; Monday.com V2 default; AI Agent/Assistant, editor and node fixes; UI rebrand.
+  **No breaking compose/env/volume/label changes; no `N8N_*` renames.**
+- 2.45.1 (2026-10-21, Pre-release): core claim an idle worker's next queue job and its lock atomically;
+  Microsoft Teams Trigger Graph subscription renewal via lifecycle notifications; editors can triage
+  migration findings.
+- 2.45.2 (2026-10-22, Pre-release; **`beta` floating tag points here — newest numeric tag**): core read
+  type-availability policies from memory at enforcement points; Postgres Node prevent global pgtypes
+  overwrite; migration report CSV export; OAuth-confirmation/email logo update.
+- 2026-10-23 run: reconcile confirmed coopcloud upstream main moved to `c834df2` ("NODE_FUNCTION_ALLOW_EXTERNAL
+  in env file", on top of 3.5.1+2.40.5) and PR #9 (branch `upgrade-a144baf`, at 2.43.6) still
+  open/unmerged. Per the one-evolving-PR contract, PR #9 extended **2.40.5 → 2.45.2** (explicit target
+  = highest tag abra lists). 2.45.x is the current Pre-release/beta line; **stable = 2.44.6** (verified
+  live via Docker Hub floating tags), flagged to the operator. Window spans feature minors 2.41–2.45,
+  but release notes show **no breaking compose/env/volume/label changes and no `N8N_*` renames**.
+  Live-deployed on cc-ci `dev-n8n`: TypeORM migrations completed cleanly through
+  `CreatePromotionRunAndReviewTables1792139680417`, editor accessible, `/healthz` 200, `/rest/login`
+  401; runtime deprecation warnings only (`WEBHOOK_URL`→`N8N_WEBHOOK_URL`, internal runner mode,
+  unverified-packages default, runner-timeout/compression-limit default reductions). Advisory scan
+  2.40.5→2.45.2: 0 CVEs identified, BUT the GitHub advisory sources failed (unauthenticated rate
+  limit) — scan incomplete, not to be read as clean. Recommended release: `-y`.

cc-ci-plan/upstream/rustdesk-server.md

@@ -0,0 +1,23 @@
+# Upstream sources — rustdesk-server
+
+| service | image | source repo | releases / changelog |
+|---------|-------|-------------|----------------------|
+| app | rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server/releases |
+| signal | rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server | https://github.com/rustdesk/rustdesk-server/releases |
+
+## Standing notes
+- **NOT enrolled** in the cc-ci weekly set (operator 2026-10-30: single client, maintenance contract
+  pending). Mirror `recipe-maintainers/rustdesk-server` exists + this registry entry so one-off
+  `/recipe-upgrade` runs and advisory scans work; if the client confirms, enroll via `/recipe-enroll`
+  (test suite + bridge + used-recipes weekly row).
+- Both services (`app` = hbbs, `signal` = hbbr) use the SAME image — always bump them together.
+- Docker Hub has stray `1.4.5-*` arch tags (pushed 2026-01-12, top-level `1.4.5` tag deleted, no
+  GitHub release) — an aborted/alias push, NOT a real version line. The real line is `1.1.x`;
+  GitHub "Latest" + docker `latest` agree (1.1.16 as of 2026-07-20). Ignore 1.4.5.
+- Recipe uses `-k _` on both hbbs and hbbr (enforces key verification, enforced-key mode since
+  1.1.11) and publishes host-mode ports 21115-21119 (hbbs) + 21117/21119 (hbbr) — direct
+  client connections, only 21118/21119 are the web-socket/http endpoints behind nothing; the
+  traefik labels on `app` are vestigial (hbbs has no HTTP :80 service; healthcheck commented out).
+- 1.1.15 raised **default bandwidth limits** and added auth-failure logging (fail2ban-friendly);
+  1.1.16 fixed **unauthenticated UDP punch-hole reflection/amplification abuse** + critical `mio`
+  0.8.5→0.8.11 issue + i32 overflow (peers offline 24.9–49.7d shown online).

cc-ci-plan/used-recipes.md

@@ -28,10 +28,11 @@ lasuite-drive     weekly
 lasuite-meet      weekly
 mailu             weekly
 matrix-synapse    weekly
-mattermost-lts    weekly
+mattermost-lts    external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 mumble            weekly
 n8n               weekly
 plausible         external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
+rustdesk-server   weekly
 uptime-kuma       external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 wordpress         weekly
 ```

flake.lock

@@ -10,11 +10,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1788820034,
-        "narHash": "sha256-mkktTMeGKutfPp3pn3AhoPoGSCm4iRSnPvpmbEpMmwA=",
+        "lastModified": 1793644559,
+        "narHash": "sha256-u6xPHneKwIrd1taLVs1X12dUs46RzTzkZ9fJ8d/ZvYI=",
         "ref": "refs/heads/main",
-        "rev": "b11cc0b7385aee9fb89bd9a7ed23896401d683e5",
-        "revCount": 1543,
+        "rev": "d569e6528f2eba4309db4794b091d6a1ecc029bf",
+        "revCount": 1589,
         "type": "git",
         "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
       },

memory/MEMORY.md

@@ -1,9 +1,12 @@
 # Memory index
 
+- [Enroll gate fixtures must fail loudly](enroll-gate-fixtures-fail-loudly.md) — version-gate reads via lifecycle.deployed_identity, never None-into-if; post-host-merge deploy = --cc-ci-only after merge (staging obsolete)
+
 - [Orchestrator host: Hetzner](orchestrator-host-hetzner.md) — runs on Hetzner cpx22; rebuild cmd, loops-service bounce, git-identity gotcha
 - [Push commits to remote](push-commits-to-remote.md) — push to git.autonomic.zone right after every commit in this repo
 - [Regression canary cadence](regression-canary-cadence.md) — server E2E canaries run on polish/review/release, not every commit
-- [Recipe-mirrors public / org blocker](recipe-mirrors-public-org-blocker.md) — mirrors public but recipe-maintainers ORG is private → live PR-STATUS column dark until operator flips org public
+- [Recipe-mirrors MUST be public](recipe-mirrors-public-org-blocker.md) — org public since ~2026-06, but /recipe-enroll created late mirrors (gitea, wordpress) private-from-birth, darkening report STATUS cells; flipped public + skill fixed 2026-09-14
+- [Gitea Anubis UA challenge](gitea-anubis-ua-challenge.md) — git.autonomic.zone 307-challenges browser UAs to an un-CORS-able origin; server-side proxies representing browser JS must pin a non-browser UA (cc-ci PR #38, 2026-09-14)
 - [abra chaos-deploy checkout gotcha](abra-chaos-deploy-checkout-gotcha.md) — `abra app new` moves recipe checkout to release tag; checkout PR branch after, or chaos deploys wrong tree
 - [Shared recipe-checkout race](shared-recipe-checkout-race.md) — never git-checkout ~/.abra/recipes/<recipe> on cc-ci while its CI build runs; harness deploys from that tree
 - [immich pgvecto.rs DROP DATABASE panic](immich-pgvectors-drop-database-panic.md) — DROP DATABASE crashes immich's postgres image; use pg_dump --clean --if-exists + search_path rewrite

memory/enroll-gate-fixtures-fail-loudly.md

@@ -0,0 +1,43 @@
+---
+tags: [cc-ci, enrollment, testing, deploy]
+---
+
+# Enroll-verify gate fixtures must fail loudly
+
+## Context
+
+During rustdesk-server enrollment (2026-10-30, cc-ci PR #48) the first full-suite verification
+run came back RED only because of the *fixture*, not the app: the punch-hole regression test's
+version gate read the deployed image with a hand-rolled `docker service inspect`, got None, and
+the `if version is not None and version < (1,1,16)` gate silently decided to RUN on a 1.1.12 head
+where pre-fix answering is the expected behavior. An expected-RED looked like a real bug.
+
+## Lesson
+
+- A test gate that cannot decide its condition must `pytest.fail` LOUDLY with the raw value —
+  never default to None-into-an-if (silently runs → wrong RED) or silent skip (vacuous green).
+- cc-ci already has a first-class image reader: `runner/harness/lifecycle.py::deployed_identity(domain)`
+  → `{"version","image","chaos"}`; parse the tag from `.image` and strip any `@sha256:` digest.
+  Hand-rolled service inspects in recipe_meta/custom tests are a duplication hazard.
+- Version-gated custom tests skip on pre-fix heads by design (STYLE §2) — that is fine as long
+  as the version read itself cannot fail silently.
+
+# cc-ci host deploy mechanism (post host-merge, 2026-09+)
+
+To run any merged cc-ci main change on the live host now, use the `--cc-ci-only` path from
+/cc-ci-orchestrator-update — merge the PR first (enrollment PRs: comment verification evidence,
+then merge — invocation = authorization), then:
+
+```
+ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci &&
+  git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock &&
+  git push -q origin HEAD:main"'
+ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
+systemd-run --unit=ccci-rebuild-test --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
+  bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"   # := health checks := then switch
+```
+
+The old staging procedure (branch clone at /root/cc-ci-deploy, referenced by the retired
+/cc-ci-server-update skill and old enroll SKILL.md step 7 lines) is obsolete — the combined-host
+flake takes the cc-ci module from the lockfile, so a BRANCH cannot be staged that way; deploy
+after merge instead.

memory/gitea-anubis-ua-challenge.md

@@ -0,0 +1,26 @@
+---
+name: gitea-anubis-ua-challenge
+description: "Gitea (git.autonomic.zone) sits behind Anubis, which 307-challenges browser-like User-Agents to an un-CORS-able counter-domain — any server-side proxy that forwards a browser UA on behalf of client JS breaks; pin a non-browser UA"
+metadata: 
+  node_type: memory
+  type: project
+---
+
+`git.autonomic.zone` sits behind **Anubis** (`anubis.swarm.autonomic.zone`). Anubis
+307-challenges requests whose User-Agent looks like a real browser to
+`/.within.website/?redir=…` — a JS proof-of-work challenge page on a *different* origin with
+**no CORS headers**.
+
+Impact: any server-side proxy that serves client-side JS but forwards the *end browser's* UA
+hits this asymmetry — `curl` (non-browser UA) passes through cleanly, a real Firefox/Chrome UA
+gets the 307. Concretely: the Recipe Report's same-origin PR-STATUS proxy
+(`report./pr/<recipe>/<n>`, cc-ci `nix/modules/reports.nix`) forwarded the browser UA, so
+every live cell in the browser rendered `?` (week-2026-09-11, operator-CORS console report).
+
+Fix (cc-ci PR #38, merged 2026-09-14): `proxy_set_header User-Agent "ccci-reports-proxy/1.0";`
+in the `/pr/` nginx location — a stable non-browser UA passes Anubis unmolested. Deployed via
+`nix flake update cc-ci` → `nixos-rebuild test` → health → `switch` (flake.lock commit).
+
+Rule of thumb: server-side callers of the Gitea API (scripts, proxied fetches, harness code
+running in a browser's name) must pin a non-browser User-Agent or cookie-carry; browser-visit
+flows keep the JS challenge and that's fine. Related: [[recipe-mirrors-public-org-blocker]].

memory/recipe-mirrors-public-org-blocker.md

@@ -1,29 +1,26 @@
 ---
 name: recipe-mirrors-public-org-blocker
-description: "Recipe mirrors are public repos but the recipe-maintainers ORG is private-visibility, so anon reads 404; bot can't flip the org"
-metadata: 
+description: "Recipe mirrors MUST be public (fleet invariant: the report's tokenless PR-STATUS proxy only sees public repos); org is public since ~2026-06, but late-enrolled mirrors (gitea, wordpress) were created private-from-birth by /recipe-enroll — fixed 2026-09-14"
+metadata:
   node_type: memory
   type: project
   originSessionId: f7960036-d990-4a21-a81e-f7c486d97fea
 ---
 
-As of 2026-06-09 all 21 recipe mirrors under `recipe-maintainers` were flipped `private=false`
-(secret-scanned first), to power the Recipe Report's live PR-STATUS column via the tokenless
-same-origin proxy `report.ci.commoninternet.net/pr/<recipe>/<n>` (shipped in cc-ci
-`nix/modules/reports.nix`). BUT the **org itself is `visibility: private`**, which makes Gitea 404
-all its repos for anonymous users — so the live STATUS column shows a muted "?" instead of open/✓.
+The Recipe Report's live PR-STATUS column (`report.ci.commoninternet.net/pr/<recipe>/<n>`,
+shipped in cc-ci `nix/modules/reports.nix`) is a tokenless same-origin nginx proxy to the Gitea
+API — it can ONLY see public repos. So **`recipe-maintainers/<recipe>` mirrors MUST be public**
+(fleet invariant; cc-ci/cc-ci-orchestrator/archived repos stay deliberately private).
 
-**Blocker:** `autonomic-bot` cannot flip the org (PATCH `/orgs/recipe-maintainers` → 403 "Must be an
-organization owner"; `is_admin=false`; the basic-auth credential lacks `write:organization` scope,
-even though the bot is in the Owners team). Confirmed model: `autonomic-cooperative` is a public org
-and its repos ARE anonymously visible; `recipe-maintainers` is private and they are not.
+History: originally ALL mirrors were private; on 2026-06-09 the 21 then-existing mirrors were
+flipped `private=false` after a secret-scan, and the `recipe-maintainers` org was later flipped
+public by the operator (the old 'org is private' blocker is RESOLVED).
 
-**Why:** the whole live-status feature is dark until this is resolved. Private repos stay hidden even
-in a public org, so flipping the org public does NOT expose the four locked-private repos (`cc-ci`,
-`cc-ci-secrets`, `cc-ci-orchestrator`, `archived-cc-ci-orchestrator`).
-
-**How to apply:** operator (an org owner) must set `recipe-maintainers` org visibility to **public**
-in the Gitea UI (Settings → make org public), OR provision a token with `write:organization` scope.
-The instant that happens, the proxy returns 200 PR JSON and the column lights up — no redeploy needed.
-Verify: `curl https://report.ci.commoninternet.net/pr/cryptpad/5` should return PR JSON, not a 404.
+Recurrence (2026-09-14): /recipe-enroll still carried the old instruction to create mirrors with
+`"private":true` — so the later-enrolled mirrors `gitea` (2026-06-11) and `wordpress` (2026-08-03)
+were **private from birth**, leaving the report STATUS cells for those rows permanently "?"
+(gitea/wordpress rows in week-2026-09-11). Fixed: secret-scanned both, flipped `private=false`
+via PATCH `/repos/recipe-maintainers/<r>`, patched the skill to create `private:false`. The
+report now lights up for every row — verify any new PR row with
+`curl https://report.ci.commoninternet.net/pr/<recipe>/<n>` returning PR JSON, not 404.
 Related: [[push-commits-to-remote]].

memory/weekly-upgrader-opencode-go-glm.md

@@ -0,0 +1,111 @@
+---
+name: weekly-upgrader-opencode-go-glm
+description: weekly upgrade run now defaults to opencode-go/glm-5.3-flash (PR #20, 2026-09-14); zen tier dead on this host; how backend/model + monitoring are wired
+metadata: 
+  node_type: memory
+  type: project
+  originSessionId: 85355980-5e4f-4f90-b1ca-d0e4fe82f04b
+---
+
+The weekly recipe-upgrade run (`cc-ci-plan/launch-upgrader.py`, fired by the
+`cc-ci-upgrade-all` systemd timer Thu 22:00 Boston) now **defaults to backend=opencode,
+model=opencode-go/glm-5.3-flash** when no env is set (operator choice 2026-09-08 via
+`/srv/cc-ci/upgrader.env`; PR #20 hardcoded the same defaults into the launchers 2026-09-14 —
+for all three launchers: upgrader, report, supervisor). Model default tracks the
+backend — an explicit `LOOP_BACKEND=claude` (no model) falls back to `sonnet`.
+
+**2026-09-14 — ZEN IS DEAD ON THIS HOST.** The ZEN provider models (`opencode/glm-5.2`) are
+gone from the provider catalogue AND the host holds no ZEN key, so any launch that bypassed
+upgrader.env died instantly with `ProviderModelNotFoundError: opencode/glm-5.2` — this is what
+killed the 2026-09-11 report agent at launch and every manual retry (all valid ZEN credentials
+mean nothing without the ZEN models existing server-side). If a launcher log shows
+`ProviderModelNotFoundError`, the model id does not resolve on this host: run
+`opencode models | grep glm` for what does, then fix the tier/model default. See PR
+recipe-maintainers/cc-ci-orchestrator#20.
+
+**Provider:** "OpenCode **Go**" — a flat $10/mo subscription, DISTINCT from "opencode Zen"
+(pay-as-you-go). Endpoint `https://opencode.ai/zen/go/v1/` (OpenAI-compatible), model id
+**`opencode-go/glm-5.2`** (Zen tops out at glm-5.1 — `opencode.ai/zen/v1` does NOT have 5.2;
+updating the opencode CLI does NOT change this, the catalog is server-side). API key lives in
+`~/.local/share/opencode/auth.json` (mode 600, entry `opencode-go` type api) — **never in the
+repo**. opencode CLI installed: 1.15.13.
+
+**Override per run without a rebuild:** drop `/srv/cc-ci/upgrader.env` (read via
+`EnvironmentFile=-` on the unit) with e.g. `LOOP_BACKEND=claude` / `LOOP_MODEL=sonnet`. Absent
+file → the glm-5.2 default.
+
+**Monitoring (opencode backend):** the launcher does `opencode run -m <model> --share --attach
+http://127.0.0.1:4096`. `opencode-web.service` (=`opencode serve` :4096, nginx → 
+http://oc.commoninternet.net, tailnet-only) must be active. `OPENCODE_SHARE=1` (default) also
+mints a public `https://opncd.ai/share/<id>` link (printed in the TUI / readable from the server
+API `GET /session/<id>` → `.share.url`). Note: launch-upgrader.py's `session_busy()` regex is
+claude-tuned, so it misreports an active opencode run as "idle/finishing" — check the process
+(`opencode run`) or web UI instead.
+
+Gotcha fixed: opencode `-m/--model`/`--attach`/`--share`/`--title` are flags on the `run`
+SUBCOMMAND — they must come AFTER `run` (a global `opencode --model X run` is silently ignored).
+
+`launch-report.py` also defaults to opencode-go/glm-5.2 now (overridable via REPORT_BACKEND/REPORT_MODEL).
+In the weekly flow the upgrader writes the summary inline AND auto-launches the report (upgrade-all §6).
+
+**Usage limit + auto-resume watchdog (added 2026-06-23):** opencode-go enforces a ROLLING usage window
+(~5h) capped by dollar value. A full glm-5.2 upgrade run is heavy — ~8 recipes (≈2h20m wall) exhausted
+it mid-run, returning HTTP 429 + `retry-after` (seconds) on the go endpoint. When limited, the
+`opencode run` agent loop ENDS and does NOT self-resume (process lingers but idle). Fix: `launch-upgrader.py`
+now has `resume` (continue the SAME session, context preserved, via `opencode run -s <id> --continue` —
+finds the session from the web server, kills the idle proc via /proc scan NOT `pkill -f` which self-matches)
+and `watchdog` (polls the session log; on >15min stall, waits out the 429 retry-after then auto-resumes).
+An opencode `start` auto-spawns the watchdog; logs to `cc-ci-upgrader-watchdog.log`. To resume by hand:
+`python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py resume`. (Subcommand `watchdog`, matching agents.py —
+renamed from `babysit` 2026-06-23.)
+
+**Watchdog covers ALL parts of the weekly run (2026-06-23):** (1) it runs in a tmux session
+(`<session>-watchdog`) under the persistent tmux server, NOT a Popen child — so it survives the
+`cc-ci-upgrade-all` systemd Type=oneshot whose cgroup is reaped on exit (a Popen child would be killed).
+(2) The watchdog is generic (env: `UPGRADER_SESSION/_MODEL/_DONE_MARKER/_RESUME_FILE`), so `launch-report.py`
+reuses it for the REPORT job (`cc-ci-report` session, marker `RECIPE REPORT COMPLETE`) — the report runs on
+glm-5.2 sharing the same opencode-go budget the upgrade run drains, so it can 429-stall too. `_run_pids()`
+is scoped to the managed session (title or `-s <sid>`) so the two watchdogs never kill each other's process.
+Weekly timer: Thu 22:00 America/New_York; no `LOOP_BACKEND`/`upgrader.env` override ⇒ glm-5.2 defaults.
+See [[orchestrator-backend-switch-gotcha]].
+
+**2026-07-04 hardening (after the 2026-07-03 run stalled on a HOST DISK-FULL and never resumed —
+the log-idle/429 watchdog does NOT cover an environmental wedge).** Three changes (commits 1bd156e,
+399e999), host rebuilt:
+- **Step-0 stale-image pre-reclaim** (`launch-upgrader.py` `prereclaim_cc_ci()`): before each weekly
+  run, `ssh cc-ci docker image prune -af --filter until=168h` — only images unused AND >1 week old, so
+  this week's likely-reused images stay. Best-effort; env `UPGRADER_PRERECLAIM*`. A full run's images
+  pile up week-over-week and had run root to 100%. (The 2026-07-03 pile was 93 GB / 194 images; pruning
+  took it 88%→17%.)
+- **watchdog proc-death detection**: an `opencode run` proc EXITS when the model ends its turn, so a
+  long /upgrade-all dies+restarts many times before completing (and the log mtime freezes on death,
+  making log-idle useless). `watchdog()` now resumes on "no live run proc for the session + not
+  `_completed()`", not just log-idle. Capped at `UPGRADER_MAX_RESUMES` (20) → hands off if stuck.
+  `resume()` now auto-spawns a watchdog if none is alive (skips when the watchdog itself called it).
+- **Hourly glm-5.2 SUPERVISOR** (`launch-supervisor.py`, systemd `cc-ci-upgrade-supervisor` .service +
+  .timer, hourly at :07): cheap deterministic gate — no-ops (0 tokens) if the run is complete /
+  progressing / a per-run watchdog is alive; only when a run stalled+died AND the watchdog gave up does
+  it launch a short-lived glm-5.2 agent to diagnose (disk/wedge) + drive to completion. `force`/`status`/
+  `stop` subcommands. Progress judged by `lu._run_pids(sid)` + log mtime (session_busy() is claude-tuned,
+  misreads a headless opencode run as idle — do NOT use it for opencode). Window guard: won't
+  auto-resurrect a session >96h old. Overrides via `/srv/cc-ci/upgrader.env` (e.g. `SUPERVISOR_MODEL=`).
+  Ownership: **watchdog = prompt recovery (single writer while alive); supervisor = backstop when the
+  watchdog is absent/gave up.** They're mutually exclusive (gate defers while watchdog tmux alive).
+
+**2026-07-04 finish + more watchdog fixes (commits 69dd178, ddedea8, b6a95c4).** Finishing the stalled
+2026-07-03 run exposed that a glm-5.2 `opencode run` is FLAKY for long autonomous jobs — its turn ENDS
+(proc exits) repeatedly, `--continue` of a bloated session often doesn't advance, and it can wedge on a
+hung subagent. Bugs fixed in `launch-upgrader.py`: (1) `_session_idle_min()` now measures staleness
+across the whole session TREE (parent + descendant subagents) via the opencode server time.updated —
+NOT the tmux log mtime (freezes on `--continue`) and NOT the parent alone (a 20-40min recipe subagent
+made the parent look idle → false-resume killed the productive run). (2) watchdog loop is wrapped in
+try/except so a transient blip can't kill it. (3) **`_completed()` false-positive**: it matched
+DONE_MARKER inside TOOL parts (a subagent `task` prompt / bash echoing "print UPGRADE RUN COMPLETE") →
+watchdog exited early, run wedged. Now requires the marker in the LAST assistant TEXT (prose) message.
+(4) supervisor gate also treats the run as done once the weekly summary file exists (a run can finish
+work+summary+report yet never print a clean marker). **Operational fallback that WORKED:** when the
+report agent wedged twice, published the report by hand from the deterministic path —
+`recipe-report.py survey <date>` → build the spec JSON from survey.json → `render` → `publish`. This
+is reliable and bypasses the flaky agent entirely; keep it as the go-to when an agent-driven report
+stalls. The 2026-07-03 report is live at report.ci.commoninternet.net/week-2026-07-03.html (16 GREEN,
+plausible + lasuite-drive RED, discourse 5-PR reconcile).

nix/hosts/cc-ci/configuration.nix

@@ -30,8 +30,10 @@
   # Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix).
   cc-ci-orchestrator.autoUpdate.enable = true;
 
-  # The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth.
-  cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
+  # The opencode UI: traefik (public 443, the dual-zone wildcard cert) → nginx basic auth.
+  # Domain cutover 2026-09: new name primary; legacy name kept answering during the bake window.
+  cc-ci-orchestrator.opencodeUiHost = "oc.ci.autonomic.zone";
+  cc-ci-orchestrator.opencodeUiExtraHosts = [ "oc.ci.commoninternet.net" ];
   cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy";
 
   # ---- no tailscale on this host (operator 2026-09-07) --------------------------------------
@@ -124,7 +126,13 @@
   # This host's own public names resolve to itself regardless of external DNS state (host
   # processes: the drone runner, the harness, the orchestrator; containers use the resolvers
   # above). Per-run recipe domains are random and cannot be pinned — those follow public DNS.
+  # Domain cutover 2026-09: new names pinned; legacy names kept during the bake window.
   networking.hosts."195.201.88.249" = [
+    "ci.autonomic.zone"
+    "drone.ci.autonomic.zone"
+    "report.ci.autonomic.zone"
+    "traefik.ci.autonomic.zone"
+    "oc.ci.autonomic.zone"
     "ci.commoninternet.net"
     "drone.ci.commoninternet.net"
     "report.ci.commoninternet.net"

nix/hosts/cc-ci/ssh-keys

@@ -6,6 +6,6 @@ ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQFuqUB2qNZSDNjDsjjhVA/WnnQNVAMmsUscW6OgMDN
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS notplants-orchestrator
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
-ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
-ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp cc-ci-root-ed25519@cc-ci-orchestrator-sandbox
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKmGDZC6wrOQNJAW5PPDpxgEXXrcsnIU4b3QJLtq05RQ cc-ci-loops-to-root@cc-ci
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHIa8iunWtA4mqLKV6MuiTo3RkVj2ucsk3gLL4ArMEPO notplants-orchestrator

nix/modules/cc-ci.nix

@@ -92,15 +92,13 @@
       User = "loops"; Group = "users";
       WorkingDirectory = "/srv/cc-ci";
       # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
-      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
-      # Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
-      # (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
-      # (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
-      # here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
-      # so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
-      # the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
-      # rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
-      # timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
+      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → the in-code defaults,
+      # which are opencode + LOOP_TIER=go (the only tier with a key on this host; ZEN models are
+      # dead here — the in-code default is 'go' for exactly that reason, see launch-upgrader.py).
+      # Current config: LOOP_TIER=go with LOOP_MODEL / REPORT_MODEL / SUPERVISOR_MODEL all
+      # opencode-go/glm-5.3-flash (upgrader, report, supervisor). Subagents bind deepseek via the
+      # cc-ci repo's opencode config. No rebuild needed to switch — the env file is read at each
+      # timer fire. Holds no secrets (the API key lives in the opencode config / auth.json).
       EnvironmentFile = "-/srv/cc-ci/upgrader.env";
     };
     environment = { HOME = "/home/loops"; };

nix/modules/orchestrator-host.nix

@@ -62,6 +62,16 @@ in
       description = "nginx server_name for the opencode web UI (TLS + basic auth).";
     };
 
+    opencodeUiExtraHosts = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      description = ''
+        Extra hostnames routed to the opencode UI alongside opencodeUiHost, rendered as
+        `||`-joined Host conditions on the same router. Domain cutover 2026-09: the
+        legacy oc.ci.commoninternet.net keeps answering here during the bake window.
+      '';
+    };
+
     opencodeUiHtpasswdFile = lib.mkOption {
       type = lib.types.str;
       default = "/secrets/nginx/oc-htpasswd";
@@ -178,6 +188,94 @@ SSHCFG
       '';
     };
 
+    # ---- weekly opencode CLI auto-upgrade ----------------------------------------------------
+    # opencode is not in nixpkgs: opencode-install (above) only installs the standalone CLI when
+    # the binary is MISSING, so the installed version just ages in place (it sat on 1.18.29 for
+    # weeks while 1.18.33 was out). opencode's built-in autoupdate does not cover this host: it
+    # auto-applies patch releases only and only fires on a fresh interactive TUI start, and every
+    # agent here lives inside the long-lived `opencode serve` (opencode-web). This unit is the
+    # mechanism instead. It runs weekly in the Tuesday maintenance window (an hour BEFORE the
+    # host auto-update at Tue 03:00 UTC, and clear of the Thursday recipe-upgrade run and the
+    # Sunday canonical sweep), and:
+    #   1. compares the installed version with the latest GitHub release (no-op when equal);
+    #   2. reinstalls via the official installer (same code path as opencode-install) when they
+    #      differ, and re-links ~/.local/bin/opencode;
+    #   3. restarts opencode-web so the new binary actually takes effect — sessions attached to
+    #      it (orchestrator, upgrader, report) drop and their supervisors re-attach/resume, which
+    #      is why the busy gate below must hold: a mid-flight CI/upgrade run is never cut, and a
+    #      skipped run simply retries next week.
+    # Deploying this module never itself bumps opencode: boot/activation installs stay
+    # install-if-missing in opencode-install, and this unit is timer-driven only.
+    # The outcome of each run lands in .cc-ci-logs/opencode-update-state (read by /cc-ci-status).
+    systemd.services.opencode-upgrade = {
+      description = "Weekly opencode CLI auto-upgrade (latest release → reinstall → restart opencode-web)";
+      after = [ "network-online.target" "opencode-install.service" "opencode-web.service" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; TimeoutStartSec = "30min"; };
+      path = with pkgs; [ curl bash coreutils gnugrep gnutar gzip unzip systemd util-linux procps ];
+      environment = { HOME = "/home/loops"; OPENCODE_UI_HOST = cfg.opencodeUiHost; };
+      script = ''
+        set -u
+        BIN=/home/loops/.local/bin/opencode
+        STATE=/srv/cc-ci-orch/.cc-ci-logs/opencode-update-state
+        ocver() { "$BIN" --version 2>/dev/null | tail -1 || true; }
+        state() { printf '%s result=%s installed=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > "$STATE"; chown loops:users "$STATE" 2>/dev/null || true; }
+        busy() { echo "BUSY: $1 — skipping this week's opencode upgrade (retries next week)"; state skipped "$(ocver)" "$1"; exit 0; }
+
+        # Same busy gate as auto-update.nix: never cut a CI run, the weekly upgrade or the sweep.
+        pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
+        systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
+        runuser -u loops -- tmux has-session -t cc-ci-upgrader 2>/dev/null && busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
+        runuser -u loops -- tmux has-session -t cc-ci-report 2>/dev/null && busy "the weekly report is being written (tmux cc-ci-report)"
+        if [ -r /run/secrets/bridge_drone_token ]; then
+          running=$(curl -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
+            "https://drone.ci.commoninternet.net/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
+            | grep -o '"status":"running"' | wc -l)
+          [ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
+        fi
+
+        INSTALLED=$(ocver)
+        LATEST=$(curl -fsSL -m 30 https://api.github.com/repos/anomalyco/opencode/releases/latest \
+          | grep -Po '"tag_name":\s*"v?\K[0-9][0-9.]*' || true)
+        [ -n "$LATEST" ] || { echo "could not determine the latest opencode release"; state failed "''${INSTALLED:-none}" "latest-unresolved"; exit 1; }
+        echo "installed: ''${INSTALLED:-none}   latest: $LATEST"
+        if [ "$INSTALLED" = "$LATEST" ]; then
+          echo "opencode is up to date"
+          state ok "$INSTALLED" "up-to-date"
+          exit 0
+        fi
+
+        echo "upgrading opencode: ''${INSTALLED:-none} -> $LATEST"
+        runuser -u loops -- env HOME=/home/loops bash -c 'curl -fsSL https://opencode.ai/install | bash' \
+          || { echo "install failed — the previously installed version is untouched"; state failed "''${INSTALLED:-none}" "install-failed"; exit 1; }
+        mkdir -p /home/loops/.local/bin
+        ln -sfn /home/loops/.opencode/bin/opencode /home/loops/.local/bin/opencode
+        NEWVER=$(ocver)
+        [ "$NEWVER" = "$LATEST" ] || { echo "install ran but opencode reports $NEWVER (wanted $LATEST) — not restarting"; state failed "$NEWVER" "install-mismatch"; exit 1; }
+
+        echo "restarting opencode-web so the new binary takes effect (attached sessions drop; their supervisors resume)"
+        systemctl restart opencode-web.service
+        sleep 10
+        systemctl is-active --quiet opencode-web.service || { echo "opencode-web did not come back after the upgrade"; state failed "$NEWVER" "web-restart-failed"; exit 1; }
+        code=$(curl -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$OPENCODE_UI_HOST:443:127.0.0.1" "https://$OPENCODE_UI_HOST/")
+        [ "$code" = "401" ] || { echo "opencode UI answered $code, not the 401 auth challenge"; state failed "$NEWVER" "ui-check-$code"; exit 1; }
+        echo "opencode upgraded to $NEWVER; opencode-web restarted and healthy"
+        state ok "$NEWVER" "upgraded; web restarted"
+      '';
+    };
+
+    systemd.timers.opencode-upgrade = {
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        # Weekly slot in the Tuesday maintenance window, an hour BEFORE the host auto-update
+        # (cc-ci-auto-update.timer: Tue 03:00 UTC) so a restarted opencode-web has settled before
+        # that run's health check; Persistent=false, like the auto-update — no catch-up at boot.
+        OnCalendar = "Tue *-*-* 02:00:00 UTC";
+        Persistent = false;
+        RandomizedDelaySec = "10min";
+      };
+    };
+
     # ---- opencode web server: one shared instance the opencode-backed agents attach to -------
     # Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
     systemd.services.opencode-web = {
@@ -216,6 +314,7 @@ SSHCFG
       recommendedProxySettings = true;
       virtualHosts.${cfg.opencodeUiHost} = {
         listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiBackendPort; } ];
+        serverAliases = cfg.opencodeUiExtraHosts;
         basicAuthFile = cfg.opencodeUiHtpasswdFile;
         extraConfig = ''
           # traefik sits on the docker networks (ingress 10.0.0.0/24, gwbridge 172.18.0.0/16)
@@ -256,7 +355,7 @@ SSHCFG
                 replicas: 1
                 labels:
                   - "traefik.enable=true"
-                  - "traefik.http.routers.opencode-ui.rule=Host(`${cfg.opencodeUiHost}`)"
+                  - "traefik.http.routers.opencode-ui.rule=${lib.concatStringsSep " || " (map (h: "Host(`${h}`)") ([ cfg.opencodeUiHost ] ++ cfg.opencodeUiExtraHosts))}"
                   - "traefik.http.routers.opencode-ui.entrypoints=web-secure"
                   - "traefik.http.routers.opencode-ui.tls=true"
                   - "traefik.http.services.opencode-ui.loadbalancer.server.port=${toString cfg.opencodeUiBackendPort}"

scripts/gen-cctest-skills.py

@@ -50,6 +50,76 @@ run `/cctest-intro` / `/cctest-setup-sandbox` first.
   a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
 """
 
+# Per-skill extra body sections appended after BODY_TEMPLATE (survive regeneration).
+PER_SKILL_NOTES: dict[str, str] = {
+    "recipe-upstream": """
+**Sandboxed vs non-sandboxed mode.** This skill needs only git + (optionally) the Gitea API —
+not the test server — so it can run either way. Probe, then follow that branch:
+
+- **Sandboxed** (ARM env configured: `test-ssh/.testenv` with `GITEA_USERNAME`/`GITEA_PASSWORD`/
+  `GITEA_URL`, sandbox/test instances): the canonical `recipe-upstream` script in
+  `references/recipe-maintainer/.claude/commands/recipe-upstream.md` runs as written. Note its
+  WORKSPACE probing expects `/workspace` or `~/Documents/recipe-maintainer`; on a bare host pass
+  the submodule dir explicitly instead.
+- **Non-sandboxed** (no ARM env on the host — e.g. the orchestrator, where the recipe-maintainer
+  checkout is only a pinned submodule): no sandbox/test instances are needed and NONE of the
+  setup skills are. Recipe-maintainer mirrors on `git.autonomic.zone` are publicly readable, so:
+  1. Check out the recipe if missing: `abra recipe fetch <recipe>` (lands in `~/.abra/recipes/<recipe>`)
+     — or a plain anonymous `git clone https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     if abra is unavailable.
+  2. Fetch the PR head branch from the mirror **anonymously** — no credentials in the remote URL:
+     `git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git`
+     (remote update rather than re-add if it exists), then
+     `git fetch gitea +refs/pull/<N>/head:refs/heads/<head_ref>`.
+  3. Fetch PR metadata (head/base refs, merged flag, release bump line) from
+     `https://git.autonomic.zone/api/v1/repos/recipe-maintainers/<recipe>/pulls/<N>` —
+     unauthenticated; use bot creds only if the repo turns out to be private (orchestrator hosts
+     can read them from `/srv/cc-ci-orch/.testenv` — never written anywhere else).
+  4. Everything else in the canonical script (origin/dev remote setup, release recommendation,
+     emitted next-steps) is identical.
+
+Every time the branch was prepared **here**, remember it exists only on this host — the
+operator's machine must fetch it first. Always emit this **step 0** before the push step
+(anonymous public fetch, no credentials needed):
+
+```
+  # 0. On a machine WITHOUT the branch pre-fetched, get it from the autonomic mirror
+  #    (fetch by URL — works regardless of what the local remotes are named):
+  cd <local checkout of the recipe>
+  git fetch https://git.autonomic.zone/recipe-maintainers/<recipe>.git +refs/pull/<PR_NUM>/head:refs/heads/<HEAD_REF>
+  git checkout <HEAD_REF>
+```
+
+If the operator's checkout does NOT yet have the mirror remote, emit once before the fetch:
+
+```
+  git remote add gitea https://git.autonomic.zone/recipe-maintainers/<recipe>.git
+```
+
+In both modes the final output is a set of commands for the operator to run on a machine **with
+push access to `git.coopcloud.tech`** — always print them, even when everything local is
+already prepared.
+
+**Verify the merge-base BEFORE rebasing — bases can be traps.** Upstream repos can
+carry a stray, divergent `main` ALONGSIDE the real `master` base branch. Rebase targets must be
+chosen by evidence: `git ls-remote origin | grep -E 'refs/heads/(main|master)$'`, then
+`git merge-base <head_ref> origin/<candidate>` — the correct base is the one where the upgrade
+branch's merge-base is its tip's parent (i.e. `<head_ref>` is already directly on that line —
+then NO cherry-pick is needed, the raw branch compares clean). Cherry-picking onto the wrong
+base produces the very messy 3-dot compare this guard exists to prevent (real case: gitea —
+first "rebase" landed on a stray `main` and had to be redone onto `master`).
+
+**Branch-name mismatch mirror vs upstream.** The mirror and upstream can use different base-branch
+names, and a mirror can even carry a stray, divergent `main` alongside the real base (gitea: mirror
+PR against `main`, upstream `master`; keycloak PR #9: base `main`, both mirror and upstream on
+`master`). The canonical script now detects the upstream default branch by evidence (`git
+ls-remote --symref origin HEAD`, then the git.coopcloud.tech API, then the PR base with a warning)
+and uses it for the compare URL AND the post-merge release steps — never hardcode `main`, and when
+it prints the base-mismatch warning, confirm the upgrade branch actually sits on that line
+(`git merge-base` against both candidates) before emitting the push step.
+""",
+}
+
 WRAPPER_TEMPLATE = """# {wrapped} (thin wrapper)
 
 The canonical definition of this skill lives in the **opencode** position:
@@ -99,7 +169,7 @@ def main() -> int:
         )
         canon = ROOT / ".opencode/skills" / wrapped / "SKILL.md"
         canon.parent.mkdir(parents=True, exist_ok=True)
-        canon.write_text(frontmatter + BODY_TEMPLATE.format(name=name, wrapped=wrapped))
+        canon.write_text(frontmatter + BODY_TEMPLATE.format(name=name, wrapped=wrapped) + PER_SKILL_NOTES.get(name, ""))
         thin = ROOT / ".claude/skills" / wrapped / "SKILL.md"
         thin.parent.mkdir(parents=True, exist_ok=True)
         thin.write_text(frontmatter + WRAPPER_TEMPLATE.format(name=name, wrapped=wrapped))