Merge pull request 'CI server health: image pruning, disk thresholds, reconcile-upstream' (#5 ) from review/4-ci-health into review/3-upstream-resolution

cc-ci-status: correct the ENOSPC note — inode recreation was coincidence
I recorded that recreating the runs directory with a fresh inode preceded recovery. It recurred afterwards (build 1252), so that was not the fix. The real signal is that it is INTERMITTENT and tracks concurrent activity: every failure landed while a second run or manual deploy was in flight, and every build on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a failing build. Practical guidance is therefore to wait for the host to go quiet and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is the obvious knob to test if it becomes disruptive. Root cause still not established, and the note now says so plainly rather than presenting a coincidence as understood.
2026-08-11 19:03:54 +00:00 · 2026-08-11 18:54:58 +00:00 · 2026-08-11 18:38:09 +00:00 · 2026-08-11 17:39:56 +00:00 · 2026-08-11 15:41:40 +00:00 · 2026-08-11 15:02:06 +00:00
15 changed files with 1382 additions and 39 deletions
@@ -79,12 +79,29 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
  it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
  `git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
  recipe.
+- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
+  change must satisfy — set state up through the app's interface rather than its database, gate on
+  version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
+  the app's own telemetry before concluding a test is stale.
 - **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
  (`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
  `GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
  branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
  (`/cc-ci`, `/cc-ci-adv`) or their in-flight state.

+> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
+> ```
+> cc-ci-plan/reconcile-upstream.sh <recipe>...   # or --all
+> ```
+> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
+> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
+> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
+> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
+> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
+> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
+> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
+> two CVSS-9.8 RCEs" finding.
+
 ### 5. VERIFY each PR on the CI server (deterministic; still never merge)
 A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
 the bug. Verification is deterministic (the harness), not an AI judgement.
@@ -0,0 +1,91 @@
+---
+name: cve-check-and-upgrade
+description: Security-driven upgrade run. Does a full /cve-check sweep first (per-image advisory scan of every recipe's available upgrade, with adjudication), then runs /recipe-upgrade ONLY on the recipes whose upgrade fixes at least one CVE — worst severity first — opening a verified recipe PR for each, and finally publishes one report covering both the sweep and the PRs. Recipes with no CVEs are left alone; that is the point. NEVER merges. Invoke as /cve-check-and-upgrade [recipe ...] [--min-severity high] [--capacity N] [--dry-run].
+---
+
+# cve-check-and-upgrade
+
+`/upgrade-all` upgrades everything that *has* an upgrade. **This upgrades what has a reason.** It runs
+the `/cve-check` sweep, then spends CI time only on the recipes where an upgrade actually closes a
+vulnerability, handling the worst first.
+
+Use it when you want to act on security rather than churn the whole fleet: after a vendor announcement,
+when CI capacity is short, or between weekly runs. When you only want to *know*, use `/cve-check`. When
+you want everything current regardless of CVEs, use `/upgrade-all`.
+
+**Creates PRs. Never merges.** Every PR is verified green on cc-ci and left for a human.
+
+## Arguments
+- `<recipe> …` — restrict the whole run to these recipes.
+- `--min-severity critical|high|medium|low` — only upgrade recipes whose fixed CVEs reach this
+  severity. Default **`low`** (any CVE at all justifies the upgrade). `--min-severity high` is the
+  useful "just the urgent ones" setting.
+- `--capacity N` — subagent pool size; defaults to the live `DRONE_RUNNER_CAPACITY` (the drone
+  runner's slots), matching `/upgrade-all`'s rolling-pool behaviour.
+- `--dry-run` — do the whole sweep and print exactly which recipes *would* be upgraded and why, then
+  stop without spawning a single upgrade. **Publishes no report and opens no PR.**
+
+## Procedure
+
+### 1. Sweep — run `/cve-check` in full
+Follow `.claude/skills/cve-check/SKILL.md` steps 1–5 exactly: candidate list, per-image upgrade windows,
+the advisory scan per recipe, pass-2 adjudication of anything undecided, and the severity classification.
+**Do not publish its report** — this run produces one combined report at the end instead.
+
+Keep, per recipe: the windows scanned, the CVE count, the CVE ids with severities, and whether the count
+is a floor (undetermined advisories remain) or unknown.
+
+### 2. Decide what to upgrade
+`RECIPES_TO_UPGRADE` = recipes where **the scan found ≥1 CVE** at or above `--min-severity`.
+
+Deliberate exclusions, each recorded in the report with its reason:
+- **0 CVEs** — an upgrade may exist, but nothing security-relevant. Left alone; that is the point of
+  this skill. `/upgrade-all` is what sweeps those up.
+- **`external` tier** — swept for visibility, **never upgraded here**; someone else maintains it. Flag
+  it loudly in the report if it has a critical, since the action is to tell them, not to open a PR.
+- **`UPTODATE`** — nothing available.
+- **count `?` / UNKNOWN** — do **not** upgrade blind, and do **not** treat it as clean. Put it in the
+  Addendum as needing a look. An unknown is a gap in our knowledge, not evidence of safety.
+
+Order the queue by **worst severity first** (critical → high → …), count breaking ties. If `--dry-run`,
+print this queue with each recipe's CVE ids and severities, and STOP here.
+
+### 3. Upgrade each one — via `/recipe-upgrade` subagents
+Run `/recipe-upgrade <recipe>` per queued recipe as a **subagent**, in the queue order above, as a
+**rolling pool** keeping `--capacity` (default `DRONE_RUNNER_CAPACITY`) running at once and starting the
+next as each finishes — the same concurrency discipline as `/upgrade-all` §3, and safe for the same
+reason (per-run recipe trees + app-domain locks).
+
+Each subagent does the full job: plan, implement the bump, verify green on cc-ci with `!testme`, and
+open a recipe PR. **Default mode — no `--with-tests`**: a genuinely stale test gets an explanatory PR
+comment, not a test edit.
+
+**Tell each subagent which CVEs justify its upgrade**, with ids and severities, so the PR description
+says why it exists. That is most of this skill's value to a reviewer: a PR that names the CVSS-9.8 RCE
+it closes gets merged today, an unexplained version bump waits a fortnight.
+
+Collect per recipe: PR url + number, the `!testme` verdict and build number, and any failure.
+
+### 4. Report — one page covering sweep AND PRs
+Write `/tmp/cve-spec-<DATE>.json` per `/cve-check` step 6, with these differences:
+- Rows for upgraded recipes carry the real `ci` (`build N ✓` / `RED N · <stage>`) + `ci_url`, and
+  `pr`/`pr_url`. `status` is the CI verdict (`GREEN`/`FAILED`/`STALE`); the live PR-status column
+  derives itself from `recipe` + `pr`.
+- Rows for swept-but-not-upgraded recipes keep `PENDING`/`UPTODATE` with empty `ci`/`pr`, and a
+  `notes` reason (`0 CVEs — not upgraded`, `external — maintained elsewhere`, `below --min-severity`).
+- Include `changes[]` — one entry per recipe that got a PR, describing what the upgrade changes **and
+  the CVEs it closes**.
+- Keep `"kind": "cve"`: it titles the page "The Recipe Report — CVE check" and files it as
+  `cve-<DATE>.html`, alongside the weekly editions in the same archive index.
+
+Then render + publish exactly as `/cve-check` step 7, and verify as its step 8. Print the report URL,
+`N swept · M upgraded · K PRs green · J failed`, and `CVE CHECK AND UPGRADE COMPLETE`.
+
+## Guardrails
+- **NEVER merge.** Create and verify; a human merges. Never push to true upstream.
+- **Never weaken a test** to make a PR green, and never edit a test without `--with-tests`.
+- **Never upgrade a recipe whose CVE count is unknown** on the assumption it is fine — surface it.
+- **Never upgrade an `external` recipe** here, even with a critical; report it instead.
+- **Public-safe report only** — no secrets, tokens, internal hostnames, raw logs, or spend figures.
+- If the sweep finds **nothing** at or above `--min-severity`, that is a good outcome: publish the
+  report saying so and open no PRs. Do not manufacture work.
@@ -0,0 +1,183 @@
+---
+name: cve-check
+description: Fleet-wide CVE sweep WITHOUT upgrading anything. For every recipe cc-ci deploys, works out what upgrade is available (current pinned tag → newest supported tag, per image including sidecars), runs the deterministic advisory scan over that window, adjudicates whatever the scan could not decide, and publishes a CVE report to report.ci.commoninternet.net as cve-<DATE>.html. READ-ONLY — opens no PRs, edits no recipes, runs no CI, merges nothing. Answers "what are we exposed to that an upgrade would fix?" in minutes rather than the hours a full upgrade run takes. Invoke as /cve-check [recipe ...] [--weekly-only].
+---
+
+# cve-check
+
+A **security sweep, not an upgrade run.** It answers one question for every recipe cc-ci deploys:
+
+> If we upgraded this recipe today, how many CVEs would that fix, and how bad are they?
+
+It is the cheap, safe half of `/upgrade-all`: the same version research and the same advisory scan,
+with **no implementation, no CI, and no PRs**. Use it when you want the security picture now — after a
+vendor announcement, before deciding what to prioritise, or between weekly runs. When you want the PRs
+too, use **`/cve-check-and-upgrade`**.
+
+**Read-only, absolutely.** Never edit a recipe, never open or comment on a PR, never merge, never
+deploy. The only thing it writes is its own log and the published report page.
+
+## Arguments
+- `<recipe> …` — sweep only these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
+- `--weekly-only` — skip rows tagged `external`. **Off by default on purpose**: an `external` recipe is
+  still deployed and still exposes us, so a security sweep that silently skipped it would misreport the
+  fleet's exposure. Externals are swept and clearly marked "maintained elsewhere" in the report.
+
+## Procedure
+
+> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
+> `abra` needs a TTY. Wrap every abra call: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
+> (`git` and other commands do NOT need the wrapper.)
+
+### 1. Build the candidate list
+Read `cc-ci-plan/used-recipes.md` — the canonical inventory. Take every row (both tiers), recording the
+tier per recipe; with `--weekly-only`, drop the `external` rows. An explicit recipe argument overrides
+any skip.
+
+### 2. Per recipe — establish the upgrade window WITHOUT upgrading
+
+This is `/recipe-upgrade` step 1's research, stopping before it implements anything.
+
+> ⚠️ **The same four things that silently skip recipes apply here — handle ALL FOUR:**
+> 1. **pseudo-TTY** — per the box above.
+> 2. **go-git auth to git.autonomic.zone** — recipes on the private mirror FATA
+>    `authentication required: Unauthorized`. Bake creds into origin first (idempotent, only when
+>    origin is on git.autonomic.zone):
+>    `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
+> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
+>    after. Only a genuinely dirty TRACKED tree is a skip.
+> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
+>    hand-check the registry; run the resolver, which is abra-independent and covers every image:
+>    ```
+>    python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
+>    ```
+>    It reports, per image, `newest_within_major` (the compatibility-safe pick) and
+>    `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
+>    checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
+>    vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
+>    the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
+
+**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
+`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
+`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
+research a stale checkout, and on the first real run that produced **two recipes with no survey output
+at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
+lives in **branches**, never directly on `main`, so a force-sync of `main` to upstream discards
+nothing; it also auto-closes mirror PRs whose changes upstream has already merged.
+
+> ### ⚠️ The default branch may be `master`, not `main` — check, do not assume
+> Several coopcloud recipes keep a **stale `main` alongside the real default `master`**. gitea is one:
+> `main` sits at 1.24.2-rootless while `master` has 1.27.1-rootless plus the merged PRs and the 3.6.3
+> release. Reading `main` there tells you the recipe is three releases behind and missing two CVSS-9.8
+> RCE fixes — a false alarm that reads exactly like a real one. Resolve the default branch from the
+> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
+> `git reset --hard origin/main` on a checkout that tracks `master`.
+
+**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
+nothing for images it cannot parse, and it reported "no new versions" for images that did have them
+(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
+the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
+(plausible's clickhouse) carried four CVEs.
+
+Then read versions:
+```
+set -a; . /srv/cc-ci/.testenv; set +a
+ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
+   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
+ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
+  git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
+  script -qec "abra recipe fetch $R --force -n" /dev/null; \
+  script -qec "abra recipe upgrade $R -m -n" /dev/null; \
+  git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
+```
+
+For each recipe produce **one window per image**: `current pinned tag → newest supported tag`. You need
+the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
+CRITICAL came from, and an image with no window is not counted at all.
+
+- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
+  nothing an upgrade could fix. Record it and move on.
+- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
+  leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
+  Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
+  Only report `?` once BOTH the abra check and the direct check have failed. On the first real run this
+  distinction was the difference between two false zeros and the truth (both recipes turned out fine,
+  but nothing in the survey said so).
+
+### 3. Run the advisory scan over that window
+```
+python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
+        [--image <name>=<old>:<new>]...
+```
+**One call per recipe with every image in it** — the count is a union across images, and the
+UNKNOWN guarantee only holds when a single run sees them all. Paste the markdown block verbatim into
+the per-recipe log at `/srv/cc-ci/.cc-ci-logs/cve-check/<DATE>/<recipe>.md`.
+
+### 4. Adjudicate what the scan could not decide (pass 2)
+If the block reports advisories it **could NOT judge**, or the count is **UNKNOWN**, re-run with
+`--adjudicate` and decide each open case yourself:
+
+```
+python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> … --adjudicate
+```
+
+Answer **FIXED / NOT-FIXED / STILL-UNKNOWN** per case, each with a one-line reason **citing the
+evidence shown** — never from memory of the project, which is the exact failure that let two CVSS-9.8
+gitea RCEs be published as "none". Every FIXED is added to the count; pass 1's number is a floor. The
+block also lists what pass 1 already decided — if a verdict looks wrong given its evidence, say so.
+Record your verdicts in the per-recipe log so the number is auditable.
+
+### 5. Classify severity and priority
+For each recipe collect the CVE ids with **severities** (the scan gives them, with GHSA ids). Sort the
+report rows by what an operator should deal with first:
+1. recipes with a **critical**, then **high**, then anything else with CVEs (more CVEs higher within a band);
+2. then `?` (a count that could not be established — investigate, do not ignore);
+3. then recipes with an upgrade available but **0** CVEs;
+4. then `UPTODATE`.
+
+Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin advisories.
+
+### 6. Write the report spec
+`/tmp/cve-spec-<DATE>.json`, same shape as `/recipe-report` (see `recipe-report.py`'s header), with:
+- **`"kind": "cve"`** — titles the page "The Recipe Report — CVE check" and files it as
+  `cve-<DATE>.html`. It appears in the SAME archive index as the weekly editions, suffixed
+  "— CVE check" so the two are told apart at a glance. Without this field you would overwrite that
+  date's weekly edition.
+- `date`, `subtitle` "CVE check <human date>",
+- `lead` — **one short paragraph**: fleet exposure in a sentence and what to do first.
+- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
+  `1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
+  `PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
+  rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for
+  `external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
+- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
+  established, a scan whose count is a floor with many undetermined advisories.
+- `security[]` — one entry per **critical/high** finding: recipe · CVE id(s) + severity · what it fixes
+  · **which image** it is in. Name the image: `CVE-2025-49844` is a redis flaw, and an operator reading
+  "discourse" needs to know that.
+- `changes[]` — **omit** (nothing changed; there are no PRs).
+
+**`?` must stay RARE.** Use it only when a scan ran and reported genuinely failed sources, or the count
+came back UNKNOWN and adjudication could not settle it. Never `none` for an unknown — a blank reads as
+clean. A recipe with no upgrade available is `0`, not `?`. Many `?` is a bug for the Addendum.
+
+### 7. Render and publish — via the script only
+```
+python3 /srv/cc-ci/cc-ci-plan/recipe-report.py render /tmp/cve-spec-<DATE>.json /tmp/cve-<DATE>.html
+python3 /srv/cc-ci/cc-ci-plan/recipe-report.py publish /tmp/cve-<DATE>.html <DATE> cve
+```
+All layout is owned by `recipe-report.py`. Never hand-write or post-process HTML; if `render` errors,
+fix the spec JSON and re-render. **Public page — no secrets, tokens, internal hostnames, raw logs, or
+any billing/spend figures.**
+
+### 8. Verify and stop
+`curl -fsS https://report.ci.commoninternet.net/cve-<DATE>.html` renders and the index lists it. Print
+the URL, a one-line summary (`N recipes swept · M with CVEs · K critical`), and `CVE CHECK COMPLETE`,
+then go idle. One-shot — do not loop, and do not start upgrading anything.
+
+## Guardrails
+- **Read-only.** No PRs, no edits, no merges, no deploys, no CI runs. If a recipe looks urgent, say so
+  in the report — do not act on it. `/cve-check-and-upgrade` is the skill that acts.
+- **Never report `0` for something you could not scan.** `0` means checked-and-clean; unknown is `?`.
+- A count with undetermined advisories is a **floor** — say so in the notes rather than rounding away.
+- **Public-safe output only.**
@@ -312,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**:
  Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
  stale test; operator to decide).
 - **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
+  0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
+     it is written against the failures this pipeline has actually produced. The two that matter most
+     here: **set state up through the app's own interface, never its database** (a plausible fixture
+     that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
+     and **gate on version rather than writing a fixture that supports both** — old-version tests can
+     simply be deleted, since the older version is only exercised through the upgrade tier.
  1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
     never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
  2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
@@ -73,6 +73,29 @@ done
 # 5) Stray exited containers (debug one-shots) — best-effort prune.
 docker container prune -f >/dev/null 2>&1 || true

+# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
+#    nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
+#    the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
+#    `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
+#    from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
+#    "failure" and looked like recipe regressions.
+#
+#    Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
+#    `image prune -a` removes only images no container references, so anything deployed (infra +
+#    warm-* canonicals) is untouched; anything else is re-pulled on demand.
+#
+#    Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
+#    data-warm and their volumes are legitimately dangling between runs.
+DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
+used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
+if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
+  echo "  disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
+  freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
+  echo "  reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
+else
+  echo "  disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
+fi
+
 if [ "$removed" -eq 0 ]; then
  echo "== orphan sweep: clean (nothing to remove) =="
 else
@@ -78,8 +78,38 @@ ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls
 systemctl --failed --no-legend; df -h / | tail -1; tmux ls
 ```
 - Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
-  note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable →
+  note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
  HIGH: recommend `hetzner-server-recovery`.
+  > **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
+  > pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
+  > That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
+  > pruned them), the filesystem filled during a run, and the harness died at startup with
+  > `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
+  > spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
+  > `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
+- **!testme actually produces results** (the check that would have caught the above days earlier):
+  the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
+  dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
+  it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way.
+  Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
+  and the drone step log"*. The step log lives in drone's sqlite
+  (`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
+  read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
+  (distroless container, swarm secret).
+  > **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
+  > died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
+  > mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
+  > stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
+  > `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
+  > inode (it recurred afterwards — that apparent fix was coincidence).
+  >
+  > **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
+  > landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
+  > still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
+  > never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
+  > and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
+  > `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
+  > if it becomes disruptive.
 - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
  has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
  A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
@@ -106,8 +136,15 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls
 1. <finding> → /<skill> (or operator action)
 …
 ```
+When a finding is that the fleet's **security exposure is unknown** — the last weekly run failed or
+is stale, so nobody has scanned for CVEs recently — the recommended step is **`/cve-check`** (read-only,
+minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend
+**`/cve-check-and-upgrade`** (add `--min-severity high` when only the urgent ones matter). Prefer
+`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
+
 `ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
-CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
+CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
+producing results.json, disk under
 thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
 ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).

@@ -83,8 +83,29 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
  changed upstream, what the test currently asserts.
 - **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).

+> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
+> ```
+> cc-ci-plan/reconcile-upstream.sh <recipe>...   # or --all
+> ```
+> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
+> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
+> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
+> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
+> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
+> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
+> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
+> two CVSS-9.8 RCEs" finding.
+
 ### 2. For each stale test — author the minimal test update (AI; never weaken)

+> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
+> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
+> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
+> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
+> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
+> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
+> wait but **never the assertion**.
+
 Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
 Swarm). For each `STALE_TESTS` entry:

@@ -31,6 +31,14 @@ Then present the roster grouped as follows, and close with the situation guide.
  PR). `--with-tests` also fixes that recipe's stale test.
 - **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.

+**Security (CVEs)**
+- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
+  upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
+  Read-only and quick — the "what are we exposed to?" answer without an upgrade run.
+- **/cve-check-and-upgrade** — the same sweep, then open verified PRs **only** for the recipes whose
+  upgrade actually fixes a CVE, worst severity first. `--min-severity high` for just the urgent ones.
+  Never merges.
+
 **Tests**
 - **/cc-ci-tests-update** — fleet-wide stale-test cleanup: find tests broken by legitimate
  upstream changes, fix without weakening, verify, merge the test PRs.
@@ -73,6 +81,9 @@ ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-ccte
 | "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
 | "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
 | "The report site is stale/missing a week" | `/recipe-report` |
+| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
+| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
+| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
 | "Tests are red because upstream changed" | `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <r> --with-tests` |
 | "A CI run failed and I don't know why" | `/ci-test-review` |
 | "Update the CI server OS/deps" | `/cc-ci-server-update` |
@@ -164,6 +164,21 @@ def _vkey(v: str | None) -> tuple:
    return tuple(out)


+def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool:
+    """Does a patched version on the TARGET's own release line sit ABOVE the target?
+
+    Projects maintain several branches at once and backport per branch, so "some patched version is
+    inside the numeric window" is not the same as "the version we land on has the fix". ClickHouse
+    fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20; an upgrade landing on 23.10.4.25 crosses the
+    23.9 fix numerically but is still BELOW its own line's fix, so it does NOT have it. When the
+    advisory names a fix on the target's own line and the target is older than it, that is proof of
+    absence and outranks any other candidate."""
+    if len(kt) < 2:
+        return False
+    line = kt[:2]
+    return any(c[:2] == line and c > kt for c in cands if len(c) >= 2)
+
+
 def _within(kf: tuple, kt: tuple, c: tuple) -> bool:
    """Is patched-version `c` inside the window (kf, kt] — exclusive lower, inclusive upper?

@@ -333,6 +348,19 @@ def _releases(owner: str, repo: str, max_pages: int = 4) -> list[tuple[str, str]
    return out


+def _source_repo(source: str) -> tuple[str, str] | None:
+    """(owner, repo) for a source, whether it is an advisory feed or a vendor page on GitHub.
+
+    A CVE that appears ONLY on a vendor page still deserves the release-note method when that page
+    lives on GitHub — mailu announces its Roundcube CVEs on github.com/Mailu/Mailu/releases and
+    nowhere structured, so requiring an advisory feed sent a deterministic case to pass 2."""
+    if source.startswith("github-advisories:"):
+        owner, _, repo = source.split(":", 1)[1].partition("/")
+        return (owner, repo) if owner and repo else None
+    m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", source)
+    return (m.group(1), m.group(2).removesuffix(".git")) if m else None
+
+
 def release_fix_versions(source: str, cve: str) -> list[str]:
    """Release tags whose notes NAME this CVE — a deterministic fix version when the advisory has none.

@@ -342,10 +370,10 @@ def release_fix_versions(source: str, cve: str) -> list[str]:
    (CVE-2025-32023 → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Ignoring that evidence undercounted
    discourse by 12 CVEs, so this is checked BEFORE giving up on an advisory.
    """
-    if not source.startswith("github-advisories:"):
+    ref = _source_repo(source)
+    if not ref:
        return []
-    owner, _, repo = source.split(":", 1)[1].partition("/")
-    return [tag for tag, body in _releases(owner, repo) if cve in body]
+    return [tag for tag, body in _releases(*ref) if cve in body]


 def advisory_text(ghsa: str, source: str | None = None) -> dict:
@@ -652,6 +680,10 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
                    continue
                patched = e.get("patched") or ""
                cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
+                if kf and kt and _superseded_on_target_line(kt, cands):
+                    # The target's own line got the fix LATER than the target: not fixed here.
+                    e.setdefault("classification", "outside-window")
+                    continue
                if kf and kt and any(_within(kf, kt, c) for c in cands):
                    got.add(cve)
                elif not patched or PLACEHOLDER_RE.search(patched):
@@ -707,17 +739,31 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
    # THIRD METHOD: before declaring an advisory undecidable, look for the CVE id in the project's
    # own release notes. A tag that names it, inside the window, IS the fix version the advisory
    # failed to publish. Deterministic and citable — not a judgement call.
-    resolved_by_release = {}
-    for cve in sorted(indeterminate - fixed_set):
+    # A window is keyed by advisory-feed source; map it to its repo so a vendor page on the SAME
+    # repo can be judged by the same window.
+    win_by_repo = {}
+    for wsrc, wv in windows.items():
+        ref = _source_repo(wsrc)
+        if ref:
+            win_by_repo[ref] = wv
+    resolved_by_release, already_fixed = {}, set()
+    candidates = set(indeterminate) | {
+        c for c in unknown
+        if not any(s in windows for s in report["cves"][c]["sources"])
+        and any(_source_repo(s) in win_by_repo for s in report["cves"][c]["sources"])
+    }
+    for cve in sorted(candidates - fixed_set):
        e = report["cves"][cve]
        for src in e["sources"]:
-            if src not in windows:
+            ref = _source_repo(src)
+            if src not in windows and ref not in win_by_repo:
                continue
-            wf, wt = windows[src]
+            wf, wt = windows[src] if src in windows else win_by_repo[ref]
            kf, kt = _vkey(wf), _vkey(wt)
            if not (kf and kt):
                continue
-            hits = [t for t in release_fix_versions(src, cve) if _within(kf, kt, _vkey(t))]
+            naming = release_fix_versions(src, cve)
+            hits = [t for t in naming if _within(kf, kt, _vkey(t))]
            if hits:
                fixed_set.add(cve)
                resolved_by_release[cve] = sorted(hits)
@@ -725,10 +771,23 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
                                       f"{', '.join(sorted(hits))}) via {src}")
                e["fix_versions_from_release_notes"] = sorted(hits)
                break
+            # Naming releases exist but ALL predate the version we were already on: the fix shipped
+            # before this upgrade, so the upgrade did not deliver it. That is a DECISION, not an
+            # unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 advisories all fixed by 8.6.3 or
+            # earlier, and reporting them as "could not judge" overstates the uncertainty.
+            if naming and all(_vkey(t) and not _within(kf, kt, _vkey(t)) for t in naming) \
+                    and max(_vkey(t) for t in naming if _vkey(t)) <= kf:
+                e["classification"] = ("outside-window: fixed in "
+                                       f"{', '.join(sorted(naming))}, all at or before {wf}")
+                e["fix_versions_from_release_notes"] = sorted(naming)
+                already_fixed.add(cve)
+                break
    if resolved_by_release:
        report["resolved_by_release_notes"] = resolved_by_release

-    indeterminate -= fixed_set
+    indeterminate -= fixed_set | already_fixed
+    if already_fixed:
+        report["already_fixed_before_upgrade"] = sorted(already_fixed)
    for cve in indeterminate:
        report["cves"][cve]["classification"] = "indeterminate: no fix version published"
    unknown = [c for c in unknown if c not in fixed_set]
@@ -738,8 +797,19 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
    report["unclassified"] = sorted(unknown)
    # NEVER report 0 for something we could not determine — a 0 asserts safety. If ANY requested
    # window could not be ordered at all, the total is UNKNOWN rather than a partial number.
-    report["count_known"] = not unresolved_any
-    report["cve_count_fixed"] = len(fixed_set) if not unresolved_any else None
+    # A scan with NO usable source has not measured anything, so it must not report a number —
+    # least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and
+    # still produced "0 identified", which was then published as a clean 0 in a CVE report.
+    usable_sources = [
+        s for s in report["sources"]
+        if s["status"] == "ok" or s["status"].startswith("no-advisories-published")
+    ]
+    no_sources = not usable_sources
+    if no_sources:
+        report["no_usable_sources"] = True
+    report["count_known"] = not unresolved_any and not no_sources
+    report["cve_count_fixed"] = (len(fixed_set)
+                                 if (not unresolved_any and not no_sources) else None)
    report["cve_count_total_seen"] = len(report["cves"])
    # Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
    # no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
@@ -762,10 +832,16 @@ def markdown(rep: dict) -> str:
         f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
    if not rep.get("count_known", True):
        L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
-        L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes numeric "
-                 "ordering meaningless across this jump, so no advisory could be classified. Render "
-                 "this recipe's cve cell as `?`, never `0`. Read the vendor's release notes for the "
-                 "jump and count by hand.")
+        if rep.get("no_usable_sources"):
+            L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry "
+                     "file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so "
+                     "nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add "
+                     "the registry file.")
+        else:
+            L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes "
+                     "numeric ordering meaningless across this jump, so no advisory could be "
+                     "classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's "
+                     "release notes for the jump and count by hand.")
        if rep["unclassified"]:
            L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
                     f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))
@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+"""audit-sources — are we still looking in the right place for each recipe's updates?
+
+A recipe tracks an image repo and a set of registry URLs. Upstreams move: they rename the image,
+switch registry, archive the GitHub repo, or split a community edition out of the original. When that
+happens nothing errors — the old repo simply stops receiving tags, and the recipe looks "up to date"
+forever while real releases happen somewhere else.
+
+plausible is the worked example. It tracked `plausible/analytics` on Docker Hub; upstream moved to
+`ghcr.io/plausible/community-edition`. The old repo still exists and still serves v2.0.0, so every
+survey said "no upgrades available" while v3 shipped elsewhere.
+
+This reports the signals that catch that, per image and per registry URL:
+
+  * IMAGE GONE QUIET     — newest tag is older than --quiet-days (default 365). The single strongest
+                           signal that releases moved somewhere else.
+  * DEPRECATION WORDING  — the registry description says deprecated / moved / no longer maintained.
+  * GITHUB REPO ARCHIVED — upstream archived it.
+  * GITHUB REPO RENAMED  — the API redirects to a different owner/name than we ask for.
+  * GITHUB REPO GONE     — 404.
+
+Everything is a SIGNAL, not a verdict: a genuinely stable image (mumble, custom-html) can be quiet
+for good reason. The output is for a human to judge, so each finding says what was measured.
+
+    audit-sources.py [recipe ...] [--ssh HOST] [--quiet-days N] [--json]
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+_spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HERE, "resolve-images.py"))
+RI = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(RI)
+
+REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream"))
+USED_RECIPES = os.path.join(HERE, "used-recipes.md")
+DEPRECATION_RE = re.compile(
+    r"\b(deprecat|no longer maintain|unmaintained|superseded|moved to|migrated to|"
+    r"has moved|discontinued|end.of.life|archived)\b", re.I)
+
+
+def _days_since(iso: str | None) -> int | None:
+    if not iso:
+        return None
+    try:
+        d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
+    except ValueError:
+        return None
+    return (datetime.now(timezone.utc) - d).days
+
+
+def hub_repo_meta(repo: str) -> dict:
+    """Docker Hub repo metadata: when it was last pushed to, and how it describes itself."""
+    try:
+        d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}", RI._hub_auth())
+    except urllib.error.HTTPError as e:
+        return {"status": f"HTTP {e.code}"}
+    except Exception as e:  # noqa: BLE001
+        return {"status": f"{type(e).__name__}"}
+    text = f"{d.get('description') or ''}\n{d.get('full_description') or ''}"
+    m = DEPRECATION_RE.search(text)
+    return {"status": "ok", "last_updated": d.get("last_updated"),
+            "deprecation_hint": (m.group(0) if m else None),
+            "archived": bool(d.get("is_archived") or d.get("status") == "inactive")}
+
+
+def github_repo_meta(owner: str, repo: str) -> dict:
+    """GitHub repo state — archived, renamed (the API answers with the CURRENT full_name), or gone."""
+    hdrs = {"Accept": "application/vnd.github+json"}
+    tok = RI._gh_token()
+    if tok:
+        hdrs["Authorization"] = f"Bearer {tok}"
+    try:
+        d = RI._json(f"https://api.github.com/repos/{owner}/{repo}", hdrs)
+    except urllib.error.HTTPError as e:
+        return {"status": f"HTTP {e.code}"}
+    except Exception as e:  # noqa: BLE001
+        return {"status": f"{type(e).__name__}"}
+    asked, got = f"{owner}/{repo}".lower(), (d.get("full_name") or "").lower()
+    return {"status": "ok", "archived": bool(d.get("archived")), "pushed_at": d.get("pushed_at"),
+            "renamed_to": (d.get("full_name") if got and got != asked else None),
+            "description": d.get("description") or ""}
+
+
+def newest_tag_date(registry: str, repo: str, tag: str) -> str | None:
+    """When was the repo's newest same-shape tag pushed? Docker Hub only (it dates its tags)."""
+    if registry not in ("docker.io", "registry-1.docker.io"):
+        return None
+    try:
+        d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}/tags"
+                     f"?page_size=100&ordering=last_updated", RI._hub_auth())
+    except Exception:  # noqa: BLE001
+        return None
+    want = RI.shape(tag)
+    for row in d.get("results", []):
+        if RI.shape(row.get("name") or "") == want:
+            return row.get("last_updated")
+    return (d.get("results") or [{}])[0].get("last_updated")
+
+
+def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
+    out = {"recipe": recipe, "findings": [], "images": [], "sources": []}
+    try:
+        refs = (RI.compose_images_ssh(recipe, ssh, "~/.abra/recipes") if ssh
+                else RI.compose_images(recipe, RI.RECIPE_DIR))
+    except Exception as e:  # noqa: BLE001
+        out["findings"].append({"level": "error", "what": f"could not read compose: {e}"})
+        return out
+
+    for ref in refs:
+        if "${" in ref:
+            continue
+        info = RI.parse_ref(ref)
+        row = {"ref": ref, "registry": info["registry"], "repo": info["repo"], "tag": info["tag"]}
+        if info["registry"] in ("docker.io", "registry-1.docker.io"):
+            meta = hub_repo_meta(info["repo"])
+            row.update(meta)
+            newest = newest_tag_date(info["registry"], info["repo"], info["tag"])
+            row["newest_tag_pushed"] = newest
+            age = _days_since(newest)
+            row["newest_tag_age_days"] = age
+            if age is not None and age > quiet_days:
+                out["findings"].append({
+                    "level": "warn", "what": "image has gone quiet",
+                    "detail": f"{info['repo']}: newest {RI.shape(info['tag'])}-shaped tag pushed "
+                              f"{age} days ago — releases may have moved elsewhere"})
+            if meta.get("deprecation_hint"):
+                out["findings"].append({
+                    "level": "warn", "what": "registry text suggests deprecation",
+                    "detail": f"{info['repo']}: says {meta['deprecation_hint']!r}"})
+            if meta.get("archived"):
+                out["findings"].append({"level": "warn", "what": "registry repo archived/inactive",
+                                        "detail": info["repo"]})
+        out["images"].append(row)
+
+    urls, reg_path = ([], None)
+    try:
+        urls, reg_path = _registry_urls(recipe)
+    except Exception:  # noqa: BLE001
+        pass
+    if reg_path is None:
+        out["findings"].append({"level": "warn", "what": "no upstream registry file",
+                                "detail": f"cc-ci-plan/upstream/{recipe}.md is missing — the advisory "
+                                          f"scan has nowhere to look"})
+    seen = set()
+    for u in urls:
+        m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u)
+        if not m:
+            continue
+        owner, repo = m.group(1), m.group(2).removesuffix(".git")
+        if (owner, repo) in seen:
+            continue
+        seen.add((owner, repo))
+        meta = github_repo_meta(owner, repo)
+        row = {"repo": f"{owner}/{repo}", **meta}
+        age = _days_since(meta.get("pushed_at"))
+        row["pushed_age_days"] = age
+        out["sources"].append(row)
+        if meta.get("status") != "ok":
+            out["findings"].append({"level": "warn", "what": "registry source unreachable",
+                                    "detail": f"{owner}/{repo}: {meta['status']}"})
+            continue
+        if meta.get("renamed_to"):
+            out["findings"].append({"level": "alert", "what": "GitHub repo has MOVED",
+                                    "detail": f"{owner}/{repo} now answers as {meta['renamed_to']}"})
+        if meta.get("archived"):
+            out["findings"].append({"level": "alert", "what": "GitHub repo is ARCHIVED",
+                                    "detail": f"{owner}/{repo} — upstream development has stopped here"})
+        if age is not None and age > quiet_days:
+            out["findings"].append({"level": "warn", "what": "GitHub repo quiet",
+                                    "detail": f"{owner}/{repo}: last push {age} days ago"})
+    return out
+
+
+def _registry_urls(recipe: str):
+    path = os.path.join(REGISTRY_DIR, f"{recipe}.md")
+    if not os.path.exists(path):
+        return [], None
+    text = open(path).read()
+    urls = []
+    for u in re.findall(r"https?://[^\s)|\]]+", text):
+        u = u.rstrip("`'\"*.,;:>)")
+        if u and u not in urls:
+            urls.append(u)
+    return urls, path
+
+
+def all_recipes() -> list[str]:
+    out = []
+    for ln in open(USED_RECIPES):
+        ln = ln.strip()
+        if not ln or ln.startswith("#") or ln.startswith("`"):
+            continue
+        parts = ln.split()
+        if len(parts) >= 2 and parts[1] in ("weekly", "external"):
+            out.append(parts[0])
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("recipes", nargs="*")
+    ap.add_argument("--ssh", default=None)
+    ap.add_argument("--quiet-days", type=int, default=365)
+    ap.add_argument("--json", action="store_true")
+    a = ap.parse_args()
+
+    recipes = a.recipes or all_recipes()
+    reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes]
+    if a.json:
+        print(json.dumps(reports, indent=2))
+        return 0
+    alerts = 0
+    for rep in reports:
+        fs = rep["findings"]
+        mark = "OK " if not fs else ("!! " if any(f["level"] == "alert" for f in fs) else " ? ")
+        print(f"{mark} {rep['recipe']}")
+        for f in fs:
+            alerts += f["level"] == "alert"
+            print(f"      [{f['level']}] {f['what']}: {f.get('detail','')}")
+    print(f"\n{len(reports)} recipes audited · "
+          f"{sum(len(r['findings']) for r in reports)} findings · {alerts} alerts")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -9,7 +9,11 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica
  survey [DATE]            JSON of the run + every recipe's open PRs + CI verdict + per-recipe upgrade
                           notes (breaking-change/CVE analysis), and the /upgrade-all summary.
  render SPEC.json OUT.html   render the agent's report spec -> a self-contained newspaper HTML page
-  publish OUT.html DATE    copy to cc-ci:/var/lib/cc-ci-reports/week-DATE.html and regen the archive index
+  publish OUT.html DATE [KIND]  copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
+                           archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
+                           (a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
+                           newest first, each row suffixed "full" or "CVE check"; the distinct
+                           filename prefix just stops a sweep overwriting a weekly edition.

 Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
 Bulletin → per-recipe "What changed".
@@ -44,6 +48,10 @@ LOGDIR = "/srv/cc-ci/.cc-ci-logs"
 TESTENV = "/srv/cc-ci/.testenv"
 INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
 HOST_REPORTS = "/var/lib/cc-ci-reports"
+# Both kinds live in ONE archive, distinguished by a suffix on a common title.
+# prefix -> (page title, index label)
+KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),
+         "cve":  ("The Recipe Report — CVE check", "{d} — CVE check")}


 def _env():
@@ -215,8 +223,16 @@ def _table(rows, repo_url=None):
        if repo_url and r.get("recipe") in repo_url:
            name = f'<a href="{repo_url[r["recipe"]]}">{name}</a>'
        cve = r.get("cve")
-        cve_cell = (f'<span class="cve">{int(cve)}</span>' if isinstance(cve, (int, float)) and cve
-                    else '<span class="muted">none</span>')
+        # "?" = advisory scan absent or had failed sources → count NOT authoritative. Per the
+        # /recipe-report guardrail this must NEVER render as "none" (a blank-that-reads-clean is
+        # exactly how two CVSS-9.8 gitea RCEs were misreported as "none" on 2026-08-07). A positive
+        # int is the confirmed CVE count; 0/omit is a confirmed-clean scan.
+        if isinstance(cve, str) and cve.strip() == "?":
+            cve_cell = '<span class="muted" title="advisory scan incomplete or absent — CVE count unknown">?</span>'
+        elif isinstance(cve, (int, float)) and cve:
+            cve_cell = f'<span class="cve">{int(cve)}</span>'
+        else:
+            cve_cell = '<span class="muted">none</span>'
        ci = _esc(r.get("ci"))
        if r.get("ci_url"):
            ci = f'<a href="{_esc(r["ci_url"])}">{ci}</a>'
@@ -270,8 +286,10 @@ def _mast():

 def render(spec_path, out_path):
    s = json.load(open(spec_path))
+    kind = s.get("kind", "week")
+    title = KINDS.get(kind, KINDS["week"])[0]
    gen = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
-    sub = s.get("subtitle", "Week of " + s["date"])
+    sub = s.get("subtitle", ("Week of " if kind == "week" else "CVE check ") + s["date"])
    lead = s.get("lead", "") or ""
    # Auto-link recipe-name mentions in the lead to their mirror repos.
    gitea = _env().get("GITEA_URL", "git.autonomic.zone")
@@ -285,7 +303,9 @@ def render(spec_path, out_path):
            f'<span>report.ci.commoninternet.net</span><span>{gen}</span></div>'
            f'<div class="lead">{lead}</div>')
    # 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column.
-    body += f'<h2>The full wire — every recipe, in priority order</h2>{_table(s.get("table"), repo_url)}'
+    wire = ("The full wire — every recipe, in priority order" if kind == "week"
+            else "Advisory sweep — every recipe, worst first")
+    body += f'<h2>{wire}</h2>{_table(s.get("table"), repo_url)}'
    # 2) addendum — special issues to look into (normal-size header); omitted entirely if there are none.
    add = [a for a in (s.get("addendum") or []) if str(a).strip()]
    if add:
@@ -298,19 +318,30 @@ def render(spec_path, out_path):
    # 4) what changed — a short section per recipe that has a PR
    if s.get("changes"):
        body += f'<h2>What changed</h2>{_changes(s.get("changes"), repo_url)}'
-    body += (f'<footer>The Recipe Report · generated {gen} · '
+    body += (f'<footer>{title} · generated {gen} · '
             f'<a href="https://ci.commoninternet.net/">dashboard</a> · <a href="./">archive</a></footer>')
-    open(out_path, "w").write(_page("The Recipe Report — " + s["date"], body))
+    open(out_path, "w").write(_page(f"{title} · " + s["date"], body))
    print("wrote", out_path)


-def publish(html_path, date):
-    page = f"week-{date}.html"
+def publish(html_path, date, kind="week"):
+    if kind not in KINDS:
+        print(f"unknown kind {kind!r}; expected one of {', '.join(KINDS)}"); sys.exit(2)
+    page = f"{kind}-{date}.html"
    subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/{page}"], input=open(html_path, "rb").read(), check=True)
-    listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/week-*.html 2>/dev/null"],
-                             capture_output=True, text=True).stdout.split()
-    dates = sorted({os.path.basename(p)[5:-5] for p in listing}, reverse=True)
-    lis = "\n".join(f'<li><a href="week-{d}.html">Week of {d}</a><span class="d">{d}</span></li>' for d in dates)
+    # One index over BOTH families, newest first, each row labelled by its kind — an operator looking
+    # for "the latest security picture" should not have to know which skill produced which page.
+    entries = []
+    for k in KINDS:
+        listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/{k}-*.html 2>/dev/null"],
+                                 capture_output=True, text=True).stdout.split()
+        for pth in listing:
+            d = os.path.basename(pth)[len(k) + 1:-5]
+            if re.fullmatch(r"\d{4}-\d{2}-\d{2}", d):
+                entries.append((d, k))
+    lis = "\n".join(
+        f'<li><a href="{k}-{d}.html">{KINDS[k][1].format(d=d)}</a><span class="d">{d}</span></li>'
+        for d, k in sorted(set(entries), reverse=True))
    idx = _page("The Recipe Report — Archive", _mast() +
                '<div class="dateline"><span>Weekly review of Co-op Cloud recipe upgrades &amp; CI</span>'
                '<span>report.ci.commoninternet.net</span></div>'
@@ -328,7 +359,7 @@ def main():
    elif cmd == "render":
        render(a[1], a[2])
    elif cmd == "publish":
-        publish(a[1], a[2])
+        publish(a[1], a[2], a[3] if len(a) > 3 else "week")
    else:
        print(__doc__); sys.exit(2)

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# reconcile-upstream — sync recipe mirrors from TRUE upstream. Run this FIRST, always.
+# ----------------------------------------------------------------------------------
+# Every recipe we maintain is a MIRROR of a coopcloud recipe. Work done against a stale
+# mirror is wasted or wrong, in three ways we have actually hit:
+#
+#   1. A PR whose changes upstream ALREADY MERGED. mailu #6 (2024.06.57 + redis 8.10,
+#      two internet-facing Roundcube CVEs) sat open and was reported as the fix for
+#      those CVEs — while upstream had merged and released it as 3.1.3+2024.06.57. The
+#      work was done; only our mirror was behind.
+#   2. A survey that reads the stale mirror and reports "no upgrades available", so a
+#      recipe silently drops out of the weekly run.
+#   3. Reading the WRONG BRANCH. Several coopcloud recipes keep a stale `main` beside
+#      the real default `master` — gitea's `main` is at 1.24.2-rootless while `master`
+#      has 1.27.1-rootless plus the merged PRs. Reading `main` there says the recipe is
+#      three releases behind and missing two CVSS-9.8 RCE fixes, which reads exactly
+#      like a real finding. open-recipe-pr.sh resolves the default branch itself
+#      (main OR master) — never hand-pick one.
+#
+# This is deterministic: it force-syncs each mirror's `main` to upstream's default
+# branch and closes any mirror PR whose changes are already upstream. No AI judgement.
+#
+#   reconcile-upstream.sh <recipe>...     # specific recipes
+#   reconcile-upstream.sh --all           # every recipe in used-recipes.md
+#
+# Safe to run repeatedly; a mirror already in sync is a no-op. Recipe work lives in
+# BRANCHES, never on mirror `main`, so force-syncing `main` discards nothing.
+set -o errexit -o nounset -o pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ORCH="$(dirname "$HERE")"
+SSH="${SSH:-cc-ci}"
+TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
+RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
+USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
+
+[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
+set -a; . "$TESTENV"; set +a
+: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
+
+if [ "${1:-}" = "--all" ]; then
+  mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
+else
+  [ "$#" -gt 0 ] || { echo "usage: reconcile-upstream.sh <recipe>... | --all" >&2; exit 2; }
+  RECIPES=("$@")
+fi
+
+synced=0; closed=0; failed=0
+for r in "${RECIPES[@]}"; do
+  echo "── $r"
+  if out="$(ssh "$SSH" "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s $r --reconcile-only" < "$RECONCILE" 2>&1)"; then
+    printf '%s\n' "$out" | grep -E "Force-syncing|already in sync|closed PR|still open|✓" | sed 's/^/   /' || true
+    synced=$((synced + 1))
+    closed=$((closed + $(printf '%s' "$out" | grep -c "closed PR" || true)))
+  else
+    printf '%s\n' "$out" | tail -3 | sed 's/^/   /'
+    echo "   ✗ FAILED — do NOT proceed against this mirror until it reconciles"
+    failed=$((failed + 1))
+  fi
+done
+
+echo
+echo "reconcile-upstream: ${synced} mirror(s) synced, ${closed} already-upstream PR(s) closed, ${failed} failed"
+[ "$failed" -eq 0 ]
@@ -0,0 +1,459 @@
+#!/usr/bin/env python3
+"""resolve-images — what version is each of a recipe's images on, and what is newest?
+
+An abra-independent version resolver. `abra recipe upgrade` is the normal path, but it has a hard
+failure mode: an image pinned with BOTH a tag and a digest makes it FATA and abandon the WHOLE
+recipe — even images it already parsed. immich pins two that way:
+
+    ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf6…
+    docker.io/valkey/valkey:9@sha256:3acc…
+
+so immich contributes NO version data at all and silently drops out of every survey. That is
+indistinguishable from "up to date" unless a human notices the missing row — which is exactly how it
+kept getting skipped, and why a CVE sweep reported it as unknown.
+
+This reads the compose files directly and queries the registries itself, so a digest pin is just a
+digest pin. Output is JSON (default) or a table.
+
+    resolve-images.py <recipe> [--ssh HOST] [--recipe-dir DIR] [--table] [--only IMAGE]
+
+The cc-ci host has no python3, so `--ssh cc-ci` reads the compose files from that host's checkout
+over ssh and does the resolving locally. That keeps the source of truth the SAME tree abra and CI
+use, rather than a second copy that can drift.
+
+TAG SHAPES. Registries mix wildly different tag conventions in one repo, so "newest" is meaningless
+without a shape. Each tag is reduced to a signature by replacing digit runs with '#':
+
+    v3.1.0                              -> v#.#.#
+    1.27.1-rootless                     -> #.#.#-rootless
+    8.10-alpine                         -> #.#-alpine
+    14-vectorchord0.4.3-pgvectors0.2.0  -> #-vectorchord#.#.#-pgvectors#.#.#
+
+Only tags sharing the CURRENT pin's shape are candidates. That keeps `-alpine` on `-alpine`, and
+stops a `latest`/`release`/`sha-…` tag from ever being proposed as an upgrade.
+
+TWO ANSWERS, NOT ONE. It reports `newest_same_shape` AND `newest_within_major` (same leading number).
+For a plain app image they usually agree. For a compatibility-pinned sidecar they do not, and taking
+the max would be wrong: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
+versions that immich-server is built against, so jumping pg major because a newer tag exists breaks
+the deployment. The caller picks; this tool refuses to guess and shows both.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import shlex
+import subprocess
+import gzip
+import json
+import os
+import re
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+UA = "cc-ci-resolve-images (+https://git.autonomic.zone/recipe-maintainers/cc-ci)"
+TIMEOUT = int(os.environ.get("RESOLVE_IMAGES_TIMEOUT", "45"))
+RECIPE_DIR = os.environ.get("ABRA_RECIPE_DIR", os.path.expanduser("~/.abra/recipes"))
+MAX_TAG_PAGES = int(os.environ.get("RESOLVE_IMAGES_MAX_PAGES", "40"))
+
+IMAGE_RE = re.compile(r"""^\s*image:\s*["']?([^"'\s]+)["']?\s*$""", re.M)
+
+
+RETRIES = int(os.environ.get("RESOLVE_IMAGES_RETRIES", "4"))
+
+
+def _fetch(url: str, headers: dict | None = None) -> bytes:
+    """GET with backoff on rate limits.
+
+    Docker Hub throttles anonymous clients hard, and a sweep re-reads the same popular repos
+    (nginx, redis, postgres) for recipe after recipe. A 429 mid-sweep used to surface as
+    'unresolved', which is indistinguishable from a real lookup failure — so retry, and let the
+    per-repo cache below remove most of the requests entirely."""
+    h = {"User-Agent": UA, "Accept-Encoding": "gzip"}
+    h.update(headers or {})
+    delay = 2.0
+    for attempt in range(RETRIES):
+        try:
+            with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r:
+                raw = r.read()
+                if r.headers.get("Content-Encoding") == "gzip":
+                    raw = gzip.decompress(raw)
+                return raw
+        except urllib.error.HTTPError as e:
+            if e.code in (429, 503) and attempt < RETRIES - 1:
+                time.sleep(delay)
+                delay *= 2
+                continue
+            raise
+    raise RuntimeError("unreachable")
+
+
+_HUB_JWT: list = []
+
+
+def _hub_auth() -> dict:
+    """Authenticated Docker Hub calls get a far higher rate limit than anonymous ones.
+
+    Credentials come from /srv/cc-ci/.testenv (DOCKERHUB_USERNAME / DOCKERHUB_TOKEN), the same pair
+    the CI host already uses. Absent creds are fine — the sweep just runs anonymous and slower."""
+    if _HUB_JWT:
+        return _HUB_JWT[0]
+    env = {}
+    try:
+        for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
+            if "=" in ln and not ln.strip().startswith("#"):
+                k, v = ln.strip().split("=", 1)
+                env[k] = v.strip().strip("\"'")
+    except OSError:
+        pass
+    u = os.environ.get("DOCKERHUB_USERNAME") or env.get("DOCKERHUB_USERNAME")
+    t = os.environ.get("DOCKERHUB_TOKEN") or env.get("DOCKERHUB_TOKEN")
+    hdrs = {}
+    if u and t:
+        try:
+            body = json.dumps({"username": u, "password": t}).encode()
+            req = urllib.request.Request("https://hub.docker.com/v2/users/login",
+                                         data=body, method="POST",
+                                         headers={"Content-Type": "application/json", "User-Agent": UA})
+            with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
+                tokj = json.load(r).get("token")
+            if tokj:
+                hdrs = {"Authorization": f"JWT {tokj}"}
+        except Exception:  # noqa: BLE001 — anonymous is a valid fallback
+            hdrs = {}
+    _HUB_JWT.append(hdrs)
+    return hdrs
+
+
+def _json(url: str, headers: dict | None = None):
+    return json.loads(_fetch(url, headers))
+
+
+def shape(tag: str) -> str:
+    """Signature of a tag with every digit run replaced by '#'. See module docstring."""
+    return re.sub(r"\d+", "#", tag)
+
+
+def vkey(tag: str) -> tuple:
+    """Ordering key: every number in the tag, in order. '1.27.10' > '1.27.9'; text ignored."""
+    return tuple(int(x) for x in re.findall(r"\d+", tag))
+
+
+def parse_ref(ref: str) -> dict:
+    """Split an image reference into registry / repo / tag / digest."""
+    digest = None
+    if "@" in ref:
+        ref, _, digest = ref.partition("@")
+    host, repo, tag = "docker.io", ref, "latest"
+    # A leading component is a REGISTRY only when there is a path after it. Without the slash test,
+    # a bare `postgres:15.18` looks like host "postgres:15.18" because of the tag's colon — which
+    # silently sent every library image to a nonexistent registry.
+    if "/" in ref:
+        first = ref.split("/")[0]
+        if "." in first or ":" in first or first == "localhost":
+            host, _, repo = ref.partition("/")
+    if ":" in repo.split("/")[-1]:
+        repo, _, tag = repo.rpartition(":")
+    if host == "docker.io" and "/" not in repo:
+        repo = f"library/{repo}"          # bare `redis` is really `library/redis`
+    return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
+
+
+HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10"))
+
+
+def _hub_tag_exists(repo: str, tag: str) -> bool:
+    try:
+        _json(f"https://hub.docker.com/v2/repositories/{repo}/tags/{tag}", _hub_auth())
+        return True
+    except Exception:  # noqa: BLE001
+        return False
+
+
+def _hub_tags(repo: str) -> list[str]:
+    """Recently-pushed tags, newest first.
+
+    Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical —
+    but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so
+    ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade.
+    (ghcr offers no ordering, which is why that path needs a different strategy.)
+    """
+    tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
+                     f"?page_size=100&ordering=last_updated")
+    auth = _hub_auth()
+    for _ in range(HUB_RECENT_PAGES):
+        d = _json(url, auth)
+        tags += [r["name"] for r in d.get("results", [])]
+        url = d.get("next")
+        if not url:
+            break
+    return tags
+
+
+def _oci_bearer(host: str, repo: str) -> dict:
+    """Token for an OCI registry, discovered from its own auth challenge.
+
+    Registries do NOT share a token endpoint. ghcr answers at /token?scope=…&service=ghcr.io, but
+    lscr.io and dock.mau.dev advertise different realms, and assuming ghcr's shape made both 401 —
+    which then read as "could not resolve" rather than "asked the wrong URL". The registry tells us
+    where to go in its WWW-Authenticate header; use that."""
+    try:
+        urllib.request.urlopen(
+            urllib.request.Request(f"https://{host}/v2/{repo}/tags/list?n=1",
+                                   headers={"User-Agent": UA}), timeout=TIMEOUT)
+        return {}                                     # no auth needed
+    except urllib.error.HTTPError as e:
+        if e.code != 401:
+            return {}
+        chal = e.headers.get("WWW-Authenticate", "") or ""
+    except Exception:  # noqa: BLE001
+        return {}
+    if not chal.lower().startswith("bearer"):
+        return {}
+    parts = dict(re.findall(r'(\w+)="([^"]*)"', chal))
+    realm = parts.get("realm")
+    if not realm:
+        return {}
+    q = {"service": parts.get("service", host), "scope": parts.get("scope", f"repository:{repo}:pull")}
+    url = realm + ("&" if "?" in realm else "?") + urllib.parse.urlencode(q)
+    try:
+        tok = (_json(url) or {}).get("token") or (_json(url) or {}).get("access_token")
+        return {"Authorization": f"Bearer {tok}"} if tok else {}
+    except Exception:  # noqa: BLE001
+        return {}
+
+
+def _oci_tags(host: str, repo: str) -> list[str]:
+    """Tags from any OCI/v2 registry, with challenge-derived auth and Link pagination.
+
+    ghcr paginates hard — immich-server has >40,000 tags — and a truncated listing silently hides
+    the newest release line, so follow the cursor and let the caller's integrity check catch a read
+    that never reached the current pin."""
+    hdrs = _oci_bearer(host, repo)
+    tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
+    for _ in range(MAX_TAG_PAGES):
+        req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
+        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
+            tags += (json.load(r) or {}).get("tags") or []
+            link = r.headers.get("Link", "") or ""
+        m = re.search(r'<([^>]+)>;\s*rel="next"', link)
+        if not m:
+            break
+        nxt = m.group(1)
+        url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
+    return tags
+
+
+def _gh_token() -> str | None:
+    tok = os.environ.get("GITHUB_TOKEN")
+    if tok:
+        return tok.strip()
+    try:
+        return open(os.environ.get("GITHUB_TOKEN_FILE", "/srv/cc-ci/.github-token")).read().strip() or None
+    except OSError:
+        return None
+
+
+def github_release_tags(owner: str, repo: str, max_pages: int = 4) -> list[str]:
+    """Release tag names for a GitHub repo, newest first.
+
+    FALLBACK for registries whose tag listing cannot be enumerated. ghcr has no ordering and no
+    server-side filter, and immich-machine-learning carries >40,000 tags — a full read is impractical
+    and a partial read silently hides the newest release line. The project's RELEASES are ordered,
+    small, and authoritative: container tags track them. (The GitHub Packages API would answer this
+    directly but needs a scoped token; this scan's token deliberately has none.)
+    """
+    hdrs = {"Accept": "application/vnd.github+json"}
+    tok = _gh_token()
+    if tok:
+        hdrs["Authorization"] = f"Bearer {tok}"
+    out = []
+    for page in range(1, max_pages + 1):
+        try:
+            rows = _json(f"https://api.github.com/repos/{owner}/{repo}/releases"
+                         f"?per_page=100&page={page}", hdrs)
+        except Exception:  # noqa: BLE001
+            break
+        if not rows:
+            break
+        out += [r.get("tag_name") or "" for r in rows]
+    return [t for t in out if t]
+
+
+def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
+    """Candidate GitHub repos whose releases track this image's tags."""
+    if "ghcr.io" not in registry:
+        return []
+    parts = repo.split("/")
+    if len(parts) < 2:
+        return []
+    owner, name = parts[0], parts[-1]
+    cands = [(owner, name)]
+    # ghcr.io/immich-app/immich-machine-learning is built from immich-app/immich.
+    if name.startswith(owner.split("-")[0]):
+        cands.append((owner, owner.split("-")[0]))
+    return cands
+
+
+_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
+
+
+def list_tags(registry: str, repo: str) -> tuple[list[str], str | None]:
+    if (registry, repo) in _TAG_CACHE:
+        return _TAG_CACHE[(registry, repo)]
+    res = _list_tags_uncached(registry, repo)
+    _TAG_CACHE[(registry, repo)] = res
+    return res
+
+
+def _list_tags_uncached(registry: str, repo: str) -> tuple[list[str], str | None]:
+    try:
+        return (_hub_tags(repo) if registry in ("docker.io", "registry-1.docker.io")
+                else _oci_tags(registry, repo)), None
+    except urllib.error.HTTPError as e:
+        return [], f"HTTP {e.code}"
+    except Exception as e:  # noqa: BLE001
+        return [], f"{type(e).__name__}: {e}"
+
+
+def resolve(ref: str) -> dict:
+    """Current pin -> newest same-shape tag, and newest within the current major."""
+    if "${" in ref or "$(" in ref:
+        # The tag is a compose variable (ghost pins `ghost:${IMAGE_VERSION}-alpine`). Its real value
+        # lives in .env, not here. Report it as skipped, never as a failed lookup.
+        return {**parse_ref(ref), "ref": ref, "shape": None, "candidates": 0,
+                "newest_same_shape": None, "newest_within_major": None,
+                "upgrade_available": False,
+                "status": "skipped: templated ref (tag comes from a compose variable)"}
+    info = parse_ref(ref)
+    out = {**info, "ref": ref, "shape": shape(info["tag"]), "status": "ok",
+           "newest_same_shape": None, "newest_within_major": None, "candidates": 0,
+           "upgrade_available": False}
+    tags, err = list_tags(info["registry"], info["repo"])
+    if err:
+        out["status"] = f"error: {err}"
+        return out
+    out["tags_seen"] = len(set(tags))
+    # INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
+    # the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
+    # of thousands of tags and a truncated read silently hides whole release lines. immich's
+    # machine-learning image is pinned v3.1.0, which EXISTS, yet a short read reported v1.134.0 as
+    # newest; without this check that becomes a confident, wrong answer.
+    if info["tag"] not in set(tags):
+        # Docker Hub: the window is recency-ordered, so the pin being outside it just means the pin
+        # is old — which is fine, because anything NEWER is necessarily inside the window. Confirm
+        # the pin genuinely exists (so a typo is still caught) and carry on.
+        if info["registry"] in ("docker.io", "registry-1.docker.io") and _hub_tag_exists(info["repo"], info["tag"]):
+            out["source"] = f"docker-hub:recent-{HUB_RECENT_PAGES * 100}"
+            tags = list(tags) + [info["tag"]]
+        else:
+            for owner, name in _release_fallback_repos(info["registry"], info["repo"]):
+                rel = github_release_tags(owner, name)
+                if info["tag"] in rel:
+                    tags = rel
+                    out["source"] = f"github-releases:{owner}/{name}"
+                    out["tags_seen"] = len(set(rel))
+                    break
+            else:
+                out["status"] = ("error: tag listing incomplete — the current pin "
+                                 f"{info['tag']!r} is absent from {len(set(tags))} registry tags "
+                                 f"and from the project's GitHub releases")
+                return out
+    want, cur = out["shape"], vkey(info["tag"])
+    same = [t for t in set(tags) if shape(t) == want and vkey(t)]
+    out["candidates"] = len(same)
+    if not same:
+        # Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings.
+        out["status"] = "no comparable tags (shape has no numeric siblings)"
+        return out
+    newest = max(same, key=vkey)
+    out["newest_same_shape"] = newest
+    if cur:
+        within = [t for t in same if vkey(t)[:1] == cur[:1]]
+        if within:
+            out["newest_within_major"] = max(within, key=vkey)
+    out["upgrade_available"] = bool(cur and vkey(newest) > cur)
+    return out
+
+
+def compose_images_ssh(recipe: str, host: str, recipe_dir: str) -> list[str]:
+    """Same as compose_images, but the recipe tree lives on another host (cc-ci has no python3)."""
+    # NB: no shell-quoting of the directory — it may legitimately start with ~ or $HOME, and
+    # quoting it stops the remote shell expanding it, which yields an empty (and silent) result.
+    d = f"{recipe_dir}/{shlex.quote(recipe)}".replace("~", "$HOME")
+    cmd = (f'for f in {d}/compose*.yml; do case "$f" in *compose.ccci.yml) continue;; esac; '
+           f'[ -f "$f" ] && {{ cat "$f"; echo; }}; done; exit 0')
+    out = subprocess.run(["ssh", host, cmd], capture_output=True, text=True, timeout=120)
+    if out.returncode != 0:
+        raise RuntimeError(f"ssh {host}: {(out.stderr.strip() or 'no output')[:200]}")
+    if not out.stdout.strip():
+        raise RuntimeError(f"ssh {host}: no compose files found under {d}")
+    refs = []
+    for m in IMAGE_RE.finditer(out.stdout):
+        if m.group(1) not in refs:
+            refs.append(m.group(1))
+    return refs
+
+
+def compose_images(recipe: str, recipe_dir: str) -> list[str]:
+    """Every `image:` ref in the recipe's own compose files (the cc-ci overlay is NOT the recipe)."""
+    refs, base = [], os.path.join(recipe_dir, recipe)
+    for path in sorted(glob.glob(os.path.join(base, "compose*.yml"))):
+        if os.path.basename(path) == "compose.ccci.yml":
+            continue
+        try:
+            for m in IMAGE_RE.finditer(open(path).read()):
+                if m.group(1) not in refs:
+                    refs.append(m.group(1))
+        except OSError:
+            continue
+    return refs
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("recipe")
+    ap.add_argument("--ssh", default=None, metavar="HOST",
+                    help="read the recipe's compose files from HOST over ssh (e.g. --ssh cc-ci); "
+                         "resolving still happens locally")
+    ap.add_argument("--recipe-dir", default=RECIPE_DIR)
+    ap.add_argument("--table", action="store_true", help="human-readable table instead of JSON")
+    ap.add_argument("--only", default=None, help="resolve just the images whose ref contains this")
+    a = ap.parse_args()
+
+    rdir = a.recipe_dir if a.recipe_dir != RECIPE_DIR or not a.ssh else "~/.abra/recipes"
+    refs = (compose_images_ssh(a.recipe, a.ssh, rdir) if a.ssh
+            else compose_images(a.recipe, a.recipe_dir))
+    if a.only:
+        refs = [r for r in refs if a.only in r]
+    results = [resolve(r) for r in refs]
+    report = {
+        "recipe": a.recipe,
+        "images": results,
+        "upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
+        "unresolved": [r["ref"] for r in results if r["status"].startswith("error")],
+        # The whole point: distinguish "checked, current" from "could not check".
+        "all_resolved": not any(r["status"].startswith("error") for r in results),
+    }
+    if not a.table:
+        print(json.dumps(report, indent=2))
+        return 0
+    print(f"{a.recipe} — {len(results)} images")
+    for r in results:
+        flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current")
+        print(f"  [{flag:7}] {r['repo']}:{r['tag']}" + ("  (digest-pinned)" if r["digest"] else ""))
+        print(f"            shape={r['shape']}  candidates={r['candidates']}"
+              f"  newest_same_shape={r['newest_same_shape']}"
+              f"  newest_within_major={r['newest_within_major']}")
+        if r["status"] != "ok":
+            print(f"            status: {r['status']}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -455,13 +455,38 @@ class TestReleaseNoteResolution(unittest.TestCase):
        self.assertEqual(rep["indeterminate"], [])
        self.assertEqual(rep["resolved_by_release_notes"]["CVE-TBD"], ["7.4.5", "8.0.3"])

-    def test_release_naming_it_only_outside_the_window_stays_indeterminate(self):
+    def test_naming_releases_all_below_the_window_means_ALREADY_fixed(self):
+        # Every known fix predates the version we were already on, so this upgrade did not deliver
+        # it. That is a DECISION, not an unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 such
+        # advisories, and calling them "could not judge" overstates the uncertainty.
        rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
                       v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
                       releases={"CVE-TBD": ["6.2.19"]})
        self.assertEqual(rep["fixed_by_this_upgrade"], [])
+        self.assertEqual(rep["indeterminate"], [])
+        self.assertIn("CVE-TBD", rep["already_fixed_before_upgrade"])
+        self.assertIn("outside-window", rep["cves"]["CVE-TBD"]["classification"])
+
+    def test_naming_releases_only_ABOVE_the_window_stays_indeterminate(self):
+        # The fix landed after our target, so we are still exposed. Deliberately NOT decided as a
+        # tidy "not fixed": it is an open vulnerability and must stay visible to the operator.
+        rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
+                       v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
+                       releases={"CVE-TBD": ["9.0.0"]})
+        self.assertEqual(rep["fixed_by_this_upgrade"], [])
        self.assertIn("CVE-TBD", rep["indeterminate"])

+    def test_vendor_page_cve_on_the_same_repo_uses_release_notes(self):
+        # mailu announces its Roundcube CVEs only on github.com/Mailu/Mailu/releases. Requiring an
+        # advisory feed sent a deterministic case to pass 2; it is now decided in pass 1.
+        rep = run_scan([gh("Mailu/Mailu", [])],
+                       [vendor("https://github.com/Mailu/Mailu/releases", ["CVE-2026-54432"])],
+                       v_from="2024.06.55", v_to="2024.06.57",
+                       urls=["https://github.com/Mailu/Mailu"],
+                       releases={"CVE-2026-54432": ["2024.06.56"]})
+        self.assertIn("CVE-2026-54432", rep["fixed_by_this_upgrade"])
+        self.assertEqual(rep["cve_count_fixed"], 1)
+
    def test_release_evidence_is_recorded_for_audit(self):
        rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
                       v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
@@ -478,6 +503,40 @@ class TestReleaseNoteResolution(unittest.TestCase):
        self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {})


+class TestReleaseLineSemantics(unittest.TestCase):
+    """A fix inside the numeric window is not a fix on the branch you actually land on."""
+
+    def test_fix_later_on_the_targets_own_line_is_not_counted(self):
+        # ClickHouse fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20. Landing on 23.10.4.25 crosses
+        # the 23.9 fix numerically but is BELOW its own line's fix, so it does not have it.
+        rep = run_scan([gh("ClickHouse/ClickHouse",
+                           [adv("CVE-2023-48704", patched="v23.10.5.20; v23.9.6.20; v23.8.8.20")])],
+                       v_from="23.4.2.11", v_to="23.10.4.25",
+                       urls=["https://github.com/ClickHouse/ClickHouse"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], [])
+
+    def test_fix_earlier_on_the_targets_own_line_is_counted(self):
+        rep = run_scan([gh("ClickHouse/ClickHouse",
+                           [adv("CVE-2023-47118", patched="v23.10.2.13; v23.8.6.16")])],
+                       v_from="23.4.2.11", v_to="23.10.4.25",
+                       urls=["https://github.com/ClickHouse/ClickHouse"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-47118"])
+
+    def test_fix_exactly_at_the_target_is_counted(self):
+        rep = run_scan([gh("ClickHouse/ClickHouse",
+                           [adv("CVE-2023-48298", patched="v23.10.4.25; v23.9.5.29")])],
+                       v_from="23.4.2.11", v_to="23.10.4.25",
+                       urls=["https://github.com/ClickHouse/ClickHouse"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-48298"])
+
+    def test_no_fix_on_the_target_line_falls_back_to_the_window(self):
+        # redis fixes 7.4.6/8.0.4/8.2.2 with no 8.10.x entry; landing on 8.10 still has them,
+        # because nothing on the 8.10 line is named as a LATER fix.
+        rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2")])],
+                       v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
+        self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"])
+
+
 class TestAdjudicationEvidenceAssembly(unittest.TestCase):
    """Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""

@@ -599,16 +658,22 @@ class TestHistoricReportNumbers(unittest.TestCase):
        for cve in delta:
            self.assertIn("redis", both["cves"][cve]["sources"][0])

-    def test_mailu_scan_finds_zero_and_says_so_knowingly(self):
-        # Published as 2 via the UNION with release-note reading; the scan's own contribution is 0,
-        # and 0 here must mean "checked, none", not "could not check".
+    def test_mailu_finds_the_roundcube_pair_without_an_agent(self):
+        # Published as 2 on 2026-08-07, but only because an agent read the release notes; the scan
+        # itself contributed 0. It now reaches 2 deterministically: the CVEs appear only on
+        # github.com/Mailu/Mailu/releases, and release 2024.06.56 (inside the window) names them.
        rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")])
-        self.assertEqual(rep["cve_count_fixed"], 0)
-        self.assertTrue(rep["count_known"])
+        self.assertEqual(rep["cve_count_fixed"], 2)
+        self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-54432", "CVE-2026-54433"})
+        # The redis bump fixes nothing new — every advisory it crosses was fixed at or before 8.6.3.
+        self.assertEqual(rep["cve_count_indeterminate"], 0)

-    def test_keycloak_26_7_0_to_26_7_1_is_7(self):
+    def test_keycloak_26_7_0_to_26_7_1_is_12(self):
+        # Was 7 while only the GHSA feed was consulted. keycloak lists five more CVEs in the 26.7.1
+        # release notes' fixed-issues section that it never filed as advisories — the gitea pattern.
        rep = self._count("keycloak", "26.7.0", "26.7.1")
-        self.assertEqual(rep["cve_count_fixed"], 7)
+        self.assertEqual(rep["cve_count_fixed"], 12)
+        self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 5)


 def _main():
@@ -0,0 +1,21 @@
+# Upstream sources — mumble
+
+| service | image | source repo | releases / changelog |
+|---------|-------|-------------|----------------------|
+| app | mumblevoip/mumble-server | https://github.com/mumble-voip/mumble | https://github.com/mumble-voip/mumble/releases |
+| web | rankenstein/mumble-web | https://github.com/rankenstein/mumble-web | https://github.com/rankenstein/mumble-web/releases |
+
+## Standing notes
+- This file was **missing entirely** until 2026-08-11. Without it the advisory scan had no source to
+  query, and still printed "0 identified by the deterministic scan" — which was then published as a
+  clean `0` in the 2026-08-11 CVE check. The scan now refuses to emit a count when it has no usable
+  source (it reports UNKNOWN), and `audit-sources.py` flags a missing registry file directly.
+- `mumblevoip/mumble-server` tracks the upstream server releases and DOES publish GitHub security
+  advisories, so it is the recipe's primary CVE source.
+- `rankenstein/mumble-web` is a **fork** of the original `Johni0702/mumble-web`, which has been
+  dormant since 2023-05. The fork itself last pushed 2023-07 and its Docker tag `0.5` was last built
+  well over five years ago. Neither is archived, but treat the web client as effectively unmaintained:
+  if a CVE lands there, expect no upstream fix and plan a replacement rather than an upgrade.
+- The server image tag is `v<version>-<build>` (e.g. `v1.6.870-4`); the trailing number is the image
+  build, not an app version, and moves independently of upstream releases — `abra recipe upgrade`
+  reports "no new versions" for it, so use `resolve-images.py` to see those bumps.
Author	SHA1	Message	Date
autonomic-bot	a0d1b82869	Merge pull request 'CI server health: image pruning, disk thresholds, reconcile-upstream' (#5 ) from review/4-ci-health into review/3-upstream-resolution	2026-08-11 19:03:54 +00:00
autonomic-bot	94ce5c4de2	cc-ci-status: correct the ENOSPC note — inode recreation was coincidence I recorded that recreating the runs directory with a fresh inode preceded recovery. It recurred afterwards (build 1252), so that was not the fix. The real signal is that it is INTERMITTENT and tracks concurrent activity: every failure landed while a second run or manual deploy was in flight, and every build on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a failing build. Practical guidance is therefore to wait for the host to go quiet and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is the obvious knob to test if it becomes disruptive. Root cause still not established, and the note now says so plainly rather than presenting a coincidence as understood.	2026-08-11 18:54:58 +00:00
autonomic-bot	bb7ebb4a27	reconcile-upstream.sh: one deterministic entry point, mandated before PR work Working against a stale mirror has cost us three different ways: - mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while upstream had already merged AND released it (3.1.3+2024.06.57). The work was done; only our mirror was behind. Reconciling closed the PR automatically. - a stale mirror makes a survey report 'no upgrades available', so the recipe silently drops out of the weekly run. - reading the wrong branch: several coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's main is 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs, so reading main manufactures a false 'three releases behind, missing two CVSS-9.8 RCEs' finding. The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and already resolves the default branch itself. What was missing was a single obvious entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or --all, and is idempotent — recipe work lives in branches, never on mirror main, so force-syncing main discards nothing. /ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now point at the shared script.	2026-08-11 18:38:09 +00:00
autonomic-bot	ecf126d98d	cc-ci-status: record the ENOSPC-with-free-disk failure and what recovered it Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes. Ruled out: actual disk (df sampled every 2s across a failing build never moved), inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded), runner sandboxing (namespaces identical to the host), and a wedged runner (restart changed nothing). The same harness with the same numeric run id, run by hand outside drone, worked every time. Recreating the runs directory with a fresh inode preceded recovery; builds have run normally since. The root cause is NOT established, so the note says so rather than presenting a fix that might be coincidence.	2026-08-11 17:39:56 +00:00
autonomic-bot	ab88e59c21	cc-ci: prune unused images in the sweep; catch a starving host before CI dies The CI server filled up and every !testme from build 1236 to 1242 died at harness startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got far enough to write results.json, the PR badges just said 'failure' — so it read as recipe regressions, and plausible's genuinely-fixed suite looked still-broken. Cause: every run pulls each recipe's images and nothing ever removed the old ones. 72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%. Two changes so it does not recur: - sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the layer cache so runs stay fast. 'docker image prune -a' spares anything a container references, so infra and warm-* canonicals are safe. Volumes are still NOT pruned — warm-* canonical volumes are data-warm and legitimately dangling. - /cc-ci-status flags server disk at >65% rather than >80%, because this is not a steady-state measure: the host was at 73% when runs started failing. It also now checks that recent builds actually produced results.json — an empty run dir is the fingerprint of a host problem masquerading as a recipe failure — and records how to read a drone step log out of its sqlite when the API token is unreachable.	2026-08-11 15:41:40 +00:00
autonomic-bot	e89da2d842	audit-sources: check we are still looking where releases actually happen A recipe tracks an image repo and a set of registry URLs. When upstream moves, nothing errors — the old repo just stops receiving tags and the recipe looks 'up to date' forever. plausible is the case: it tracked plausible/analytics on Docker Hub while upstream moved to ghcr.io/plausible/community-edition. Every survey said 'no upgrades available' while v3 shipped elsewhere. audit-sources.py reports the signals that catch it, per image and per registry URL: image gone quiet (newest tag older than --quiet-days), deprecation wording in the registry description, and GitHub repos that are archived, renamed or gone. Signals, not verdicts — a stable image can be quiet for good reason — so each finding says what was measured. First run over 22 recipes, 11 findings, 4 alerts. It independently re-derived the plausible case (analytics quiet 1126 days), and found: - drone: harness/drone now answers as harness/harness (the image is fine) - lasuite-docs, lasuite-drive: minio/minio is ARCHIVED on GitHub - lasuite-docs: docspecio/api is ARCHIVED - matrix-synapse: halfshot/matrix-appservice-discord image quiet 2078 days - mumble: NO cc-ci-plan/upstream/mumble.md at all That last one exposed a scanner bug. With no registry file there is no source to query, yet the scan still printed '0 identified by the deterministic scan' — and that 0 was published as a clean count in the 2026-08-11 CVE check. A scan with no usable source has measured nothing and must not report a number, least of all 0. It now returns UNKNOWN and says the registry file is missing. upstream/mumble.md added; mumble now scans 6 sources for a genuine 0.	2026-08-11 15:02:06 +00:00
autonomic-bot	6c91373357	skills: point every test-editing path at tests/STYLE.md /recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author test changes, and all three had only 'never weaken a test' as guidance. That did not stop the plausible failure: the fixture INSERTed rows into the app's database, which was correct for v2 and silently wrong for v3, where a site must belong to a team. Events were acked 202 and discarded; the recipe sat RED for six weeks. The rule that would have prevented it — set state up through the app's own interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and each of the three paths is told to read it before editing a test.	2026-08-11 14:41:10 +00:00
autonomic-bot	1db85a7e77	resolve-images: abra-independent version resolver; fix release-line over-count immich pins two images with BOTH a tag and a digest, which makes abra FATA and abandon the WHOLE recipe. It therefore contributed no version data at all and silently dropped out of every survey — indistinguishable from 'up to date'. The standing answer was prose in three skills telling an agent to check registries by hand. This replaces it with a tool. resolve-images.py reads the compose files and queries registries itself: - Docker Hub, ghcr, and any OCI registry via its own auth challenge (lscr.io and dock.mau.dev advertise different realms; assuming ghcr's shape 401'd). - tag SHAPES (digits -> '#') so -alpine stays on -alpine and 'latest' is never proposed as an upgrade. - reports newest_within_major AND newest_same_shape, and refuses to choose: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors build immich-server expects, so taking the newest breaks the deploy. - integrity check: if the CURRENT pin is absent from the listing, the listing was truncated and any 'newest' is a guess. ghcr caps out past 40k tags, so that falls back to the project's GitHub releases. - per-repo cache + backoff + Docker Hub auth: a fleet sweep re-reads nginx, redis and postgres many times and was getting 429s reported as 'unresolved'. 21/21 recipes now resolve. It found upgrades abra missed entirely in five: mumble (abra said 'no new versions'; four patches behind), plausible's clickhouse, lasuite-drive's collabora, gitea's mariadb, immich's postgres. plausible's carried four CVEs, three high. Also fixes a real over-count found while validating that: a fix inside the numeric window is not a fix on the branch you land on. ClickHouse patched CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20 — landing on 23.10.4.25 crosses the 23.9 fix but sits below its own line's, so it does NOT have it. A fix named on the target's own line and above the target is now proof of absence. 70 tests (64 offline + 6 live). keycloak's live expectation moves 7 -> 12 and mailu's 0 -> 2: both are the release-note source finding real fixes that were never filed as advisories.	2026-08-11 04:56:39 +00:00
autonomic-bot	8df32edfcf	cve-check: reconcile is mandatory; do not assume the default branch is main Skipping the reconcile to keep the sweep 'read-only' was wrong. It researches a stale checkout, and on the first real run left two recipes with no survey output at all — indistinguishable from 'no upgrades available' unless you look. The reconcile is safe precisely because recipe work lives in branches, not on main. Also documents the trap that produced a false finding in that run: several coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's main is at 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs, so reading main reports a recipe three releases behind and missing two CVSS-9.8 RCE fixes. Resolve default_branch from the API before reading any file. And: no output is not 'no upgrade'. It is a third outcome, and only becomes '?' after the direct registry check has also failed.	2026-08-11 04:32:49 +00:00
autonomic-bot	18caf047bf	advisory-scan: two more cases decided in pass 1, found by the first real /cve-check 1. Release-note resolution now covers vendor pages on the same repo. It required a github-advisories: source, so mailu's Roundcube CVEs — announced only on github.com/Mailu/Mailu/releases — went to pass 2 even though the answer was sitting in the release notes. mailu now reports 2 deterministically, matching what previously took an agent reading the notes. 2. 'All known fix versions predate the version we were on' is now a DECISION, not an unknown. mailu's redis 8.8.0 -> 8.10.0 crosses 12 advisories all fixed by 8.6.3 or earlier; reporting them as 'could not judge' overstated the uncertainty. Recorded as outside-window with the naming tags as evidence. A fix landing ABOVE the window still stays indeterminate on purpose: that is an open vulnerability and must stay visible. 60 offline tests (was 58). discourse 140 / gitea 2 unchanged.	2026-08-11 04:24:15 +00:00
autonomic-bot	b0bdce2c15	add /cve-check and /cve-check-and-upgrade /cve-check answers 'what are we exposed to that an upgrade would fix?' without running an upgrade: per-recipe, resolve the available window for EVERY image (sidecars included), run the advisory scan over it, adjudicate whatever pass 1 could not decide, publish a report. Read-only — no PRs, no CI, no merges. /cve-check-and-upgrade does that sweep, then runs /recipe-upgrade only on the recipes whose upgrade actually closes a CVE, worst severity first, and reports on both. --min-severity high for just the urgent ones; --dry-run prints the queue and stops. Never merges. Deliberate choices, each written into the skills: - externals are SWEPT but never upgraded here — a security sweep that skipped deployed software would misreport exposure, but we don't maintain them. - an unknown count never justifies an upgrade AND is never treated as clean; it goes to the Addendum. - no upgrade available means 0 CVEs, not '?'. - subagents are told which CVEs justify their upgrade, so the PR says why it exists — a PR naming the RCE it closes gets reviewed sooner. recipe-report.py grows a page kind: 'cve' files as cve-DATE.html so a sweep can't overwrite a weekly edition, while BOTH appear in the same archive index, suffixed 'full report' / 'CVE check'. /help and /cc-ci-status updated to route to them.	2026-08-11 04:04:59 +00:00