diff --git a/.claude/skills/ci-test-review/SKILL.md b/.claude/skills/ci-test-review/SKILL.md index 0e7c54a..de06743 100644 --- a/.claude/skills/ci-test-review/SKILL.md +++ b/.claude/skills/ci-test-review/SKILL.md @@ -79,6 +79,10 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m it handles the mirror to `git.autonomic.zone/recipe-maintainers/` (upstream `git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the recipe. +- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test + change must satisfy — set state up through the app's interface rather than its database, gate on + version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from + the app's own telemetry before concluding a test is stale. - **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo (`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the `GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated diff --git a/.claude/skills/cve-check/SKILL.md b/.claude/skills/cve-check/SKILL.md index 8f2d683..1a64867 100644 --- a/.claude/skills/cve-check/SKILL.md +++ b/.claude/skills/cve-check/SKILL.md @@ -46,10 +46,16 @@ This is `/recipe-upgrade` step 1's research, stopping before it implements anyth > `git -C ~/.abra/recipes/ remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/.git"` > 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop` > after. Only a genuinely dirty TRACKED tree is a skip. -> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). This is -> **not** "not fetchable": enumerate the compose's `image:` refs yourself and check the upstream -> registry directly for the ones abra could not read, picking the newest tag the app version -> supports rather than the numerically highest. +> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not +> hand-check the registry; run the resolver, which is abra-independent and covers every image: +> ``` +> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py --ssh cc-ci --table +> ``` +> It reports, per image, `newest_within_major` (the compatibility-safe pick) and +> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have +> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the +> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break +> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0. **Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile `/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you @@ -66,6 +72,12 @@ nothing; it also auto-closes mirror PRs whose changes upstream has already merge > API (`/api/v1/repos/coop-cloud/` → `default_branch`) before reading any file, and never > `git reset --hard origin/main` on a checkout that tracks `master`. +**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes +nothing for images it cannot parse, and it reported "no new versions" for images that did have them +(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on +the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which +(plausible's clickhouse) carried four CVEs. + Then read versions: ``` set -a; . /srv/cc-ci/.testenv; set +a diff --git a/.claude/skills/recipe-upgrade/SKILL.md b/.claude/skills/recipe-upgrade/SKILL.md index 9d3f0cf..785b678 100644 --- a/.claude/skills/recipe-upgrade/SKILL.md +++ b/.claude/skills/recipe-upgrade/SKILL.md @@ -312,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**: Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a stale test; operator to decide). - **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way: + 0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and + it is written against the failures this pipeline has actually produced. The two that matter most + here: **set state up through the app's own interface, never its database** (a plausible fixture + that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks), + and **gate on version rather than writing a fixture that supports both** — old-version tests can + simply be deleted, since the older version is only exercised through the upgrade tier. 1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`, never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay. 2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the diff --git a/.opencode/skills/cc-ci-tests-update/SKILL.md b/.opencode/skills/cc-ci-tests-update/SKILL.md index cd0a211..d94bbcb 100644 --- a/.opencode/skills/cc-ci-tests-update/SKILL.md +++ b/.opencode/skills/cc-ci-tests-update/SKILL.md @@ -85,6 +85,14 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis): ### 2. For each stale test — author the minimal test update (AI; never weaken) +> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test +> changes, written from failures this pipeline actually produced. Most load-bearing: set state up +> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows +> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the +> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be +> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the +> wait but **never the assertion**. + Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared Swarm). For each `STALE_TESTS` entry: diff --git a/cc-ci-plan/advisory-scan.py b/cc-ci-plan/advisory-scan.py index 622f741..1a3b0e6 100755 --- a/cc-ci-plan/advisory-scan.py +++ b/cc-ci-plan/advisory-scan.py @@ -164,6 +164,21 @@ def _vkey(v: str | None) -> tuple: return tuple(out) +def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool: + """Does a patched version on the TARGET's own release line sit ABOVE the target? + + Projects maintain several branches at once and backport per branch, so "some patched version is + inside the numeric window" is not the same as "the version we land on has the fix". ClickHouse + fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20; an upgrade landing on 23.10.4.25 crosses the + 23.9 fix numerically but is still BELOW its own line's fix, so it does NOT have it. When the + advisory names a fix on the target's own line and the target is older than it, that is proof of + absence and outranks any other candidate.""" + if len(kt) < 2: + return False + line = kt[:2] + return any(c[:2] == line and c > kt for c in cands if len(c) >= 2) + + def _within(kf: tuple, kt: tuple, c: tuple) -> bool: """Is patched-version `c` inside the window (kf, kt] — exclusive lower, inclusive upper? @@ -665,6 +680,10 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str, continue patched = e.get("patched") or "" cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)] + if kf and kt and _superseded_on_target_line(kt, cands): + # The target's own line got the fix LATER than the target: not fixed here. + e.setdefault("classification", "outside-window") + continue if kf and kt and any(_within(kf, kt, c) for c in cands): got.add(cve) elif not patched or PLACEHOLDER_RE.search(patched): @@ -778,8 +797,19 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str, report["unclassified"] = sorted(unknown) # NEVER report 0 for something we could not determine — a 0 asserts safety. If ANY requested # window could not be ordered at all, the total is UNKNOWN rather than a partial number. - report["count_known"] = not unresolved_any - report["cve_count_fixed"] = len(fixed_set) if not unresolved_any else None + # A scan with NO usable source has not measured anything, so it must not report a number — + # least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and + # still produced "0 identified", which was then published as a clean 0 in a CVE report. + usable_sources = [ + s for s in report["sources"] + if s["status"] == "ok" or s["status"].startswith("no-advisories-published") + ] + no_sources = not usable_sources + if no_sources: + report["no_usable_sources"] = True + report["count_known"] = not unresolved_any and not no_sources + report["cve_count_fixed"] = (len(fixed_set) + if (not unresolved_any and not no_sources) else None) report["cve_count_total_seen"] = len(report["cves"]) # Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has # no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict. @@ -802,10 +832,16 @@ def markdown(rep: dict) -> str: f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"] if not rep.get("count_known", True): L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**") - L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes numeric " - "ordering meaningless across this jump, so no advisory could be classified. Render " - "this recipe's cve cell as `?`, never `0`. Read the vendor's release notes for the " - "jump and count by hand.") + if rep.get("no_usable_sources"): + L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry " + "file `cc-ci-plan/upstream/.md` is missing or every source failed, so " + "nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add " + "the registry file.") + else: + L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes " + "numeric ordering meaningless across this jump, so no advisory could be " + "classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's " + "release notes for the jump and count by hand.") if rep["unclassified"]: L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes " f"other images in this recipe: " + ", ".join(rep["unclassified"][:12])) diff --git a/cc-ci-plan/audit-sources.py b/cc-ci-plan/audit-sources.py new file mode 100755 index 0000000..dda4da6 --- /dev/null +++ b/cc-ci-plan/audit-sources.py @@ -0,0 +1,238 @@ +#!/usr/bin/env python3 +"""audit-sources — are we still looking in the right place for each recipe's updates? + +A recipe tracks an image repo and a set of registry URLs. Upstreams move: they rename the image, +switch registry, archive the GitHub repo, or split a community edition out of the original. When that +happens nothing errors — the old repo simply stops receiving tags, and the recipe looks "up to date" +forever while real releases happen somewhere else. + +plausible is the worked example. It tracked `plausible/analytics` on Docker Hub; upstream moved to +`ghcr.io/plausible/community-edition`. The old repo still exists and still serves v2.0.0, so every +survey said "no upgrades available" while v3 shipped elsewhere. + +This reports the signals that catch that, per image and per registry URL: + + * IMAGE GONE QUIET — newest tag is older than --quiet-days (default 365). The single strongest + signal that releases moved somewhere else. + * DEPRECATION WORDING — the registry description says deprecated / moved / no longer maintained. + * GITHUB REPO ARCHIVED — upstream archived it. + * GITHUB REPO RENAMED — the API redirects to a different owner/name than we ask for. + * GITHUB REPO GONE — 404. + +Everything is a SIGNAL, not a verdict: a genuinely stable image (mumble, custom-html) can be quiet +for good reason. The output is for a human to judge, so each finding says what was measured. + + audit-sources.py [recipe ...] [--ssh HOST] [--quiet-days N] [--json] +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import os +import re +import sys +import urllib.error +import urllib.request +from datetime import datetime, timezone + +HERE = os.path.dirname(os.path.abspath(__file__)) +_spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HERE, "resolve-images.py")) +RI = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(RI) + +REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream")) +USED_RECIPES = os.path.join(HERE, "used-recipes.md") +DEPRECATION_RE = re.compile( + r"\b(deprecat|no longer maintain|unmaintained|superseded|moved to|migrated to|" + r"has moved|discontinued|end.of.life|archived)\b", re.I) + + +def _days_since(iso: str | None) -> int | None: + if not iso: + return None + try: + d = datetime.fromisoformat(iso.replace("Z", "+00:00")) + except ValueError: + return None + return (datetime.now(timezone.utc) - d).days + + +def hub_repo_meta(repo: str) -> dict: + """Docker Hub repo metadata: when it was last pushed to, and how it describes itself.""" + try: + d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}", RI._hub_auth()) + except urllib.error.HTTPError as e: + return {"status": f"HTTP {e.code}"} + except Exception as e: # noqa: BLE001 + return {"status": f"{type(e).__name__}"} + text = f"{d.get('description') or ''}\n{d.get('full_description') or ''}" + m = DEPRECATION_RE.search(text) + return {"status": "ok", "last_updated": d.get("last_updated"), + "deprecation_hint": (m.group(0) if m else None), + "archived": bool(d.get("is_archived") or d.get("status") == "inactive")} + + +def github_repo_meta(owner: str, repo: str) -> dict: + """GitHub repo state — archived, renamed (the API answers with the CURRENT full_name), or gone.""" + hdrs = {"Accept": "application/vnd.github+json"} + tok = RI._gh_token() + if tok: + hdrs["Authorization"] = f"Bearer {tok}" + try: + d = RI._json(f"https://api.github.com/repos/{owner}/{repo}", hdrs) + except urllib.error.HTTPError as e: + return {"status": f"HTTP {e.code}"} + except Exception as e: # noqa: BLE001 + return {"status": f"{type(e).__name__}"} + asked, got = f"{owner}/{repo}".lower(), (d.get("full_name") or "").lower() + return {"status": "ok", "archived": bool(d.get("archived")), "pushed_at": d.get("pushed_at"), + "renamed_to": (d.get("full_name") if got and got != asked else None), + "description": d.get("description") or ""} + + +def newest_tag_date(registry: str, repo: str, tag: str) -> str | None: + """When was the repo's newest same-shape tag pushed? Docker Hub only (it dates its tags).""" + if registry not in ("docker.io", "registry-1.docker.io"): + return None + try: + d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}/tags" + f"?page_size=100&ordering=last_updated", RI._hub_auth()) + except Exception: # noqa: BLE001 + return None + want = RI.shape(tag) + for row in d.get("results", []): + if RI.shape(row.get("name") or "") == want: + return row.get("last_updated") + return (d.get("results") or [{}])[0].get("last_updated") + + +def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict: + out = {"recipe": recipe, "findings": [], "images": [], "sources": []} + try: + refs = (RI.compose_images_ssh(recipe, ssh, "~/.abra/recipes") if ssh + else RI.compose_images(recipe, RI.RECIPE_DIR)) + except Exception as e: # noqa: BLE001 + out["findings"].append({"level": "error", "what": f"could not read compose: {e}"}) + return out + + for ref in refs: + if "${" in ref: + continue + info = RI.parse_ref(ref) + row = {"ref": ref, "registry": info["registry"], "repo": info["repo"], "tag": info["tag"]} + if info["registry"] in ("docker.io", "registry-1.docker.io"): + meta = hub_repo_meta(info["repo"]) + row.update(meta) + newest = newest_tag_date(info["registry"], info["repo"], info["tag"]) + row["newest_tag_pushed"] = newest + age = _days_since(newest) + row["newest_tag_age_days"] = age + if age is not None and age > quiet_days: + out["findings"].append({ + "level": "warn", "what": "image has gone quiet", + "detail": f"{info['repo']}: newest {RI.shape(info['tag'])}-shaped tag pushed " + f"{age} days ago — releases may have moved elsewhere"}) + if meta.get("deprecation_hint"): + out["findings"].append({ + "level": "warn", "what": "registry text suggests deprecation", + "detail": f"{info['repo']}: says {meta['deprecation_hint']!r}"}) + if meta.get("archived"): + out["findings"].append({"level": "warn", "what": "registry repo archived/inactive", + "detail": info["repo"]}) + out["images"].append(row) + + urls, reg_path = ([], None) + try: + urls, reg_path = _registry_urls(recipe) + except Exception: # noqa: BLE001 + pass + if reg_path is None: + out["findings"].append({"level": "warn", "what": "no upstream registry file", + "detail": f"cc-ci-plan/upstream/{recipe}.md is missing — the advisory " + f"scan has nowhere to look"}) + seen = set() + for u in urls: + m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u) + if not m: + continue + owner, repo = m.group(1), m.group(2).removesuffix(".git") + if (owner, repo) in seen: + continue + seen.add((owner, repo)) + meta = github_repo_meta(owner, repo) + row = {"repo": f"{owner}/{repo}", **meta} + age = _days_since(meta.get("pushed_at")) + row["pushed_age_days"] = age + out["sources"].append(row) + if meta.get("status") != "ok": + out["findings"].append({"level": "warn", "what": "registry source unreachable", + "detail": f"{owner}/{repo}: {meta['status']}"}) + continue + if meta.get("renamed_to"): + out["findings"].append({"level": "alert", "what": "GitHub repo has MOVED", + "detail": f"{owner}/{repo} now answers as {meta['renamed_to']}"}) + if meta.get("archived"): + out["findings"].append({"level": "alert", "what": "GitHub repo is ARCHIVED", + "detail": f"{owner}/{repo} — upstream development has stopped here"}) + if age is not None and age > quiet_days: + out["findings"].append({"level": "warn", "what": "GitHub repo quiet", + "detail": f"{owner}/{repo}: last push {age} days ago"}) + return out + + +def _registry_urls(recipe: str): + path = os.path.join(REGISTRY_DIR, f"{recipe}.md") + if not os.path.exists(path): + return [], None + text = open(path).read() + urls = [] + for u in re.findall(r"https?://[^\s)|\]]+", text): + u = u.rstrip("`'\"*.,;:>)") + if u and u not in urls: + urls.append(u) + return urls, path + + +def all_recipes() -> list[str]: + out = [] + for ln in open(USED_RECIPES): + ln = ln.strip() + if not ln or ln.startswith("#") or ln.startswith("`"): + continue + parts = ln.split() + if len(parts) >= 2 and parts[1] in ("weekly", "external"): + out.append(parts[0]) + return out + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("recipes", nargs="*") + ap.add_argument("--ssh", default=None) + ap.add_argument("--quiet-days", type=int, default=365) + ap.add_argument("--json", action="store_true") + a = ap.parse_args() + + recipes = a.recipes or all_recipes() + reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes] + if a.json: + print(json.dumps(reports, indent=2)) + return 0 + alerts = 0 + for rep in reports: + fs = rep["findings"] + mark = "OK " if not fs else ("!! " if any(f["level"] == "alert" for f in fs) else " ? ") + print(f"{mark} {rep['recipe']}") + for f in fs: + alerts += f["level"] == "alert" + print(f" [{f['level']}] {f['what']}: {f.get('detail','')}") + print(f"\n{len(reports)} recipes audited · " + f"{sum(len(r['findings']) for r in reports)} findings · {alerts} alerts") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/cc-ci-plan/recipe-report.py b/cc-ci-plan/recipe-report.py index b124996..f73cf78 100755 --- a/cc-ci-plan/recipe-report.py +++ b/cc-ci-plan/recipe-report.py @@ -12,7 +12,7 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/-DATE.html and regen the archive index. KIND is `week` (default, the weekly /recipe-report) or `cve` (a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index, - newest first, each row suffixed "full report" or "CVE check"; the distinct + newest first, each row suffixed "full" or "CVE check"; the distinct filename prefix just stops a sweep overwriting a weekly edition. Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security @@ -50,7 +50,7 @@ INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"} HOST_REPORTS = "/var/lib/cc-ci-reports" # Both kinds live in ONE archive, distinguished by a suffix on a common title. # prefix -> (page title, index label) -KINDS = {"week": ("The Recipe Report", "Week of {d} — full report"), +KINDS = {"week": ("The Recipe Report", "Week of {d} — full"), "cve": ("The Recipe Report — CVE check", "{d} — CVE check")} diff --git a/cc-ci-plan/resolve-images.py b/cc-ci-plan/resolve-images.py new file mode 100644 index 0000000..8721763 --- /dev/null +++ b/cc-ci-plan/resolve-images.py @@ -0,0 +1,459 @@ +#!/usr/bin/env python3 +"""resolve-images — what version is each of a recipe's images on, and what is newest? + +An abra-independent version resolver. `abra recipe upgrade` is the normal path, but it has a hard +failure mode: an image pinned with BOTH a tag and a digest makes it FATA and abandon the WHOLE +recipe — even images it already parsed. immich pins two that way: + + ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf6… + docker.io/valkey/valkey:9@sha256:3acc… + +so immich contributes NO version data at all and silently drops out of every survey. That is +indistinguishable from "up to date" unless a human notices the missing row — which is exactly how it +kept getting skipped, and why a CVE sweep reported it as unknown. + +This reads the compose files directly and queries the registries itself, so a digest pin is just a +digest pin. Output is JSON (default) or a table. + + resolve-images.py [--ssh HOST] [--recipe-dir DIR] [--table] [--only IMAGE] + +The cc-ci host has no python3, so `--ssh cc-ci` reads the compose files from that host's checkout +over ssh and does the resolving locally. That keeps the source of truth the SAME tree abra and CI +use, rather than a second copy that can drift. + +TAG SHAPES. Registries mix wildly different tag conventions in one repo, so "newest" is meaningless +without a shape. Each tag is reduced to a signature by replacing digit runs with '#': + + v3.1.0 -> v#.#.# + 1.27.1-rootless -> #.#.#-rootless + 8.10-alpine -> #.#-alpine + 14-vectorchord0.4.3-pgvectors0.2.0 -> #-vectorchord#.#.#-pgvectors#.#.# + +Only tags sharing the CURRENT pin's shape are candidates. That keeps `-alpine` on `-alpine`, and +stops a `latest`/`release`/`sha-…` tag from ever being proposed as an upgrade. + +TWO ANSWERS, NOT ONE. It reports `newest_same_shape` AND `newest_within_major` (same leading number). +For a plain app image they usually agree. For a compatibility-pinned sidecar they do not, and taking +the max would be wrong: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors +versions that immich-server is built against, so jumping pg major because a newer tag exists breaks +the deployment. The caller picks; this tool refuses to guess and shows both. +""" + +from __future__ import annotations + +import argparse +import glob +import shlex +import subprocess +import gzip +import json +import os +import re +import sys +import time +import urllib.error +import urllib.parse +import urllib.request + +UA = "cc-ci-resolve-images (+https://git.autonomic.zone/recipe-maintainers/cc-ci)" +TIMEOUT = int(os.environ.get("RESOLVE_IMAGES_TIMEOUT", "45")) +RECIPE_DIR = os.environ.get("ABRA_RECIPE_DIR", os.path.expanduser("~/.abra/recipes")) +MAX_TAG_PAGES = int(os.environ.get("RESOLVE_IMAGES_MAX_PAGES", "40")) + +IMAGE_RE = re.compile(r"""^\s*image:\s*["']?([^"'\s]+)["']?\s*$""", re.M) + + +RETRIES = int(os.environ.get("RESOLVE_IMAGES_RETRIES", "4")) + + +def _fetch(url: str, headers: dict | None = None) -> bytes: + """GET with backoff on rate limits. + + Docker Hub throttles anonymous clients hard, and a sweep re-reads the same popular repos + (nginx, redis, postgres) for recipe after recipe. A 429 mid-sweep used to surface as + 'unresolved', which is indistinguishable from a real lookup failure — so retry, and let the + per-repo cache below remove most of the requests entirely.""" + h = {"User-Agent": UA, "Accept-Encoding": "gzip"} + h.update(headers or {}) + delay = 2.0 + for attempt in range(RETRIES): + try: + with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r: + raw = r.read() + if r.headers.get("Content-Encoding") == "gzip": + raw = gzip.decompress(raw) + return raw + except urllib.error.HTTPError as e: + if e.code in (429, 503) and attempt < RETRIES - 1: + time.sleep(delay) + delay *= 2 + continue + raise + raise RuntimeError("unreachable") + + +_HUB_JWT: list = [] + + +def _hub_auth() -> dict: + """Authenticated Docker Hub calls get a far higher rate limit than anonymous ones. + + Credentials come from /srv/cc-ci/.testenv (DOCKERHUB_USERNAME / DOCKERHUB_TOKEN), the same pair + the CI host already uses. Absent creds are fine — the sweep just runs anonymous and slower.""" + if _HUB_JWT: + return _HUB_JWT[0] + env = {} + try: + for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")): + if "=" in ln and not ln.strip().startswith("#"): + k, v = ln.strip().split("=", 1) + env[k] = v.strip().strip("\"'") + except OSError: + pass + u = os.environ.get("DOCKERHUB_USERNAME") or env.get("DOCKERHUB_USERNAME") + t = os.environ.get("DOCKERHUB_TOKEN") or env.get("DOCKERHUB_TOKEN") + hdrs = {} + if u and t: + try: + body = json.dumps({"username": u, "password": t}).encode() + req = urllib.request.Request("https://hub.docker.com/v2/users/login", + data=body, method="POST", + headers={"Content-Type": "application/json", "User-Agent": UA}) + with urllib.request.urlopen(req, timeout=TIMEOUT) as r: + tokj = json.load(r).get("token") + if tokj: + hdrs = {"Authorization": f"JWT {tokj}"} + except Exception: # noqa: BLE001 — anonymous is a valid fallback + hdrs = {} + _HUB_JWT.append(hdrs) + return hdrs + + +def _json(url: str, headers: dict | None = None): + return json.loads(_fetch(url, headers)) + + +def shape(tag: str) -> str: + """Signature of a tag with every digit run replaced by '#'. See module docstring.""" + return re.sub(r"\d+", "#", tag) + + +def vkey(tag: str) -> tuple: + """Ordering key: every number in the tag, in order. '1.27.10' > '1.27.9'; text ignored.""" + return tuple(int(x) for x in re.findall(r"\d+", tag)) + + +def parse_ref(ref: str) -> dict: + """Split an image reference into registry / repo / tag / digest.""" + digest = None + if "@" in ref: + ref, _, digest = ref.partition("@") + host, repo, tag = "docker.io", ref, "latest" + # A leading component is a REGISTRY only when there is a path after it. Without the slash test, + # a bare `postgres:15.18` looks like host "postgres:15.18" because of the tag's colon — which + # silently sent every library image to a nonexistent registry. + if "/" in ref: + first = ref.split("/")[0] + if "." in first or ":" in first or first == "localhost": + host, _, repo = ref.partition("/") + if ":" in repo.split("/")[-1]: + repo, _, tag = repo.rpartition(":") + if host == "docker.io" and "/" not in repo: + repo = f"library/{repo}" # bare `redis` is really `library/redis` + return {"registry": host, "repo": repo, "tag": tag, "digest": digest} + + +HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10")) + + +def _hub_tag_exists(repo: str, tag: str) -> bool: + try: + _json(f"https://hub.docker.com/v2/repositories/{repo}/tags/{tag}", _hub_auth()) + return True + except Exception: # noqa: BLE001 + return False + + +def _hub_tags(repo: str) -> list[str]: + """Recently-pushed tags, newest first. + + Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical — + but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so + ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade. + (ghcr offers no ordering, which is why that path needs a different strategy.) + """ + tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags" + f"?page_size=100&ordering=last_updated") + auth = _hub_auth() + for _ in range(HUB_RECENT_PAGES): + d = _json(url, auth) + tags += [r["name"] for r in d.get("results", [])] + url = d.get("next") + if not url: + break + return tags + + +def _oci_bearer(host: str, repo: str) -> dict: + """Token for an OCI registry, discovered from its own auth challenge. + + Registries do NOT share a token endpoint. ghcr answers at /token?scope=…&service=ghcr.io, but + lscr.io and dock.mau.dev advertise different realms, and assuming ghcr's shape made both 401 — + which then read as "could not resolve" rather than "asked the wrong URL". The registry tells us + where to go in its WWW-Authenticate header; use that.""" + try: + urllib.request.urlopen( + urllib.request.Request(f"https://{host}/v2/{repo}/tags/list?n=1", + headers={"User-Agent": UA}), timeout=TIMEOUT) + return {} # no auth needed + except urllib.error.HTTPError as e: + if e.code != 401: + return {} + chal = e.headers.get("WWW-Authenticate", "") or "" + except Exception: # noqa: BLE001 + return {} + if not chal.lower().startswith("bearer"): + return {} + parts = dict(re.findall(r'(\w+)="([^"]*)"', chal)) + realm = parts.get("realm") + if not realm: + return {} + q = {"service": parts.get("service", host), "scope": parts.get("scope", f"repository:{repo}:pull")} + url = realm + ("&" if "?" in realm else "?") + urllib.parse.urlencode(q) + try: + tok = (_json(url) or {}).get("token") or (_json(url) or {}).get("access_token") + return {"Authorization": f"Bearer {tok}"} if tok else {} + except Exception: # noqa: BLE001 + return {} + + +def _oci_tags(host: str, repo: str) -> list[str]: + """Tags from any OCI/v2 registry, with challenge-derived auth and Link pagination. + + ghcr paginates hard — immich-server has >40,000 tags — and a truncated listing silently hides + the newest release line, so follow the cursor and let the caller's integrity check catch a read + that never reached the current pin.""" + hdrs = _oci_bearer(host, repo) + tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000" + for _ in range(MAX_TAG_PAGES): + req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs}) + with urllib.request.urlopen(req, timeout=TIMEOUT) as r: + tags += (json.load(r) or {}).get("tags") or [] + link = r.headers.get("Link", "") or "" + m = re.search(r'<([^>]+)>;\s*rel="next"', link) + if not m: + break + nxt = m.group(1) + url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt + return tags + + +def _gh_token() -> str | None: + tok = os.environ.get("GITHUB_TOKEN") + if tok: + return tok.strip() + try: + return open(os.environ.get("GITHUB_TOKEN_FILE", "/srv/cc-ci/.github-token")).read().strip() or None + except OSError: + return None + + +def github_release_tags(owner: str, repo: str, max_pages: int = 4) -> list[str]: + """Release tag names for a GitHub repo, newest first. + + FALLBACK for registries whose tag listing cannot be enumerated. ghcr has no ordering and no + server-side filter, and immich-machine-learning carries >40,000 tags — a full read is impractical + and a partial read silently hides the newest release line. The project's RELEASES are ordered, + small, and authoritative: container tags track them. (The GitHub Packages API would answer this + directly but needs a scoped token; this scan's token deliberately has none.) + """ + hdrs = {"Accept": "application/vnd.github+json"} + tok = _gh_token() + if tok: + hdrs["Authorization"] = f"Bearer {tok}" + out = [] + for page in range(1, max_pages + 1): + try: + rows = _json(f"https://api.github.com/repos/{owner}/{repo}/releases" + f"?per_page=100&page={page}", hdrs) + except Exception: # noqa: BLE001 + break + if not rows: + break + out += [r.get("tag_name") or "" for r in rows] + return [t for t in out if t] + + +def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]: + """Candidate GitHub repos whose releases track this image's tags.""" + if "ghcr.io" not in registry: + return [] + parts = repo.split("/") + if len(parts) < 2: + return [] + owner, name = parts[0], parts[-1] + cands = [(owner, name)] + # ghcr.io/immich-app/immich-machine-learning is built from immich-app/immich. + if name.startswith(owner.split("-")[0]): + cands.append((owner, owner.split("-")[0])) + return cands + + +_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {} + + +def list_tags(registry: str, repo: str) -> tuple[list[str], str | None]: + if (registry, repo) in _TAG_CACHE: + return _TAG_CACHE[(registry, repo)] + res = _list_tags_uncached(registry, repo) + _TAG_CACHE[(registry, repo)] = res + return res + + +def _list_tags_uncached(registry: str, repo: str) -> tuple[list[str], str | None]: + try: + return (_hub_tags(repo) if registry in ("docker.io", "registry-1.docker.io") + else _oci_tags(registry, repo)), None + except urllib.error.HTTPError as e: + return [], f"HTTP {e.code}" + except Exception as e: # noqa: BLE001 + return [], f"{type(e).__name__}: {e}" + + +def resolve(ref: str) -> dict: + """Current pin -> newest same-shape tag, and newest within the current major.""" + if "${" in ref or "$(" in ref: + # The tag is a compose variable (ghost pins `ghost:${IMAGE_VERSION}-alpine`). Its real value + # lives in .env, not here. Report it as skipped, never as a failed lookup. + return {**parse_ref(ref), "ref": ref, "shape": None, "candidates": 0, + "newest_same_shape": None, "newest_within_major": None, + "upgrade_available": False, + "status": "skipped: templated ref (tag comes from a compose variable)"} + info = parse_ref(ref) + out = {**info, "ref": ref, "shape": shape(info["tag"]), "status": "ok", + "newest_same_shape": None, "newest_within_major": None, "candidates": 0, + "upgrade_available": False} + tags, err = list_tags(info["registry"], info["repo"]) + if err: + out["status"] = f"error: {err}" + return out + out["tags_seen"] = len(set(tags)) + # INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not, + # the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens + # of thousands of tags and a truncated read silently hides whole release lines. immich's + # machine-learning image is pinned v3.1.0, which EXISTS, yet a short read reported v1.134.0 as + # newest; without this check that becomes a confident, wrong answer. + if info["tag"] not in set(tags): + # Docker Hub: the window is recency-ordered, so the pin being outside it just means the pin + # is old — which is fine, because anything NEWER is necessarily inside the window. Confirm + # the pin genuinely exists (so a typo is still caught) and carry on. + if info["registry"] in ("docker.io", "registry-1.docker.io") and _hub_tag_exists(info["repo"], info["tag"]): + out["source"] = f"docker-hub:recent-{HUB_RECENT_PAGES * 100}" + tags = list(tags) + [info["tag"]] + else: + for owner, name in _release_fallback_repos(info["registry"], info["repo"]): + rel = github_release_tags(owner, name) + if info["tag"] in rel: + tags = rel + out["source"] = f"github-releases:{owner}/{name}" + out["tags_seen"] = len(set(rel)) + break + else: + out["status"] = ("error: tag listing incomplete — the current pin " + f"{info['tag']!r} is absent from {len(set(tags))} registry tags " + f"and from the project's GitHub releases") + return out + want, cur = out["shape"], vkey(info["tag"]) + same = [t for t in set(tags) if shape(t) == want and vkey(t)] + out["candidates"] = len(same) + if not same: + # Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings. + out["status"] = "no comparable tags (shape has no numeric siblings)" + return out + newest = max(same, key=vkey) + out["newest_same_shape"] = newest + if cur: + within = [t for t in same if vkey(t)[:1] == cur[:1]] + if within: + out["newest_within_major"] = max(within, key=vkey) + out["upgrade_available"] = bool(cur and vkey(newest) > cur) + return out + + +def compose_images_ssh(recipe: str, host: str, recipe_dir: str) -> list[str]: + """Same as compose_images, but the recipe tree lives on another host (cc-ci has no python3).""" + # NB: no shell-quoting of the directory — it may legitimately start with ~ or $HOME, and + # quoting it stops the remote shell expanding it, which yields an empty (and silent) result. + d = f"{recipe_dir}/{shlex.quote(recipe)}".replace("~", "$HOME") + cmd = (f'for f in {d}/compose*.yml; do case "$f" in *compose.ccci.yml) continue;; esac; ' + f'[ -f "$f" ] && {{ cat "$f"; echo; }}; done; exit 0') + out = subprocess.run(["ssh", host, cmd], capture_output=True, text=True, timeout=120) + if out.returncode != 0: + raise RuntimeError(f"ssh {host}: {(out.stderr.strip() or 'no output')[:200]}") + if not out.stdout.strip(): + raise RuntimeError(f"ssh {host}: no compose files found under {d}") + refs = [] + for m in IMAGE_RE.finditer(out.stdout): + if m.group(1) not in refs: + refs.append(m.group(1)) + return refs + + +def compose_images(recipe: str, recipe_dir: str) -> list[str]: + """Every `image:` ref in the recipe's own compose files (the cc-ci overlay is NOT the recipe).""" + refs, base = [], os.path.join(recipe_dir, recipe) + for path in sorted(glob.glob(os.path.join(base, "compose*.yml"))): + if os.path.basename(path) == "compose.ccci.yml": + continue + try: + for m in IMAGE_RE.finditer(open(path).read()): + if m.group(1) not in refs: + refs.append(m.group(1)) + except OSError: + continue + return refs + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("recipe") + ap.add_argument("--ssh", default=None, metavar="HOST", + help="read the recipe's compose files from HOST over ssh (e.g. --ssh cc-ci); " + "resolving still happens locally") + ap.add_argument("--recipe-dir", default=RECIPE_DIR) + ap.add_argument("--table", action="store_true", help="human-readable table instead of JSON") + ap.add_argument("--only", default=None, help="resolve just the images whose ref contains this") + a = ap.parse_args() + + rdir = a.recipe_dir if a.recipe_dir != RECIPE_DIR or not a.ssh else "~/.abra/recipes" + refs = (compose_images_ssh(a.recipe, a.ssh, rdir) if a.ssh + else compose_images(a.recipe, a.recipe_dir)) + if a.only: + refs = [r for r in refs if a.only in r] + results = [resolve(r) for r in refs] + report = { + "recipe": a.recipe, + "images": results, + "upgrades_available": [r["ref"] for r in results if r["upgrade_available"]], + "unresolved": [r["ref"] for r in results if r["status"].startswith("error")], + # The whole point: distinguish "checked, current" from "could not check". + "all_resolved": not any(r["status"].startswith("error") for r in results), + } + if not a.table: + print(json.dumps(report, indent=2)) + return 0 + print(f"{a.recipe} — {len(results)} images") + for r in results: + flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current") + print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else "")) + print(f" shape={r['shape']} candidates={r['candidates']}" + f" newest_same_shape={r['newest_same_shape']}" + f" newest_within_major={r['newest_within_major']}") + if r["status"] != "ok": + print(f" status: {r['status']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/cc-ci-plan/test-advisory-scan.py b/cc-ci-plan/test-advisory-scan.py index 69e46c3..80be3ea 100755 --- a/cc-ci-plan/test-advisory-scan.py +++ b/cc-ci-plan/test-advisory-scan.py @@ -503,6 +503,40 @@ class TestReleaseNoteResolution(unittest.TestCase): self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {}) +class TestReleaseLineSemantics(unittest.TestCase): + """A fix inside the numeric window is not a fix on the branch you actually land on.""" + + def test_fix_later_on_the_targets_own_line_is_not_counted(self): + # ClickHouse fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20. Landing on 23.10.4.25 crosses + # the 23.9 fix numerically but is BELOW its own line's fix, so it does not have it. + rep = run_scan([gh("ClickHouse/ClickHouse", + [adv("CVE-2023-48704", patched="v23.10.5.20; v23.9.6.20; v23.8.8.20")])], + v_from="23.4.2.11", v_to="23.10.4.25", + urls=["https://github.com/ClickHouse/ClickHouse"]) + self.assertEqual(rep["fixed_by_this_upgrade"], []) + + def test_fix_earlier_on_the_targets_own_line_is_counted(self): + rep = run_scan([gh("ClickHouse/ClickHouse", + [adv("CVE-2023-47118", patched="v23.10.2.13; v23.8.6.16")])], + v_from="23.4.2.11", v_to="23.10.4.25", + urls=["https://github.com/ClickHouse/ClickHouse"]) + self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-47118"]) + + def test_fix_exactly_at_the_target_is_counted(self): + rep = run_scan([gh("ClickHouse/ClickHouse", + [adv("CVE-2023-48298", patched="v23.10.4.25; v23.9.5.29")])], + v_from="23.4.2.11", v_to="23.10.4.25", + urls=["https://github.com/ClickHouse/ClickHouse"]) + self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-48298"]) + + def test_no_fix_on_the_target_line_falls_back_to_the_window(self): + # redis fixes 7.4.6/8.0.4/8.2.2 with no 8.10.x entry; landing on 8.10 still has them, + # because nothing on the 8.10 line is named as a LATER fix. + rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2")])], + v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"]) + self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"]) + + class TestAdjudicationEvidenceAssembly(unittest.TestCase): """Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown.""" @@ -624,16 +658,22 @@ class TestHistoricReportNumbers(unittest.TestCase): for cve in delta: self.assertIn("redis", both["cves"][cve]["sources"][0]) - def test_mailu_scan_finds_zero_and_says_so_knowingly(self): - # Published as 2 via the UNION with release-note reading; the scan's own contribution is 0, - # and 0 here must mean "checked, none", not "could not check". + def test_mailu_finds_the_roundcube_pair_without_an_agent(self): + # Published as 2 on 2026-08-07, but only because an agent read the release notes; the scan + # itself contributed 0. It now reaches 2 deterministically: the CVEs appear only on + # github.com/Mailu/Mailu/releases, and release 2024.06.56 (inside the window) names them. rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")]) - self.assertEqual(rep["cve_count_fixed"], 0) - self.assertTrue(rep["count_known"]) + self.assertEqual(rep["cve_count_fixed"], 2) + self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-54432", "CVE-2026-54433"}) + # The redis bump fixes nothing new — every advisory it crosses was fixed at or before 8.6.3. + self.assertEqual(rep["cve_count_indeterminate"], 0) - def test_keycloak_26_7_0_to_26_7_1_is_7(self): + def test_keycloak_26_7_0_to_26_7_1_is_12(self): + # Was 7 while only the GHSA feed was consulted. keycloak lists five more CVEs in the 26.7.1 + # release notes' fixed-issues section that it never filed as advisories — the gitea pattern. rep = self._count("keycloak", "26.7.0", "26.7.1") - self.assertEqual(rep["cve_count_fixed"], 7) + self.assertEqual(rep["cve_count_fixed"], 12) + self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 5) def _main(): diff --git a/cc-ci-plan/upstream/mumble.md b/cc-ci-plan/upstream/mumble.md new file mode 100644 index 0000000..859ea64 --- /dev/null +++ b/cc-ci-plan/upstream/mumble.md @@ -0,0 +1,21 @@ +# Upstream sources — mumble + +| service | image | source repo | releases / changelog | +|---------|-------|-------------|----------------------| +| app | mumblevoip/mumble-server | https://github.com/mumble-voip/mumble | https://github.com/mumble-voip/mumble/releases | +| web | rankenstein/mumble-web | https://github.com/rankenstein/mumble-web | https://github.com/rankenstein/mumble-web/releases | + +## Standing notes +- This file was **missing entirely** until 2026-08-11. Without it the advisory scan had no source to + query, and still printed "0 identified by the deterministic scan" — which was then published as a + clean `0` in the 2026-08-11 CVE check. The scan now refuses to emit a count when it has no usable + source (it reports UNKNOWN), and `audit-sources.py` flags a missing registry file directly. +- `mumblevoip/mumble-server` tracks the upstream server releases and DOES publish GitHub security + advisories, so it is the recipe's primary CVE source. +- `rankenstein/mumble-web` is a **fork** of the original `Johni0702/mumble-web`, which has been + dormant since 2023-05. The fork itself last pushed 2023-07 and its Docker tag `0.5` was last built + well over five years ago. Neither is archived, but treat the web client as effectively unmaintained: + if a CVE lands there, expect no upstream fix and plan a replacement rather than an upgrade. +- The server image tag is `v-` (e.g. `v1.6.870-4`); the trailing number is the image + build, not an app version, and moves independently of upstream releases — `abra recipe upgrade` + reports "no new versions" for it, so use `resolve-images.py` to see those bumps.