Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f339401bab | ||
|
|
e89da2d842 | ||
|
|
6c91373357 | ||
|
|
1db85a7e77 | ||
|
|
8df32edfcf | ||
|
|
18caf047bf | ||
|
|
b0bdce2c15 |
@@ -79,6 +79,10 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
|
||||
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
|
||||
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
|
||||
recipe.
|
||||
- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
|
||||
change must satisfy — set state up through the app's interface rather than its database, gate on
|
||||
version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
|
||||
the app's own telemetry before concluding a test is stale.
|
||||
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
|
||||
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
|
||||
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: cve-check-and-upgrade
|
||||
description: Security-driven upgrade run. Does a full /cve-check sweep first (per-image advisory scan of every recipe's available upgrade, with adjudication), then runs /recipe-upgrade ONLY on the recipes whose upgrade fixes at least one CVE — worst severity first — opening a verified recipe PR for each, and finally publishes one report covering both the sweep and the PRs. Recipes with no CVEs are left alone; that is the point. NEVER merges. Invoke as /cve-check-and-upgrade [recipe ...] [--min-severity high] [--capacity N] [--dry-run].
|
||||
---
|
||||
|
||||
# cve-check-and-upgrade
|
||||
|
||||
`/upgrade-all` upgrades everything that *has* an upgrade. **This upgrades what has a reason.** It runs
|
||||
the `/cve-check` sweep, then spends CI time only on the recipes where an upgrade actually closes a
|
||||
vulnerability, handling the worst first.
|
||||
|
||||
Use it when you want to act on security rather than churn the whole fleet: after a vendor announcement,
|
||||
when CI capacity is short, or between weekly runs. When you only want to *know*, use `/cve-check`. When
|
||||
you want everything current regardless of CVEs, use `/upgrade-all`.
|
||||
|
||||
**Creates PRs. Never merges.** Every PR is verified green on cc-ci and left for a human.
|
||||
|
||||
## Arguments
|
||||
- `<recipe> …` — restrict the whole run to these recipes.
|
||||
- `--min-severity critical|high|medium|low` — only upgrade recipes whose fixed CVEs reach this
|
||||
severity. Default **`low`** (any CVE at all justifies the upgrade). `--min-severity high` is the
|
||||
useful "just the urgent ones" setting.
|
||||
- `--capacity N` — subagent pool size; defaults to the live `DRONE_RUNNER_CAPACITY` (the drone
|
||||
runner's slots), matching `/upgrade-all`'s rolling-pool behaviour.
|
||||
- `--dry-run` — do the whole sweep and print exactly which recipes *would* be upgraded and why, then
|
||||
stop without spawning a single upgrade. **Publishes no report and opens no PR.**
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Sweep — run `/cve-check` in full
|
||||
Follow `.claude/skills/cve-check/SKILL.md` steps 1–5 exactly: candidate list, per-image upgrade windows,
|
||||
the advisory scan per recipe, pass-2 adjudication of anything undecided, and the severity classification.
|
||||
**Do not publish its report** — this run produces one combined report at the end instead.
|
||||
|
||||
Keep, per recipe: the windows scanned, the CVE count, the CVE ids with severities, and whether the count
|
||||
is a floor (undetermined advisories remain) or unknown.
|
||||
|
||||
### 2. Decide what to upgrade
|
||||
`RECIPES_TO_UPGRADE` = recipes where **the scan found ≥1 CVE** at or above `--min-severity`.
|
||||
|
||||
Deliberate exclusions, each recorded in the report with its reason:
|
||||
- **0 CVEs** — an upgrade may exist, but nothing security-relevant. Left alone; that is the point of
|
||||
this skill. `/upgrade-all` is what sweeps those up.
|
||||
- **`external` tier** — swept for visibility, **never upgraded here**; someone else maintains it. Flag
|
||||
it loudly in the report if it has a critical, since the action is to tell them, not to open a PR.
|
||||
- **`UPTODATE`** — nothing available.
|
||||
- **count `?` / UNKNOWN** — do **not** upgrade blind, and do **not** treat it as clean. Put it in the
|
||||
Addendum as needing a look. An unknown is a gap in our knowledge, not evidence of safety.
|
||||
|
||||
Order the queue by **worst severity first** (critical → high → …), count breaking ties. If `--dry-run`,
|
||||
print this queue with each recipe's CVE ids and severities, and STOP here.
|
||||
|
||||
### 3. Upgrade each one — via `/recipe-upgrade` subagents
|
||||
Run `/recipe-upgrade <recipe>` per queued recipe as a **subagent**, in the queue order above, as a
|
||||
**rolling pool** keeping `--capacity` (default `DRONE_RUNNER_CAPACITY`) running at once and starting the
|
||||
next as each finishes — the same concurrency discipline as `/upgrade-all` §3, and safe for the same
|
||||
reason (per-run recipe trees + app-domain locks).
|
||||
|
||||
Each subagent does the full job: plan, implement the bump, verify green on cc-ci with `!testme`, and
|
||||
open a recipe PR. **Default mode — no `--with-tests`**: a genuinely stale test gets an explanatory PR
|
||||
comment, not a test edit.
|
||||
|
||||
**Tell each subagent which CVEs justify its upgrade**, with ids and severities, so the PR description
|
||||
says why it exists. That is most of this skill's value to a reviewer: a PR that names the CVSS-9.8 RCE
|
||||
it closes gets merged today, an unexplained version bump waits a fortnight.
|
||||
|
||||
Collect per recipe: PR url + number, the `!testme` verdict and build number, and any failure.
|
||||
|
||||
### 4. Report — one page covering sweep AND PRs
|
||||
Write `/tmp/cve-spec-<DATE>.json` per `/cve-check` step 6, with these differences:
|
||||
- Rows for upgraded recipes carry the real `ci` (`build N ✓` / `RED N · <stage>`) + `ci_url`, and
|
||||
`pr`/`pr_url`. `status` is the CI verdict (`GREEN`/`FAILED`/`STALE`); the live PR-status column
|
||||
derives itself from `recipe` + `pr`.
|
||||
- Rows for swept-but-not-upgraded recipes keep `PENDING`/`UPTODATE` with empty `ci`/`pr`, and a
|
||||
`notes` reason (`0 CVEs — not upgraded`, `external — maintained elsewhere`, `below --min-severity`).
|
||||
- Include `changes[]` — one entry per recipe that got a PR, describing what the upgrade changes **and
|
||||
the CVEs it closes**.
|
||||
- Keep `"kind": "cve"`: it titles the page "The Recipe Report — CVE check" and files it as
|
||||
`cve-<DATE>.html`, alongside the weekly editions in the same archive index.
|
||||
|
||||
Then render + publish exactly as `/cve-check` step 7, and verify as its step 8. Print the report URL,
|
||||
`N swept · M upgraded · K PRs green · J failed`, and `CVE CHECK AND UPGRADE COMPLETE`.
|
||||
|
||||
## Guardrails
|
||||
- **NEVER merge.** Create and verify; a human merges. Never push to true upstream.
|
||||
- **Never weaken a test** to make a PR green, and never edit a test without `--with-tests`.
|
||||
- **Never upgrade a recipe whose CVE count is unknown** on the assumption it is fine — surface it.
|
||||
- **Never upgrade an `external` recipe** here, even with a critical; report it instead.
|
||||
- **Public-safe report only** — no secrets, tokens, internal hostnames, raw logs, or spend figures.
|
||||
- If the sweep finds **nothing** at or above `--min-severity`, that is a good outcome: publish the
|
||||
report saying so and open no PRs. Do not manufacture work.
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
name: cve-check
|
||||
description: Fleet-wide CVE sweep WITHOUT upgrading anything. For every recipe cc-ci deploys, works out what upgrade is available (current pinned tag → newest supported tag, per image including sidecars), runs the deterministic advisory scan over that window, adjudicates whatever the scan could not decide, and publishes a CVE report to report.ci.commoninternet.net as cve-<DATE>.html. READ-ONLY — opens no PRs, edits no recipes, runs no CI, merges nothing. Answers "what are we exposed to that an upgrade would fix?" in minutes rather than the hours a full upgrade run takes. Invoke as /cve-check [recipe ...] [--weekly-only].
|
||||
---
|
||||
|
||||
# cve-check
|
||||
|
||||
A **security sweep, not an upgrade run.** It answers one question for every recipe cc-ci deploys:
|
||||
|
||||
> If we upgraded this recipe today, how many CVEs would that fix, and how bad are they?
|
||||
|
||||
It is the cheap, safe half of `/upgrade-all`: the same version research and the same advisory scan,
|
||||
with **no implementation, no CI, and no PRs**. Use it when you want the security picture now — after a
|
||||
vendor announcement, before deciding what to prioritise, or between weekly runs. When you want the PRs
|
||||
too, use **`/cve-check-and-upgrade`**.
|
||||
|
||||
**Read-only, absolutely.** Never edit a recipe, never open or comment on a PR, never merge, never
|
||||
deploy. The only thing it writes is its own log and the published report page.
|
||||
|
||||
## Arguments
|
||||
- `<recipe> …` — sweep only these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
|
||||
- `--weekly-only` — skip rows tagged `external`. **Off by default on purpose**: an `external` recipe is
|
||||
still deployed and still exposes us, so a security sweep that silently skipped it would misreport the
|
||||
fleet's exposure. Externals are swept and clearly marked "maintained elsewhere" in the report.
|
||||
|
||||
## Procedure
|
||||
|
||||
> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
|
||||
> `abra` needs a TTY. Wrap every abra call: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
|
||||
> (`git` and other commands do NOT need the wrapper.)
|
||||
|
||||
### 1. Build the candidate list
|
||||
Read `cc-ci-plan/used-recipes.md` — the canonical inventory. Take every row (both tiers), recording the
|
||||
tier per recipe; with `--weekly-only`, drop the `external` rows. An explicit recipe argument overrides
|
||||
any skip.
|
||||
|
||||
### 2. Per recipe — establish the upgrade window WITHOUT upgrading
|
||||
|
||||
This is `/recipe-upgrade` step 1's research, stopping before it implements anything.
|
||||
|
||||
> ⚠️ **The same four things that silently skip recipes apply here — handle ALL FOUR:**
|
||||
> 1. **pseudo-TTY** — per the box above.
|
||||
> 2. **go-git auth to git.autonomic.zone** — recipes on the private mirror FATA
|
||||
> `authentication required: Unauthorized`. Bake creds into origin first (idempotent, only when
|
||||
> origin is on git.autonomic.zone):
|
||||
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
||||
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
||||
> after. Only a genuinely dirty TRACKED tree is a skip.
|
||||
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
|
||||
> hand-check the registry; run the resolver, which is abra-independent and covers every image:
|
||||
> ```
|
||||
> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
|
||||
> ```
|
||||
> It reports, per image, `newest_within_major` (the compatibility-safe pick) and
|
||||
> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
|
||||
> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
|
||||
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
|
||||
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
|
||||
|
||||
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile
|
||||
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
||||
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
||||
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
||||
lives in **branches**, never directly on `main`, so a force-sync of `main` to upstream discards
|
||||
nothing; it also auto-closes mirror PRs whose changes upstream has already merged.
|
||||
|
||||
> ### ⚠️ The default branch may be `master`, not `main` — check, do not assume
|
||||
> Several coopcloud recipes keep a **stale `main` alongside the real default `master`**. gitea is one:
|
||||
> `main` sits at 1.24.2-rootless while `master` has 1.27.1-rootless plus the merged PRs and the 3.6.3
|
||||
> release. Reading `main` there tells you the recipe is three releases behind and missing two CVSS-9.8
|
||||
> RCE fixes — a false alarm that reads exactly like a real one. Resolve the default branch from the
|
||||
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
||||
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
||||
|
||||
**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
|
||||
nothing for images it cannot parse, and it reported "no new versions" for images that did have them
|
||||
(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
|
||||
the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
|
||||
(plausible's clickhouse) carried four CVEs.
|
||||
|
||||
Then read versions:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
|
||||
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
|
||||
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
|
||||
git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
|
||||
script -qec "abra recipe fetch $R --force -n" /dev/null; \
|
||||
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
|
||||
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
|
||||
```
|
||||
|
||||
For each recipe produce **one window per image**: `current pinned tag → newest supported tag`. You need
|
||||
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
|
||||
CRITICAL came from, and an image with no window is not counted at all.
|
||||
|
||||
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
|
||||
nothing an upgrade could fix. Record it and move on.
|
||||
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
|
||||
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
|
||||
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
|
||||
Only report `?` once BOTH the abra check and the direct check have failed. On the first real run this
|
||||
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
|
||||
but nothing in the survey said so).
|
||||
|
||||
### 3. Run the advisory scan over that window
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
|
||||
[--image <name>=<old>:<new>]...
|
||||
```
|
||||
**One call per recipe with every image in it** — the count is a union across images, and the
|
||||
UNKNOWN guarantee only holds when a single run sees them all. Paste the markdown block verbatim into
|
||||
the per-recipe log at `/srv/cc-ci/.cc-ci-logs/cve-check/<DATE>/<recipe>.md`.
|
||||
|
||||
### 4. Adjudicate what the scan could not decide (pass 2)
|
||||
If the block reports advisories it **could NOT judge**, or the count is **UNKNOWN**, re-run with
|
||||
`--adjudicate` and decide each open case yourself:
|
||||
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> … --adjudicate
|
||||
```
|
||||
|
||||
Answer **FIXED / NOT-FIXED / STILL-UNKNOWN** per case, each with a one-line reason **citing the
|
||||
evidence shown** — never from memory of the project, which is the exact failure that let two CVSS-9.8
|
||||
gitea RCEs be published as "none". Every FIXED is added to the count; pass 1's number is a floor. The
|
||||
block also lists what pass 1 already decided — if a verdict looks wrong given its evidence, say so.
|
||||
Record your verdicts in the per-recipe log so the number is auditable.
|
||||
|
||||
### 5. Classify severity and priority
|
||||
For each recipe collect the CVE ids with **severities** (the scan gives them, with GHSA ids). Sort the
|
||||
report rows by what an operator should deal with first:
|
||||
1. recipes with a **critical**, then **high**, then anything else with CVEs (more CVEs higher within a band);
|
||||
2. then `?` (a count that could not be established — investigate, do not ignore);
|
||||
3. then recipes with an upgrade available but **0** CVEs;
|
||||
4. then `UPTODATE`.
|
||||
|
||||
Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin advisories.
|
||||
|
||||
### 6. Write the report spec
|
||||
`/tmp/cve-spec-<DATE>.json`, same shape as `/recipe-report` (see `recipe-report.py`'s header), with:
|
||||
- **`"kind": "cve"`** — titles the page "The Recipe Report — CVE check" and files it as
|
||||
`cve-<DATE>.html`. It appears in the SAME archive index as the weekly editions, suffixed
|
||||
"— CVE check" so the two are told apart at a glance. Without this field you would overwrite that
|
||||
date's weekly edition.
|
||||
- `date`, `subtitle` "CVE check <human date>",
|
||||
- `lead` — **one short paragraph**: fleet exposure in a sentence and what to do first.
|
||||
- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
|
||||
`1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
|
||||
`PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
|
||||
rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for
|
||||
`external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
|
||||
- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
|
||||
established, a scan whose count is a floor with many undetermined advisories.
|
||||
- `security[]` — one entry per **critical/high** finding: recipe · CVE id(s) + severity · what it fixes
|
||||
· **which image** it is in. Name the image: `CVE-2025-49844` is a redis flaw, and an operator reading
|
||||
"discourse" needs to know that.
|
||||
- `changes[]` — **omit** (nothing changed; there are no PRs).
|
||||
|
||||
**`?` must stay RARE.** Use it only when a scan ran and reported genuinely failed sources, or the count
|
||||
came back UNKNOWN and adjudication could not settle it. Never `none` for an unknown — a blank reads as
|
||||
clean. A recipe with no upgrade available is `0`, not `?`. Many `?` is a bug for the Addendum.
|
||||
|
||||
### 7. Render and publish — via the script only
|
||||
```
|
||||
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py render /tmp/cve-spec-<DATE>.json /tmp/cve-<DATE>.html
|
||||
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py publish /tmp/cve-<DATE>.html <DATE> cve
|
||||
```
|
||||
All layout is owned by `recipe-report.py`. Never hand-write or post-process HTML; if `render` errors,
|
||||
fix the spec JSON and re-render. **Public page — no secrets, tokens, internal hostnames, raw logs, or
|
||||
any billing/spend figures.**
|
||||
|
||||
### 8. Verify and stop
|
||||
`curl -fsS https://report.ci.commoninternet.net/cve-<DATE>.html` renders and the index lists it. Print
|
||||
the URL, a one-line summary (`N recipes swept · M with CVEs · K critical`), and `CVE CHECK COMPLETE`,
|
||||
then go idle. One-shot — do not loop, and do not start upgrading anything.
|
||||
|
||||
## Guardrails
|
||||
- **Read-only.** No PRs, no edits, no merges, no deploys, no CI runs. If a recipe looks urgent, say so
|
||||
in the report — do not act on it. `/cve-check-and-upgrade` is the skill that acts.
|
||||
- **Never report `0` for something you could not scan.** `0` means checked-and-clean; unknown is `?`.
|
||||
- A count with undetermined advisories is a **floor** — say so in the notes rather than rounding away.
|
||||
- **Public-safe output only.**
|
||||
@@ -312,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**:
|
||||
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
|
||||
stale test; operator to decide).
|
||||
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
|
||||
0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
|
||||
it is written against the failures this pipeline has actually produced. The two that matter most
|
||||
here: **set state up through the app's own interface, never its database** (a plausible fixture
|
||||
that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
|
||||
and **gate on version rather than writing a fixture that supports both** — old-version tests can
|
||||
simply be deleted, since the older version is only exercised through the upgrade tier.
|
||||
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
|
||||
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
|
||||
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
|
||||
|
||||
@@ -106,6 +106,12 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||||
1. <finding> → /<skill> (or operator action)
|
||||
…
|
||||
```
|
||||
When a finding is that the fleet's **security exposure is unknown** — the last weekly run failed or
|
||||
is stale, so nobody has scanned for CVEs recently — the recommended step is **`/cve-check`** (read-only,
|
||||
minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend
|
||||
**`/cve-check-and-upgrade`** (add `--min-severity high` when only the urgent ones matter). Prefer
|
||||
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||||
|
||||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
|
||||
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||||
|
||||
@@ -85,6 +85,14 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
||||
|
||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||
|
||||
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
|
||||
> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
|
||||
> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
|
||||
> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
|
||||
> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
|
||||
> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
|
||||
> wait but **never the assertion**.
|
||||
|
||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||
Swarm). For each `STALE_TESTS` entry:
|
||||
|
||||
|
||||
@@ -31,6 +31,14 @@ Then present the roster grouped as follows, and close with the situation guide.
|
||||
PR). `--with-tests` also fixes that recipe's stale test.
|
||||
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
|
||||
|
||||
**Security (CVEs)**
|
||||
- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
|
||||
upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
|
||||
Read-only and quick — the "what are we exposed to?" answer without an upgrade run.
|
||||
- **/cve-check-and-upgrade** — the same sweep, then open verified PRs **only** for the recipes whose
|
||||
upgrade actually fixes a CVE, worst severity first. `--min-severity high` for just the urgent ones.
|
||||
Never merges.
|
||||
|
||||
**Tests**
|
||||
- **/cc-ci-tests-update** — fleet-wide stale-test cleanup: find tests broken by legitimate
|
||||
upstream changes, fix without weakening, verify, merge the test PRs.
|
||||
@@ -73,6 +81,9 @@ ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-ccte
|
||||
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
|
||||
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
|
||||
| "The report site is stale/missing a week" | `/recipe-report` |
|
||||
| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
|
||||
| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
|
||||
| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
|
||||
| "Tests are red because upstream changed" | `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <r> --with-tests` |
|
||||
| "A CI run failed and I don't know why" | `/ci-test-review` |
|
||||
| "Update the CI server OS/deps" | `/cc-ci-server-update` |
|
||||
|
||||
+91
-15
@@ -164,6 +164,21 @@ def _vkey(v: str | None) -> tuple:
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool:
|
||||
"""Does a patched version on the TARGET's own release line sit ABOVE the target?
|
||||
|
||||
Projects maintain several branches at once and backport per branch, so "some patched version is
|
||||
inside the numeric window" is not the same as "the version we land on has the fix". ClickHouse
|
||||
fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20; an upgrade landing on 23.10.4.25 crosses the
|
||||
23.9 fix numerically but is still BELOW its own line's fix, so it does NOT have it. When the
|
||||
advisory names a fix on the target's own line and the target is older than it, that is proof of
|
||||
absence and outranks any other candidate."""
|
||||
if len(kt) < 2:
|
||||
return False
|
||||
line = kt[:2]
|
||||
return any(c[:2] == line and c > kt for c in cands if len(c) >= 2)
|
||||
|
||||
|
||||
def _within(kf: tuple, kt: tuple, c: tuple) -> bool:
|
||||
"""Is patched-version `c` inside the window (kf, kt] — exclusive lower, inclusive upper?
|
||||
|
||||
@@ -333,6 +348,19 @@ def _releases(owner: str, repo: str, max_pages: int = 4) -> list[tuple[str, str]
|
||||
return out
|
||||
|
||||
|
||||
def _source_repo(source: str) -> tuple[str, str] | None:
|
||||
"""(owner, repo) for a source, whether it is an advisory feed or a vendor page on GitHub.
|
||||
|
||||
A CVE that appears ONLY on a vendor page still deserves the release-note method when that page
|
||||
lives on GitHub — mailu announces its Roundcube CVEs on github.com/Mailu/Mailu/releases and
|
||||
nowhere structured, so requiring an advisory feed sent a deterministic case to pass 2."""
|
||||
if source.startswith("github-advisories:"):
|
||||
owner, _, repo = source.split(":", 1)[1].partition("/")
|
||||
return (owner, repo) if owner and repo else None
|
||||
m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", source)
|
||||
return (m.group(1), m.group(2).removesuffix(".git")) if m else None
|
||||
|
||||
|
||||
def release_fix_versions(source: str, cve: str) -> list[str]:
|
||||
"""Release tags whose notes NAME this CVE — a deterministic fix version when the advisory has none.
|
||||
|
||||
@@ -342,10 +370,10 @@ def release_fix_versions(source: str, cve: str) -> list[str]:
|
||||
(CVE-2025-32023 → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Ignoring that evidence undercounted
|
||||
discourse by 12 CVEs, so this is checked BEFORE giving up on an advisory.
|
||||
"""
|
||||
if not source.startswith("github-advisories:"):
|
||||
ref = _source_repo(source)
|
||||
if not ref:
|
||||
return []
|
||||
owner, _, repo = source.split(":", 1)[1].partition("/")
|
||||
return [tag for tag, body in _releases(owner, repo) if cve in body]
|
||||
return [tag for tag, body in _releases(*ref) if cve in body]
|
||||
|
||||
|
||||
def advisory_text(ghsa: str, source: str | None = None) -> dict:
|
||||
@@ -652,6 +680,10 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
continue
|
||||
patched = e.get("patched") or ""
|
||||
cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
|
||||
if kf and kt and _superseded_on_target_line(kt, cands):
|
||||
# The target's own line got the fix LATER than the target: not fixed here.
|
||||
e.setdefault("classification", "outside-window")
|
||||
continue
|
||||
if kf and kt and any(_within(kf, kt, c) for c in cands):
|
||||
got.add(cve)
|
||||
elif not patched or PLACEHOLDER_RE.search(patched):
|
||||
@@ -707,17 +739,31 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
# THIRD METHOD: before declaring an advisory undecidable, look for the CVE id in the project's
|
||||
# own release notes. A tag that names it, inside the window, IS the fix version the advisory
|
||||
# failed to publish. Deterministic and citable — not a judgement call.
|
||||
resolved_by_release = {}
|
||||
for cve in sorted(indeterminate - fixed_set):
|
||||
# A window is keyed by advisory-feed source; map it to its repo so a vendor page on the SAME
|
||||
# repo can be judged by the same window.
|
||||
win_by_repo = {}
|
||||
for wsrc, wv in windows.items():
|
||||
ref = _source_repo(wsrc)
|
||||
if ref:
|
||||
win_by_repo[ref] = wv
|
||||
resolved_by_release, already_fixed = {}, set()
|
||||
candidates = set(indeterminate) | {
|
||||
c for c in unknown
|
||||
if not any(s in windows for s in report["cves"][c]["sources"])
|
||||
and any(_source_repo(s) in win_by_repo for s in report["cves"][c]["sources"])
|
||||
}
|
||||
for cve in sorted(candidates - fixed_set):
|
||||
e = report["cves"][cve]
|
||||
for src in e["sources"]:
|
||||
if src not in windows:
|
||||
ref = _source_repo(src)
|
||||
if src not in windows and ref not in win_by_repo:
|
||||
continue
|
||||
wf, wt = windows[src]
|
||||
wf, wt = windows[src] if src in windows else win_by_repo[ref]
|
||||
kf, kt = _vkey(wf), _vkey(wt)
|
||||
if not (kf and kt):
|
||||
continue
|
||||
hits = [t for t in release_fix_versions(src, cve) if _within(kf, kt, _vkey(t))]
|
||||
naming = release_fix_versions(src, cve)
|
||||
hits = [t for t in naming if _within(kf, kt, _vkey(t))]
|
||||
if hits:
|
||||
fixed_set.add(cve)
|
||||
resolved_by_release[cve] = sorted(hits)
|
||||
@@ -725,10 +771,23 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
f"{', '.join(sorted(hits))}) via {src}")
|
||||
e["fix_versions_from_release_notes"] = sorted(hits)
|
||||
break
|
||||
# Naming releases exist but ALL predate the version we were already on: the fix shipped
|
||||
# before this upgrade, so the upgrade did not deliver it. That is a DECISION, not an
|
||||
# unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 advisories all fixed by 8.6.3 or
|
||||
# earlier, and reporting them as "could not judge" overstates the uncertainty.
|
||||
if naming and all(_vkey(t) and not _within(kf, kt, _vkey(t)) for t in naming) \
|
||||
and max(_vkey(t) for t in naming if _vkey(t)) <= kf:
|
||||
e["classification"] = ("outside-window: fixed in "
|
||||
f"{', '.join(sorted(naming))}, all at or before {wf}")
|
||||
e["fix_versions_from_release_notes"] = sorted(naming)
|
||||
already_fixed.add(cve)
|
||||
break
|
||||
if resolved_by_release:
|
||||
report["resolved_by_release_notes"] = resolved_by_release
|
||||
|
||||
indeterminate -= fixed_set
|
||||
indeterminate -= fixed_set | already_fixed
|
||||
if already_fixed:
|
||||
report["already_fixed_before_upgrade"] = sorted(already_fixed)
|
||||
for cve in indeterminate:
|
||||
report["cves"][cve]["classification"] = "indeterminate: no fix version published"
|
||||
unknown = [c for c in unknown if c not in fixed_set]
|
||||
@@ -738,8 +797,19 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
report["unclassified"] = sorted(unknown)
|
||||
# NEVER report 0 for something we could not determine — a 0 asserts safety. If ANY requested
|
||||
# window could not be ordered at all, the total is UNKNOWN rather than a partial number.
|
||||
report["count_known"] = not unresolved_any
|
||||
report["cve_count_fixed"] = len(fixed_set) if not unresolved_any else None
|
||||
# A scan with NO usable source has not measured anything, so it must not report a number —
|
||||
# least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and
|
||||
# still produced "0 identified", which was then published as a clean 0 in a CVE report.
|
||||
usable_sources = [
|
||||
s for s in report["sources"]
|
||||
if s["status"] == "ok" or s["status"].startswith("no-advisories-published")
|
||||
]
|
||||
no_sources = not usable_sources
|
||||
if no_sources:
|
||||
report["no_usable_sources"] = True
|
||||
report["count_known"] = not unresolved_any and not no_sources
|
||||
report["cve_count_fixed"] = (len(fixed_set)
|
||||
if (not unresolved_any and not no_sources) else None)
|
||||
report["cve_count_total_seen"] = len(report["cves"])
|
||||
# Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
|
||||
# no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
|
||||
@@ -762,10 +832,16 @@ def markdown(rep: dict) -> str:
|
||||
f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
|
||||
if not rep.get("count_known", True):
|
||||
L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
|
||||
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes numeric "
|
||||
"ordering meaningless across this jump, so no advisory could be classified. Render "
|
||||
"this recipe's cve cell as `?`, never `0`. Read the vendor's release notes for the "
|
||||
"jump and count by hand.")
|
||||
if rep.get("no_usable_sources"):
|
||||
L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry "
|
||||
"file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so "
|
||||
"nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add "
|
||||
"the registry file.")
|
||||
else:
|
||||
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes "
|
||||
"numeric ordering meaningless across this jump, so no advisory could be "
|
||||
"classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's "
|
||||
"release notes for the jump and count by hand.")
|
||||
if rep["unclassified"]:
|
||||
L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
|
||||
f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))
|
||||
|
||||
Executable
+238
@@ -0,0 +1,238 @@
|
||||
#!/usr/bin/env python3
|
||||
"""audit-sources — are we still looking in the right place for each recipe's updates?
|
||||
|
||||
A recipe tracks an image repo and a set of registry URLs. Upstreams move: they rename the image,
|
||||
switch registry, archive the GitHub repo, or split a community edition out of the original. When that
|
||||
happens nothing errors — the old repo simply stops receiving tags, and the recipe looks "up to date"
|
||||
forever while real releases happen somewhere else.
|
||||
|
||||
plausible is the worked example. It tracked `plausible/analytics` on Docker Hub; upstream moved to
|
||||
`ghcr.io/plausible/community-edition`. The old repo still exists and still serves v2.0.0, so every
|
||||
survey said "no upgrades available" while v3 shipped elsewhere.
|
||||
|
||||
This reports the signals that catch that, per image and per registry URL:
|
||||
|
||||
* IMAGE GONE QUIET — newest tag is older than --quiet-days (default 365). The single strongest
|
||||
signal that releases moved somewhere else.
|
||||
* DEPRECATION WORDING — the registry description says deprecated / moved / no longer maintained.
|
||||
* GITHUB REPO ARCHIVED — upstream archived it.
|
||||
* GITHUB REPO RENAMED — the API redirects to a different owner/name than we ask for.
|
||||
* GITHUB REPO GONE — 404.
|
||||
|
||||
Everything is a SIGNAL, not a verdict: a genuinely stable image (mumble, custom-html) can be quiet
|
||||
for good reason. The output is for a human to judge, so each finding says what was measured.
|
||||
|
||||
audit-sources.py [recipe ...] [--ssh HOST] [--quiet-days N] [--json]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
_spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HERE, "resolve-images.py"))
|
||||
RI = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(RI)
|
||||
|
||||
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream"))
|
||||
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
|
||||
DEPRECATION_RE = re.compile(
|
||||
r"\b(deprecat|no longer maintain|unmaintained|superseded|moved to|migrated to|"
|
||||
r"has moved|discontinued|end.of.life|archived)\b", re.I)
|
||||
|
||||
|
||||
def _days_since(iso: str | None) -> int | None:
|
||||
if not iso:
|
||||
return None
|
||||
try:
|
||||
d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
return (datetime.now(timezone.utc) - d).days
|
||||
|
||||
|
||||
def hub_repo_meta(repo: str) -> dict:
|
||||
"""Docker Hub repo metadata: when it was last pushed to, and how it describes itself."""
|
||||
try:
|
||||
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}", RI._hub_auth())
|
||||
except urllib.error.HTTPError as e:
|
||||
return {"status": f"HTTP {e.code}"}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"status": f"{type(e).__name__}"}
|
||||
text = f"{d.get('description') or ''}\n{d.get('full_description') or ''}"
|
||||
m = DEPRECATION_RE.search(text)
|
||||
return {"status": "ok", "last_updated": d.get("last_updated"),
|
||||
"deprecation_hint": (m.group(0) if m else None),
|
||||
"archived": bool(d.get("is_archived") or d.get("status") == "inactive")}
|
||||
|
||||
|
||||
def github_repo_meta(owner: str, repo: str) -> dict:
|
||||
"""GitHub repo state — archived, renamed (the API answers with the CURRENT full_name), or gone."""
|
||||
hdrs = {"Accept": "application/vnd.github+json"}
|
||||
tok = RI._gh_token()
|
||||
if tok:
|
||||
hdrs["Authorization"] = f"Bearer {tok}"
|
||||
try:
|
||||
d = RI._json(f"https://api.github.com/repos/{owner}/{repo}", hdrs)
|
||||
except urllib.error.HTTPError as e:
|
||||
return {"status": f"HTTP {e.code}"}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"status": f"{type(e).__name__}"}
|
||||
asked, got = f"{owner}/{repo}".lower(), (d.get("full_name") or "").lower()
|
||||
return {"status": "ok", "archived": bool(d.get("archived")), "pushed_at": d.get("pushed_at"),
|
||||
"renamed_to": (d.get("full_name") if got and got != asked else None),
|
||||
"description": d.get("description") or ""}
|
||||
|
||||
|
||||
def newest_tag_date(registry: str, repo: str, tag: str) -> str | None:
|
||||
"""When was the repo's newest same-shape tag pushed? Docker Hub only (it dates its tags)."""
|
||||
if registry not in ("docker.io", "registry-1.docker.io"):
|
||||
return None
|
||||
try:
|
||||
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}/tags"
|
||||
f"?page_size=100&ordering=last_updated", RI._hub_auth())
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
want = RI.shape(tag)
|
||||
for row in d.get("results", []):
|
||||
if RI.shape(row.get("name") or "") == want:
|
||||
return row.get("last_updated")
|
||||
return (d.get("results") or [{}])[0].get("last_updated")
|
||||
|
||||
|
||||
def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
|
||||
out = {"recipe": recipe, "findings": [], "images": [], "sources": []}
|
||||
try:
|
||||
refs = (RI.compose_images_ssh(recipe, ssh, "~/.abra/recipes") if ssh
|
||||
else RI.compose_images(recipe, RI.RECIPE_DIR))
|
||||
except Exception as e: # noqa: BLE001
|
||||
out["findings"].append({"level": "error", "what": f"could not read compose: {e}"})
|
||||
return out
|
||||
|
||||
for ref in refs:
|
||||
if "${" in ref:
|
||||
continue
|
||||
info = RI.parse_ref(ref)
|
||||
row = {"ref": ref, "registry": info["registry"], "repo": info["repo"], "tag": info["tag"]}
|
||||
if info["registry"] in ("docker.io", "registry-1.docker.io"):
|
||||
meta = hub_repo_meta(info["repo"])
|
||||
row.update(meta)
|
||||
newest = newest_tag_date(info["registry"], info["repo"], info["tag"])
|
||||
row["newest_tag_pushed"] = newest
|
||||
age = _days_since(newest)
|
||||
row["newest_tag_age_days"] = age
|
||||
if age is not None and age > quiet_days:
|
||||
out["findings"].append({
|
||||
"level": "warn", "what": "image has gone quiet",
|
||||
"detail": f"{info['repo']}: newest {RI.shape(info['tag'])}-shaped tag pushed "
|
||||
f"{age} days ago — releases may have moved elsewhere"})
|
||||
if meta.get("deprecation_hint"):
|
||||
out["findings"].append({
|
||||
"level": "warn", "what": "registry text suggests deprecation",
|
||||
"detail": f"{info['repo']}: says {meta['deprecation_hint']!r}"})
|
||||
if meta.get("archived"):
|
||||
out["findings"].append({"level": "warn", "what": "registry repo archived/inactive",
|
||||
"detail": info["repo"]})
|
||||
out["images"].append(row)
|
||||
|
||||
urls, reg_path = ([], None)
|
||||
try:
|
||||
urls, reg_path = _registry_urls(recipe)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
if reg_path is None:
|
||||
out["findings"].append({"level": "warn", "what": "no upstream registry file",
|
||||
"detail": f"cc-ci-plan/upstream/{recipe}.md is missing — the advisory "
|
||||
f"scan has nowhere to look"})
|
||||
seen = set()
|
||||
for u in urls:
|
||||
m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u)
|
||||
if not m:
|
||||
continue
|
||||
owner, repo = m.group(1), m.group(2).removesuffix(".git")
|
||||
if (owner, repo) in seen:
|
||||
continue
|
||||
seen.add((owner, repo))
|
||||
meta = github_repo_meta(owner, repo)
|
||||
row = {"repo": f"{owner}/{repo}", **meta}
|
||||
age = _days_since(meta.get("pushed_at"))
|
||||
row["pushed_age_days"] = age
|
||||
out["sources"].append(row)
|
||||
if meta.get("status") != "ok":
|
||||
out["findings"].append({"level": "warn", "what": "registry source unreachable",
|
||||
"detail": f"{owner}/{repo}: {meta['status']}"})
|
||||
continue
|
||||
if meta.get("renamed_to"):
|
||||
out["findings"].append({"level": "alert", "what": "GitHub repo has MOVED",
|
||||
"detail": f"{owner}/{repo} now answers as {meta['renamed_to']}"})
|
||||
if meta.get("archived"):
|
||||
out["findings"].append({"level": "alert", "what": "GitHub repo is ARCHIVED",
|
||||
"detail": f"{owner}/{repo} — upstream development has stopped here"})
|
||||
if age is not None and age > quiet_days:
|
||||
out["findings"].append({"level": "warn", "what": "GitHub repo quiet",
|
||||
"detail": f"{owner}/{repo}: last push {age} days ago"})
|
||||
return out
|
||||
|
||||
|
||||
def _registry_urls(recipe: str):
|
||||
path = os.path.join(REGISTRY_DIR, f"{recipe}.md")
|
||||
if not os.path.exists(path):
|
||||
return [], None
|
||||
text = open(path).read()
|
||||
urls = []
|
||||
for u in re.findall(r"https?://[^\s)|\]]+", text):
|
||||
u = u.rstrip("`'\"*.,;:>)")
|
||||
if u and u not in urls:
|
||||
urls.append(u)
|
||||
return urls, path
|
||||
|
||||
|
||||
def all_recipes() -> list[str]:
|
||||
out = []
|
||||
for ln in open(USED_RECIPES):
|
||||
ln = ln.strip()
|
||||
if not ln or ln.startswith("#") or ln.startswith("`"):
|
||||
continue
|
||||
parts = ln.split()
|
||||
if len(parts) >= 2 and parts[1] in ("weekly", "external"):
|
||||
out.append(parts[0])
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("recipes", nargs="*")
|
||||
ap.add_argument("--ssh", default=None)
|
||||
ap.add_argument("--quiet-days", type=int, default=365)
|
||||
ap.add_argument("--json", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
recipes = a.recipes or all_recipes()
|
||||
reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes]
|
||||
if a.json:
|
||||
print(json.dumps(reports, indent=2))
|
||||
return 0
|
||||
alerts = 0
|
||||
for rep in reports:
|
||||
fs = rep["findings"]
|
||||
mark = "OK " if not fs else ("!! " if any(f["level"] == "alert" for f in fs) else " ? ")
|
||||
print(f"{mark} {rep['recipe']}")
|
||||
for f in fs:
|
||||
alerts += f["level"] == "alert"
|
||||
print(f" [{f['level']}] {f['what']}: {f.get('detail','')}")
|
||||
print(f"\n{len(reports)} recipes audited · "
|
||||
f"{sum(len(r['findings']) for r in reports)} findings · {alerts} alerts")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+45
-14
@@ -9,7 +9,11 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica
|
||||
survey [DATE] JSON of the run + every recipe's open PRs + CI verdict + per-recipe upgrade
|
||||
notes (breaking-change/CVE analysis), and the /upgrade-all summary.
|
||||
render SPEC.json OUT.html render the agent's report spec -> a self-contained newspaper HTML page
|
||||
publish OUT.html DATE copy to cc-ci:/var/lib/cc-ci-reports/week-DATE.html and regen the archive index
|
||||
publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
|
||||
archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
|
||||
(a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
|
||||
newest first, each row suffixed "full" or "CVE check"; the distinct
|
||||
filename prefix just stops a sweep overwriting a weekly edition.
|
||||
|
||||
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
|
||||
Bulletin → per-recipe "What changed".
|
||||
@@ -44,6 +48,10 @@ LOGDIR = "/srv/cc-ci/.cc-ci-logs"
|
||||
TESTENV = "/srv/cc-ci/.testenv"
|
||||
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
|
||||
HOST_REPORTS = "/var/lib/cc-ci-reports"
|
||||
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
|
||||
# prefix -> (page title, index label)
|
||||
KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),
|
||||
"cve": ("The Recipe Report — CVE check", "{d} — CVE check")}
|
||||
|
||||
|
||||
def _env():
|
||||
@@ -215,8 +223,16 @@ def _table(rows, repo_url=None):
|
||||
if repo_url and r.get("recipe") in repo_url:
|
||||
name = f'<a href="{repo_url[r["recipe"]]}">{name}</a>'
|
||||
cve = r.get("cve")
|
||||
cve_cell = (f'<span class="cve">{int(cve)}</span>' if isinstance(cve, (int, float)) and cve
|
||||
else '<span class="muted">none</span>')
|
||||
# "?" = advisory scan absent or had failed sources → count NOT authoritative. Per the
|
||||
# /recipe-report guardrail this must NEVER render as "none" (a blank-that-reads-clean is
|
||||
# exactly how two CVSS-9.8 gitea RCEs were misreported as "none" on 2026-08-07). A positive
|
||||
# int is the confirmed CVE count; 0/omit is a confirmed-clean scan.
|
||||
if isinstance(cve, str) and cve.strip() == "?":
|
||||
cve_cell = '<span class="muted" title="advisory scan incomplete or absent — CVE count unknown">?</span>'
|
||||
elif isinstance(cve, (int, float)) and cve:
|
||||
cve_cell = f'<span class="cve">{int(cve)}</span>'
|
||||
else:
|
||||
cve_cell = '<span class="muted">none</span>'
|
||||
ci = _esc(r.get("ci"))
|
||||
if r.get("ci_url"):
|
||||
ci = f'<a href="{_esc(r["ci_url"])}">{ci}</a>'
|
||||
@@ -270,8 +286,10 @@ def _mast():
|
||||
|
||||
def render(spec_path, out_path):
|
||||
s = json.load(open(spec_path))
|
||||
kind = s.get("kind", "week")
|
||||
title = KINDS.get(kind, KINDS["week"])[0]
|
||||
gen = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
sub = s.get("subtitle", "Week of " + s["date"])
|
||||
sub = s.get("subtitle", ("Week of " if kind == "week" else "CVE check ") + s["date"])
|
||||
lead = s.get("lead", "") or ""
|
||||
# Auto-link recipe-name mentions in the lead to their mirror repos.
|
||||
gitea = _env().get("GITEA_URL", "git.autonomic.zone")
|
||||
@@ -285,7 +303,9 @@ def render(spec_path, out_path):
|
||||
f'<span>report.ci.commoninternet.net</span><span>{gen}</span></div>'
|
||||
f'<div class="lead">{lead}</div>')
|
||||
# 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column.
|
||||
body += f'<h2>The full wire — every recipe, in priority order</h2>{_table(s.get("table"), repo_url)}'
|
||||
wire = ("The full wire — every recipe, in priority order" if kind == "week"
|
||||
else "Advisory sweep — every recipe, worst first")
|
||||
body += f'<h2>{wire}</h2>{_table(s.get("table"), repo_url)}'
|
||||
# 2) addendum — special issues to look into (normal-size header); omitted entirely if there are none.
|
||||
add = [a for a in (s.get("addendum") or []) if str(a).strip()]
|
||||
if add:
|
||||
@@ -298,19 +318,30 @@ def render(spec_path, out_path):
|
||||
# 4) what changed — a short section per recipe that has a PR
|
||||
if s.get("changes"):
|
||||
body += f'<h2>What changed</h2>{_changes(s.get("changes"), repo_url)}'
|
||||
body += (f'<footer>The Recipe Report · generated {gen} · '
|
||||
body += (f'<footer>{title} · generated {gen} · '
|
||||
f'<a href="https://ci.commoninternet.net/">dashboard</a> · <a href="./">archive</a></footer>')
|
||||
open(out_path, "w").write(_page("The Recipe Report — " + s["date"], body))
|
||||
open(out_path, "w").write(_page(f"{title} · " + s["date"], body))
|
||||
print("wrote", out_path)
|
||||
|
||||
|
||||
def publish(html_path, date):
|
||||
page = f"week-{date}.html"
|
||||
def publish(html_path, date, kind="week"):
|
||||
if kind not in KINDS:
|
||||
print(f"unknown kind {kind!r}; expected one of {', '.join(KINDS)}"); sys.exit(2)
|
||||
page = f"{kind}-{date}.html"
|
||||
subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/{page}"], input=open(html_path, "rb").read(), check=True)
|
||||
listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/week-*.html 2>/dev/null"],
|
||||
capture_output=True, text=True).stdout.split()
|
||||
dates = sorted({os.path.basename(p)[5:-5] for p in listing}, reverse=True)
|
||||
lis = "\n".join(f'<li><a href="week-{d}.html">Week of {d}</a><span class="d">{d}</span></li>' for d in dates)
|
||||
# One index over BOTH families, newest first, each row labelled by its kind — an operator looking
|
||||
# for "the latest security picture" should not have to know which skill produced which page.
|
||||
entries = []
|
||||
for k in KINDS:
|
||||
listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/{k}-*.html 2>/dev/null"],
|
||||
capture_output=True, text=True).stdout.split()
|
||||
for pth in listing:
|
||||
d = os.path.basename(pth)[len(k) + 1:-5]
|
||||
if re.fullmatch(r"\d{4}-\d{2}-\d{2}", d):
|
||||
entries.append((d, k))
|
||||
lis = "\n".join(
|
||||
f'<li><a href="{k}-{d}.html">{KINDS[k][1].format(d=d)}</a><span class="d">{d}</span></li>'
|
||||
for d, k in sorted(set(entries), reverse=True))
|
||||
idx = _page("The Recipe Report — Archive", _mast() +
|
||||
'<div class="dateline"><span>Weekly review of Co-op Cloud recipe upgrades & CI</span>'
|
||||
'<span>report.ci.commoninternet.net</span></div>'
|
||||
@@ -328,7 +359,7 @@ def main():
|
||||
elif cmd == "render":
|
||||
render(a[1], a[2])
|
||||
elif cmd == "publish":
|
||||
publish(a[1], a[2])
|
||||
publish(a[1], a[2], a[3] if len(a) > 3 else "week")
|
||||
else:
|
||||
print(__doc__); sys.exit(2)
|
||||
|
||||
|
||||
@@ -0,0 +1,459 @@
|
||||
#!/usr/bin/env python3
|
||||
"""resolve-images — what version is each of a recipe's images on, and what is newest?
|
||||
|
||||
An abra-independent version resolver. `abra recipe upgrade` is the normal path, but it has a hard
|
||||
failure mode: an image pinned with BOTH a tag and a digest makes it FATA and abandon the WHOLE
|
||||
recipe — even images it already parsed. immich pins two that way:
|
||||
|
||||
ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf6…
|
||||
docker.io/valkey/valkey:9@sha256:3acc…
|
||||
|
||||
so immich contributes NO version data at all and silently drops out of every survey. That is
|
||||
indistinguishable from "up to date" unless a human notices the missing row — which is exactly how it
|
||||
kept getting skipped, and why a CVE sweep reported it as unknown.
|
||||
|
||||
This reads the compose files directly and queries the registries itself, so a digest pin is just a
|
||||
digest pin. Output is JSON (default) or a table.
|
||||
|
||||
resolve-images.py <recipe> [--ssh HOST] [--recipe-dir DIR] [--table] [--only IMAGE]
|
||||
|
||||
The cc-ci host has no python3, so `--ssh cc-ci` reads the compose files from that host's checkout
|
||||
over ssh and does the resolving locally. That keeps the source of truth the SAME tree abra and CI
|
||||
use, rather than a second copy that can drift.
|
||||
|
||||
TAG SHAPES. Registries mix wildly different tag conventions in one repo, so "newest" is meaningless
|
||||
without a shape. Each tag is reduced to a signature by replacing digit runs with '#':
|
||||
|
||||
v3.1.0 -> v#.#.#
|
||||
1.27.1-rootless -> #.#.#-rootless
|
||||
8.10-alpine -> #.#-alpine
|
||||
14-vectorchord0.4.3-pgvectors0.2.0 -> #-vectorchord#.#.#-pgvectors#.#.#
|
||||
|
||||
Only tags sharing the CURRENT pin's shape are candidates. That keeps `-alpine` on `-alpine`, and
|
||||
stops a `latest`/`release`/`sha-…` tag from ever being proposed as an upgrade.
|
||||
|
||||
TWO ANSWERS, NOT ONE. It reports `newest_same_shape` AND `newest_within_major` (same leading number).
|
||||
For a plain app image they usually agree. For a compatibility-pinned sidecar they do not, and taking
|
||||
the max would be wrong: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
|
||||
versions that immich-server is built against, so jumping pg major because a newer tag exists breaks
|
||||
the deployment. The caller picks; this tool refuses to guess and shows both.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import shlex
|
||||
import subprocess
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
UA = "cc-ci-resolve-images (+https://git.autonomic.zone/recipe-maintainers/cc-ci)"
|
||||
TIMEOUT = int(os.environ.get("RESOLVE_IMAGES_TIMEOUT", "45"))
|
||||
RECIPE_DIR = os.environ.get("ABRA_RECIPE_DIR", os.path.expanduser("~/.abra/recipes"))
|
||||
MAX_TAG_PAGES = int(os.environ.get("RESOLVE_IMAGES_MAX_PAGES", "40"))
|
||||
|
||||
IMAGE_RE = re.compile(r"""^\s*image:\s*["']?([^"'\s]+)["']?\s*$""", re.M)
|
||||
|
||||
|
||||
RETRIES = int(os.environ.get("RESOLVE_IMAGES_RETRIES", "4"))
|
||||
|
||||
|
||||
def _fetch(url: str, headers: dict | None = None) -> bytes:
|
||||
"""GET with backoff on rate limits.
|
||||
|
||||
Docker Hub throttles anonymous clients hard, and a sweep re-reads the same popular repos
|
||||
(nginx, redis, postgres) for recipe after recipe. A 429 mid-sweep used to surface as
|
||||
'unresolved', which is indistinguishable from a real lookup failure — so retry, and let the
|
||||
per-repo cache below remove most of the requests entirely."""
|
||||
h = {"User-Agent": UA, "Accept-Encoding": "gzip"}
|
||||
h.update(headers or {})
|
||||
delay = 2.0
|
||||
for attempt in range(RETRIES):
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r:
|
||||
raw = r.read()
|
||||
if r.headers.get("Content-Encoding") == "gzip":
|
||||
raw = gzip.decompress(raw)
|
||||
return raw
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code in (429, 503) and attempt < RETRIES - 1:
|
||||
time.sleep(delay)
|
||||
delay *= 2
|
||||
continue
|
||||
raise
|
||||
raise RuntimeError("unreachable")
|
||||
|
||||
|
||||
_HUB_JWT: list = []
|
||||
|
||||
|
||||
def _hub_auth() -> dict:
|
||||
"""Authenticated Docker Hub calls get a far higher rate limit than anonymous ones.
|
||||
|
||||
Credentials come from /srv/cc-ci/.testenv (DOCKERHUB_USERNAME / DOCKERHUB_TOKEN), the same pair
|
||||
the CI host already uses. Absent creds are fine — the sweep just runs anonymous and slower."""
|
||||
if _HUB_JWT:
|
||||
return _HUB_JWT[0]
|
||||
env = {}
|
||||
try:
|
||||
for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
|
||||
if "=" in ln and not ln.strip().startswith("#"):
|
||||
k, v = ln.strip().split("=", 1)
|
||||
env[k] = v.strip().strip("\"'")
|
||||
except OSError:
|
||||
pass
|
||||
u = os.environ.get("DOCKERHUB_USERNAME") or env.get("DOCKERHUB_USERNAME")
|
||||
t = os.environ.get("DOCKERHUB_TOKEN") or env.get("DOCKERHUB_TOKEN")
|
||||
hdrs = {}
|
||||
if u and t:
|
||||
try:
|
||||
body = json.dumps({"username": u, "password": t}).encode()
|
||||
req = urllib.request.Request("https://hub.docker.com/v2/users/login",
|
||||
data=body, method="POST",
|
||||
headers={"Content-Type": "application/json", "User-Agent": UA})
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
tokj = json.load(r).get("token")
|
||||
if tokj:
|
||||
hdrs = {"Authorization": f"JWT {tokj}"}
|
||||
except Exception: # noqa: BLE001 — anonymous is a valid fallback
|
||||
hdrs = {}
|
||||
_HUB_JWT.append(hdrs)
|
||||
return hdrs
|
||||
|
||||
|
||||
def _json(url: str, headers: dict | None = None):
|
||||
return json.loads(_fetch(url, headers))
|
||||
|
||||
|
||||
def shape(tag: str) -> str:
|
||||
"""Signature of a tag with every digit run replaced by '#'. See module docstring."""
|
||||
return re.sub(r"\d+", "#", tag)
|
||||
|
||||
|
||||
def vkey(tag: str) -> tuple:
|
||||
"""Ordering key: every number in the tag, in order. '1.27.10' > '1.27.9'; text ignored."""
|
||||
return tuple(int(x) for x in re.findall(r"\d+", tag))
|
||||
|
||||
|
||||
def parse_ref(ref: str) -> dict:
|
||||
"""Split an image reference into registry / repo / tag / digest."""
|
||||
digest = None
|
||||
if "@" in ref:
|
||||
ref, _, digest = ref.partition("@")
|
||||
host, repo, tag = "docker.io", ref, "latest"
|
||||
# A leading component is a REGISTRY only when there is a path after it. Without the slash test,
|
||||
# a bare `postgres:15.18` looks like host "postgres:15.18" because of the tag's colon — which
|
||||
# silently sent every library image to a nonexistent registry.
|
||||
if "/" in ref:
|
||||
first = ref.split("/")[0]
|
||||
if "." in first or ":" in first or first == "localhost":
|
||||
host, _, repo = ref.partition("/")
|
||||
if ":" in repo.split("/")[-1]:
|
||||
repo, _, tag = repo.rpartition(":")
|
||||
if host == "docker.io" and "/" not in repo:
|
||||
repo = f"library/{repo}" # bare `redis` is really `library/redis`
|
||||
return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
|
||||
|
||||
|
||||
HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10"))
|
||||
|
||||
|
||||
def _hub_tag_exists(repo: str, tag: str) -> bool:
|
||||
try:
|
||||
_json(f"https://hub.docker.com/v2/repositories/{repo}/tags/{tag}", _hub_auth())
|
||||
return True
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
|
||||
|
||||
def _hub_tags(repo: str) -> list[str]:
|
||||
"""Recently-pushed tags, newest first.
|
||||
|
||||
Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical —
|
||||
but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so
|
||||
ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade.
|
||||
(ghcr offers no ordering, which is why that path needs a different strategy.)
|
||||
"""
|
||||
tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
|
||||
f"?page_size=100&ordering=last_updated")
|
||||
auth = _hub_auth()
|
||||
for _ in range(HUB_RECENT_PAGES):
|
||||
d = _json(url, auth)
|
||||
tags += [r["name"] for r in d.get("results", [])]
|
||||
url = d.get("next")
|
||||
if not url:
|
||||
break
|
||||
return tags
|
||||
|
||||
|
||||
def _oci_bearer(host: str, repo: str) -> dict:
|
||||
"""Token for an OCI registry, discovered from its own auth challenge.
|
||||
|
||||
Registries do NOT share a token endpoint. ghcr answers at /token?scope=…&service=ghcr.io, but
|
||||
lscr.io and dock.mau.dev advertise different realms, and assuming ghcr's shape made both 401 —
|
||||
which then read as "could not resolve" rather than "asked the wrong URL". The registry tells us
|
||||
where to go in its WWW-Authenticate header; use that."""
|
||||
try:
|
||||
urllib.request.urlopen(
|
||||
urllib.request.Request(f"https://{host}/v2/{repo}/tags/list?n=1",
|
||||
headers={"User-Agent": UA}), timeout=TIMEOUT)
|
||||
return {} # no auth needed
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code != 401:
|
||||
return {}
|
||||
chal = e.headers.get("WWW-Authenticate", "") or ""
|
||||
except Exception: # noqa: BLE001
|
||||
return {}
|
||||
if not chal.lower().startswith("bearer"):
|
||||
return {}
|
||||
parts = dict(re.findall(r'(\w+)="([^"]*)"', chal))
|
||||
realm = parts.get("realm")
|
||||
if not realm:
|
||||
return {}
|
||||
q = {"service": parts.get("service", host), "scope": parts.get("scope", f"repository:{repo}:pull")}
|
||||
url = realm + ("&" if "?" in realm else "?") + urllib.parse.urlencode(q)
|
||||
try:
|
||||
tok = (_json(url) or {}).get("token") or (_json(url) or {}).get("access_token")
|
||||
return {"Authorization": f"Bearer {tok}"} if tok else {}
|
||||
except Exception: # noqa: BLE001
|
||||
return {}
|
||||
|
||||
|
||||
def _oci_tags(host: str, repo: str) -> list[str]:
|
||||
"""Tags from any OCI/v2 registry, with challenge-derived auth and Link pagination.
|
||||
|
||||
ghcr paginates hard — immich-server has >40,000 tags — and a truncated listing silently hides
|
||||
the newest release line, so follow the cursor and let the caller's integrity check catch a read
|
||||
that never reached the current pin."""
|
||||
hdrs = _oci_bearer(host, repo)
|
||||
tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
|
||||
for _ in range(MAX_TAG_PAGES):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
tags += (json.load(r) or {}).get("tags") or []
|
||||
link = r.headers.get("Link", "") or ""
|
||||
m = re.search(r'<([^>]+)>;\s*rel="next"', link)
|
||||
if not m:
|
||||
break
|
||||
nxt = m.group(1)
|
||||
url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
|
||||
return tags
|
||||
|
||||
|
||||
def _gh_token() -> str | None:
|
||||
tok = os.environ.get("GITHUB_TOKEN")
|
||||
if tok:
|
||||
return tok.strip()
|
||||
try:
|
||||
return open(os.environ.get("GITHUB_TOKEN_FILE", "/srv/cc-ci/.github-token")).read().strip() or None
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def github_release_tags(owner: str, repo: str, max_pages: int = 4) -> list[str]:
|
||||
"""Release tag names for a GitHub repo, newest first.
|
||||
|
||||
FALLBACK for registries whose tag listing cannot be enumerated. ghcr has no ordering and no
|
||||
server-side filter, and immich-machine-learning carries >40,000 tags — a full read is impractical
|
||||
and a partial read silently hides the newest release line. The project's RELEASES are ordered,
|
||||
small, and authoritative: container tags track them. (The GitHub Packages API would answer this
|
||||
directly but needs a scoped token; this scan's token deliberately has none.)
|
||||
"""
|
||||
hdrs = {"Accept": "application/vnd.github+json"}
|
||||
tok = _gh_token()
|
||||
if tok:
|
||||
hdrs["Authorization"] = f"Bearer {tok}"
|
||||
out = []
|
||||
for page in range(1, max_pages + 1):
|
||||
try:
|
||||
rows = _json(f"https://api.github.com/repos/{owner}/{repo}/releases"
|
||||
f"?per_page=100&page={page}", hdrs)
|
||||
except Exception: # noqa: BLE001
|
||||
break
|
||||
if not rows:
|
||||
break
|
||||
out += [r.get("tag_name") or "" for r in rows]
|
||||
return [t for t in out if t]
|
||||
|
||||
|
||||
def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
|
||||
"""Candidate GitHub repos whose releases track this image's tags."""
|
||||
if "ghcr.io" not in registry:
|
||||
return []
|
||||
parts = repo.split("/")
|
||||
if len(parts) < 2:
|
||||
return []
|
||||
owner, name = parts[0], parts[-1]
|
||||
cands = [(owner, name)]
|
||||
# ghcr.io/immich-app/immich-machine-learning is built from immich-app/immich.
|
||||
if name.startswith(owner.split("-")[0]):
|
||||
cands.append((owner, owner.split("-")[0]))
|
||||
return cands
|
||||
|
||||
|
||||
_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
|
||||
|
||||
|
||||
def list_tags(registry: str, repo: str) -> tuple[list[str], str | None]:
|
||||
if (registry, repo) in _TAG_CACHE:
|
||||
return _TAG_CACHE[(registry, repo)]
|
||||
res = _list_tags_uncached(registry, repo)
|
||||
_TAG_CACHE[(registry, repo)] = res
|
||||
return res
|
||||
|
||||
|
||||
def _list_tags_uncached(registry: str, repo: str) -> tuple[list[str], str | None]:
|
||||
try:
|
||||
return (_hub_tags(repo) if registry in ("docker.io", "registry-1.docker.io")
|
||||
else _oci_tags(registry, repo)), None
|
||||
except urllib.error.HTTPError as e:
|
||||
return [], f"HTTP {e.code}"
|
||||
except Exception as e: # noqa: BLE001
|
||||
return [], f"{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
def resolve(ref: str) -> dict:
|
||||
"""Current pin -> newest same-shape tag, and newest within the current major."""
|
||||
if "${" in ref or "$(" in ref:
|
||||
# The tag is a compose variable (ghost pins `ghost:${IMAGE_VERSION}-alpine`). Its real value
|
||||
# lives in .env, not here. Report it as skipped, never as a failed lookup.
|
||||
return {**parse_ref(ref), "ref": ref, "shape": None, "candidates": 0,
|
||||
"newest_same_shape": None, "newest_within_major": None,
|
||||
"upgrade_available": False,
|
||||
"status": "skipped: templated ref (tag comes from a compose variable)"}
|
||||
info = parse_ref(ref)
|
||||
out = {**info, "ref": ref, "shape": shape(info["tag"]), "status": "ok",
|
||||
"newest_same_shape": None, "newest_within_major": None, "candidates": 0,
|
||||
"upgrade_available": False}
|
||||
tags, err = list_tags(info["registry"], info["repo"])
|
||||
if err:
|
||||
out["status"] = f"error: {err}"
|
||||
return out
|
||||
out["tags_seen"] = len(set(tags))
|
||||
# INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
|
||||
# the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
|
||||
# of thousands of tags and a truncated read silently hides whole release lines. immich's
|
||||
# machine-learning image is pinned v3.1.0, which EXISTS, yet a short read reported v1.134.0 as
|
||||
# newest; without this check that becomes a confident, wrong answer.
|
||||
if info["tag"] not in set(tags):
|
||||
# Docker Hub: the window is recency-ordered, so the pin being outside it just means the pin
|
||||
# is old — which is fine, because anything NEWER is necessarily inside the window. Confirm
|
||||
# the pin genuinely exists (so a typo is still caught) and carry on.
|
||||
if info["registry"] in ("docker.io", "registry-1.docker.io") and _hub_tag_exists(info["repo"], info["tag"]):
|
||||
out["source"] = f"docker-hub:recent-{HUB_RECENT_PAGES * 100}"
|
||||
tags = list(tags) + [info["tag"]]
|
||||
else:
|
||||
for owner, name in _release_fallback_repos(info["registry"], info["repo"]):
|
||||
rel = github_release_tags(owner, name)
|
||||
if info["tag"] in rel:
|
||||
tags = rel
|
||||
out["source"] = f"github-releases:{owner}/{name}"
|
||||
out["tags_seen"] = len(set(rel))
|
||||
break
|
||||
else:
|
||||
out["status"] = ("error: tag listing incomplete — the current pin "
|
||||
f"{info['tag']!r} is absent from {len(set(tags))} registry tags "
|
||||
f"and from the project's GitHub releases")
|
||||
return out
|
||||
want, cur = out["shape"], vkey(info["tag"])
|
||||
same = [t for t in set(tags) if shape(t) == want and vkey(t)]
|
||||
out["candidates"] = len(same)
|
||||
if not same:
|
||||
# Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings.
|
||||
out["status"] = "no comparable tags (shape has no numeric siblings)"
|
||||
return out
|
||||
newest = max(same, key=vkey)
|
||||
out["newest_same_shape"] = newest
|
||||
if cur:
|
||||
within = [t for t in same if vkey(t)[:1] == cur[:1]]
|
||||
if within:
|
||||
out["newest_within_major"] = max(within, key=vkey)
|
||||
out["upgrade_available"] = bool(cur and vkey(newest) > cur)
|
||||
return out
|
||||
|
||||
|
||||
def compose_images_ssh(recipe: str, host: str, recipe_dir: str) -> list[str]:
|
||||
"""Same as compose_images, but the recipe tree lives on another host (cc-ci has no python3)."""
|
||||
# NB: no shell-quoting of the directory — it may legitimately start with ~ or $HOME, and
|
||||
# quoting it stops the remote shell expanding it, which yields an empty (and silent) result.
|
||||
d = f"{recipe_dir}/{shlex.quote(recipe)}".replace("~", "$HOME")
|
||||
cmd = (f'for f in {d}/compose*.yml; do case "$f" in *compose.ccci.yml) continue;; esac; '
|
||||
f'[ -f "$f" ] && {{ cat "$f"; echo; }}; done; exit 0')
|
||||
out = subprocess.run(["ssh", host, cmd], capture_output=True, text=True, timeout=120)
|
||||
if out.returncode != 0:
|
||||
raise RuntimeError(f"ssh {host}: {(out.stderr.strip() or 'no output')[:200]}")
|
||||
if not out.stdout.strip():
|
||||
raise RuntimeError(f"ssh {host}: no compose files found under {d}")
|
||||
refs = []
|
||||
for m in IMAGE_RE.finditer(out.stdout):
|
||||
if m.group(1) not in refs:
|
||||
refs.append(m.group(1))
|
||||
return refs
|
||||
|
||||
|
||||
def compose_images(recipe: str, recipe_dir: str) -> list[str]:
|
||||
"""Every `image:` ref in the recipe's own compose files (the cc-ci overlay is NOT the recipe)."""
|
||||
refs, base = [], os.path.join(recipe_dir, recipe)
|
||||
for path in sorted(glob.glob(os.path.join(base, "compose*.yml"))):
|
||||
if os.path.basename(path) == "compose.ccci.yml":
|
||||
continue
|
||||
try:
|
||||
for m in IMAGE_RE.finditer(open(path).read()):
|
||||
if m.group(1) not in refs:
|
||||
refs.append(m.group(1))
|
||||
except OSError:
|
||||
continue
|
||||
return refs
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("recipe")
|
||||
ap.add_argument("--ssh", default=None, metavar="HOST",
|
||||
help="read the recipe's compose files from HOST over ssh (e.g. --ssh cc-ci); "
|
||||
"resolving still happens locally")
|
||||
ap.add_argument("--recipe-dir", default=RECIPE_DIR)
|
||||
ap.add_argument("--table", action="store_true", help="human-readable table instead of JSON")
|
||||
ap.add_argument("--only", default=None, help="resolve just the images whose ref contains this")
|
||||
a = ap.parse_args()
|
||||
|
||||
rdir = a.recipe_dir if a.recipe_dir != RECIPE_DIR or not a.ssh else "~/.abra/recipes"
|
||||
refs = (compose_images_ssh(a.recipe, a.ssh, rdir) if a.ssh
|
||||
else compose_images(a.recipe, a.recipe_dir))
|
||||
if a.only:
|
||||
refs = [r for r in refs if a.only in r]
|
||||
results = [resolve(r) for r in refs]
|
||||
report = {
|
||||
"recipe": a.recipe,
|
||||
"images": results,
|
||||
"upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
|
||||
"unresolved": [r["ref"] for r in results if r["status"].startswith("error")],
|
||||
# The whole point: distinguish "checked, current" from "could not check".
|
||||
"all_resolved": not any(r["status"].startswith("error") for r in results),
|
||||
}
|
||||
if not a.table:
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0
|
||||
print(f"{a.recipe} — {len(results)} images")
|
||||
for r in results:
|
||||
flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current")
|
||||
print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else ""))
|
||||
print(f" shape={r['shape']} candidates={r['candidates']}"
|
||||
f" newest_same_shape={r['newest_same_shape']}"
|
||||
f" newest_within_major={r['newest_within_major']}")
|
||||
if r["status"] != "ok":
|
||||
print(f" status: {r['status']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -455,13 +455,38 @@ class TestReleaseNoteResolution(unittest.TestCase):
|
||||
self.assertEqual(rep["indeterminate"], [])
|
||||
self.assertEqual(rep["resolved_by_release_notes"]["CVE-TBD"], ["7.4.5", "8.0.3"])
|
||||
|
||||
def test_release_naming_it_only_outside_the_window_stays_indeterminate(self):
|
||||
def test_naming_releases_all_below_the_window_means_ALREADY_fixed(self):
|
||||
# Every known fix predates the version we were already on, so this upgrade did not deliver
|
||||
# it. That is a DECISION, not an unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 such
|
||||
# advisories, and calling them "could not judge" overstates the uncertainty.
|
||||
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
||||
releases={"CVE-TBD": ["6.2.19"]})
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
||||
self.assertEqual(rep["indeterminate"], [])
|
||||
self.assertIn("CVE-TBD", rep["already_fixed_before_upgrade"])
|
||||
self.assertIn("outside-window", rep["cves"]["CVE-TBD"]["classification"])
|
||||
|
||||
def test_naming_releases_only_ABOVE_the_window_stays_indeterminate(self):
|
||||
# The fix landed after our target, so we are still exposed. Deliberately NOT decided as a
|
||||
# tidy "not fixed": it is an open vulnerability and must stay visible to the operator.
|
||||
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
||||
releases={"CVE-TBD": ["9.0.0"]})
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
||||
self.assertIn("CVE-TBD", rep["indeterminate"])
|
||||
|
||||
def test_vendor_page_cve_on_the_same_repo_uses_release_notes(self):
|
||||
# mailu announces its Roundcube CVEs only on github.com/Mailu/Mailu/releases. Requiring an
|
||||
# advisory feed sent a deterministic case to pass 2; it is now decided in pass 1.
|
||||
rep = run_scan([gh("Mailu/Mailu", [])],
|
||||
[vendor("https://github.com/Mailu/Mailu/releases", ["CVE-2026-54432"])],
|
||||
v_from="2024.06.55", v_to="2024.06.57",
|
||||
urls=["https://github.com/Mailu/Mailu"],
|
||||
releases={"CVE-2026-54432": ["2024.06.56"]})
|
||||
self.assertIn("CVE-2026-54432", rep["fixed_by_this_upgrade"])
|
||||
self.assertEqual(rep["cve_count_fixed"], 1)
|
||||
|
||||
def test_release_evidence_is_recorded_for_audit(self):
|
||||
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
|
||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
|
||||
@@ -478,6 +503,40 @@ class TestReleaseNoteResolution(unittest.TestCase):
|
||||
self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {})
|
||||
|
||||
|
||||
class TestReleaseLineSemantics(unittest.TestCase):
|
||||
"""A fix inside the numeric window is not a fix on the branch you actually land on."""
|
||||
|
||||
def test_fix_later_on_the_targets_own_line_is_not_counted(self):
|
||||
# ClickHouse fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20. Landing on 23.10.4.25 crosses
|
||||
# the 23.9 fix numerically but is BELOW its own line's fix, so it does not have it.
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
||||
[adv("CVE-2023-48704", patched="v23.10.5.20; v23.9.6.20; v23.8.8.20")])],
|
||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
||||
|
||||
def test_fix_earlier_on_the_targets_own_line_is_counted(self):
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
||||
[adv("CVE-2023-47118", patched="v23.10.2.13; v23.8.6.16")])],
|
||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-47118"])
|
||||
|
||||
def test_fix_exactly_at_the_target_is_counted(self):
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
||||
[adv("CVE-2023-48298", patched="v23.10.4.25; v23.9.5.29")])],
|
||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-48298"])
|
||||
|
||||
def test_no_fix_on_the_target_line_falls_back_to_the_window(self):
|
||||
# redis fixes 7.4.6/8.0.4/8.2.2 with no 8.10.x entry; landing on 8.10 still has them,
|
||||
# because nothing on the 8.10 line is named as a LATER fix.
|
||||
rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2")])],
|
||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"])
|
||||
|
||||
|
||||
class TestAdjudicationEvidenceAssembly(unittest.TestCase):
|
||||
"""Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""
|
||||
|
||||
@@ -599,16 +658,22 @@ class TestHistoricReportNumbers(unittest.TestCase):
|
||||
for cve in delta:
|
||||
self.assertIn("redis", both["cves"][cve]["sources"][0])
|
||||
|
||||
def test_mailu_scan_finds_zero_and_says_so_knowingly(self):
|
||||
# Published as 2 via the UNION with release-note reading; the scan's own contribution is 0,
|
||||
# and 0 here must mean "checked, none", not "could not check".
|
||||
def test_mailu_finds_the_roundcube_pair_without_an_agent(self):
|
||||
# Published as 2 on 2026-08-07, but only because an agent read the release notes; the scan
|
||||
# itself contributed 0. It now reaches 2 deterministically: the CVEs appear only on
|
||||
# github.com/Mailu/Mailu/releases, and release 2024.06.56 (inside the window) names them.
|
||||
rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")])
|
||||
self.assertEqual(rep["cve_count_fixed"], 0)
|
||||
self.assertTrue(rep["count_known"])
|
||||
self.assertEqual(rep["cve_count_fixed"], 2)
|
||||
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-54432", "CVE-2026-54433"})
|
||||
# The redis bump fixes nothing new — every advisory it crosses was fixed at or before 8.6.3.
|
||||
self.assertEqual(rep["cve_count_indeterminate"], 0)
|
||||
|
||||
def test_keycloak_26_7_0_to_26_7_1_is_7(self):
|
||||
def test_keycloak_26_7_0_to_26_7_1_is_12(self):
|
||||
# Was 7 while only the GHSA feed was consulted. keycloak lists five more CVEs in the 26.7.1
|
||||
# release notes' fixed-issues section that it never filed as advisories — the gitea pattern.
|
||||
rep = self._count("keycloak", "26.7.0", "26.7.1")
|
||||
self.assertEqual(rep["cve_count_fixed"], 7)
|
||||
self.assertEqual(rep["cve_count_fixed"], 12)
|
||||
self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 5)
|
||||
|
||||
|
||||
def _main():
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# Upstream sources — mumble
|
||||
|
||||
| service | image | source repo | releases / changelog |
|
||||
|---------|-------|-------------|----------------------|
|
||||
| app | mumblevoip/mumble-server | https://github.com/mumble-voip/mumble | https://github.com/mumble-voip/mumble/releases |
|
||||
| web | rankenstein/mumble-web | https://github.com/rankenstein/mumble-web | https://github.com/rankenstein/mumble-web/releases |
|
||||
|
||||
## Standing notes
|
||||
- This file was **missing entirely** until 2026-08-11. Without it the advisory scan had no source to
|
||||
query, and still printed "0 identified by the deterministic scan" — which was then published as a
|
||||
clean `0` in the 2026-08-11 CVE check. The scan now refuses to emit a count when it has no usable
|
||||
source (it reports UNKNOWN), and `audit-sources.py` flags a missing registry file directly.
|
||||
- `mumblevoip/mumble-server` tracks the upstream server releases and DOES publish GitHub security
|
||||
advisories, so it is the recipe's primary CVE source.
|
||||
- `rankenstein/mumble-web` is a **fork** of the original `Johni0702/mumble-web`, which has been
|
||||
dormant since 2023-05. The fork itself last pushed 2023-07 and its Docker tag `0.5` was last built
|
||||
well over five years ago. Neither is archived, but treat the web client as effectively unmaintained:
|
||||
if a CVE lands there, expect no upstream fix and plan a replacement rather than an upgrade.
|
||||
- The server image tag is `v<version>-<build>` (e.g. `v1.6.870-4`); the trailing number is the image
|
||||
build, not an app version, and moves independently of upstream releases — `abra recipe upgrade`
|
||||
reports "no new versions" for it, so use `resolve-images.py` to see those bumps.
|
||||
Reference in New Issue
Block a user