Working against a stale mirror has cost us three different ways: - mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while upstream had already merged AND released it (3.1.3+2024.06.57). The work was done; only our mirror was behind. Reconciling closed the PR automatically. - a stale mirror makes a survey report 'no upgrades available', so the recipe silently drops out of the weekly run. - reading the wrong branch: several coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's main is 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs, so reading main manufactures a false 'three releases behind, missing two CVSS-9.8 RCEs' finding. The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and already resolves the default branch itself. What was missing was a single obvious entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or --all, and is idempotent — recipe work lives in branches, never on mirror main, so force-syncing main discards nothing. /ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now point at the shared script.
184 lines
12 KiB
Markdown
184 lines
12 KiB
Markdown
---
|
|
name: cve-check
|
|
description: Fleet-wide CVE sweep WITHOUT upgrading anything. For every recipe cc-ci deploys, works out what upgrade is available (current pinned tag → newest supported tag, per image including sidecars), runs the deterministic advisory scan over that window, adjudicates whatever the scan could not decide, and publishes a CVE report to report.ci.commoninternet.net as cve-<DATE>.html. READ-ONLY — opens no PRs, edits no recipes, runs no CI, merges nothing. Answers "what are we exposed to that an upgrade would fix?" in minutes rather than the hours a full upgrade run takes. Invoke as /cve-check [recipe ...] [--weekly-only].
|
|
---
|
|
|
|
# cve-check
|
|
|
|
A **security sweep, not an upgrade run.** It answers one question for every recipe cc-ci deploys:
|
|
|
|
> If we upgraded this recipe today, how many CVEs would that fix, and how bad are they?
|
|
|
|
It is the cheap, safe half of `/upgrade-all`: the same version research and the same advisory scan,
|
|
with **no implementation, no CI, and no PRs**. Use it when you want the security picture now — after a
|
|
vendor announcement, before deciding what to prioritise, or between weekly runs. When you want the PRs
|
|
too, use **`/cve-check-and-upgrade`**.
|
|
|
|
**Read-only, absolutely.** Never edit a recipe, never open or comment on a PR, never merge, never
|
|
deploy. The only thing it writes is its own log and the published report page.
|
|
|
|
## Arguments
|
|
- `<recipe> …` — sweep only these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
|
|
- `--weekly-only` — skip rows tagged `external`. **Off by default on purpose**: an `external` recipe is
|
|
still deployed and still exposes us, so a security sweep that silently skipped it would misreport the
|
|
fleet's exposure. Externals are swept and clearly marked "maintained elsewhere" in the report.
|
|
|
|
## Procedure
|
|
|
|
> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
|
|
> `abra` needs a TTY. Wrap every abra call: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
|
|
> (`git` and other commands do NOT need the wrapper.)
|
|
|
|
### 1. Build the candidate list
|
|
Read `cc-ci-plan/used-recipes.md` — the canonical inventory. Take every row (both tiers), recording the
|
|
tier per recipe; with `--weekly-only`, drop the `external` rows. An explicit recipe argument overrides
|
|
any skip.
|
|
|
|
### 2. Per recipe — establish the upgrade window WITHOUT upgrading
|
|
|
|
This is `/recipe-upgrade` step 1's research, stopping before it implements anything.
|
|
|
|
> ⚠️ **The same four things that silently skip recipes apply here — handle ALL FOUR:**
|
|
> 1. **pseudo-TTY** — per the box above.
|
|
> 2. **go-git auth to git.autonomic.zone** — recipes on the private mirror FATA
|
|
> `authentication required: Unauthorized`. Bake creds into origin first (idempotent, only when
|
|
> origin is on git.autonomic.zone):
|
|
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
|
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
|
> after. Only a genuinely dirty TRACKED tree is a skip.
|
|
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
|
|
> hand-check the registry; run the resolver, which is abra-independent and covers every image:
|
|
> ```
|
|
> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
|
|
> ```
|
|
> It reports, per image, `newest_within_major` (the compatibility-safe pick) and
|
|
> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
|
|
> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
|
|
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
|
|
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
|
|
|
|
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
|
|
`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
|
|
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
|
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
|
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
|
lives in **branches**, never directly on `main`, so a force-sync of `main` to upstream discards
|
|
nothing; it also auto-closes mirror PRs whose changes upstream has already merged.
|
|
|
|
> ### ⚠️ The default branch may be `master`, not `main` — check, do not assume
|
|
> Several coopcloud recipes keep a **stale `main` alongside the real default `master`**. gitea is one:
|
|
> `main` sits at 1.24.2-rootless while `master` has 1.27.1-rootless plus the merged PRs and the 3.6.3
|
|
> release. Reading `main` there tells you the recipe is three releases behind and missing two CVSS-9.8
|
|
> RCE fixes — a false alarm that reads exactly like a real one. Resolve the default branch from the
|
|
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
|
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
|
|
|
**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
|
|
nothing for images it cannot parse, and it reported "no new versions" for images that did have them
|
|
(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
|
|
the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
|
|
(plausible's clickhouse) carried four CVEs.
|
|
|
|
Then read versions:
|
|
```
|
|
set -a; . /srv/cc-ci/.testenv; set +a
|
|
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
|
|
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
|
|
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
|
|
git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
|
|
script -qec "abra recipe fetch $R --force -n" /dev/null; \
|
|
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
|
|
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
|
|
```
|
|
|
|
For each recipe produce **one window per image**: `current pinned tag → newest supported tag`. You need
|
|
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
|
|
CRITICAL came from, and an image with no window is not counted at all.
|
|
|
|
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
|
|
nothing an upgrade could fix. Record it and move on.
|
|
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
|
|
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
|
|
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
|
|
Only report `?` once BOTH the abra check and the direct check have failed. On the first real run this
|
|
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
|
|
but nothing in the survey said so).
|
|
|
|
### 3. Run the advisory scan over that window
|
|
```
|
|
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
|
|
[--image <name>=<old>:<new>]...
|
|
```
|
|
**One call per recipe with every image in it** — the count is a union across images, and the
|
|
UNKNOWN guarantee only holds when a single run sees them all. Paste the markdown block verbatim into
|
|
the per-recipe log at `/srv/cc-ci/.cc-ci-logs/cve-check/<DATE>/<recipe>.md`.
|
|
|
|
### 4. Adjudicate what the scan could not decide (pass 2)
|
|
If the block reports advisories it **could NOT judge**, or the count is **UNKNOWN**, re-run with
|
|
`--adjudicate` and decide each open case yourself:
|
|
|
|
```
|
|
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> … --adjudicate
|
|
```
|
|
|
|
Answer **FIXED / NOT-FIXED / STILL-UNKNOWN** per case, each with a one-line reason **citing the
|
|
evidence shown** — never from memory of the project, which is the exact failure that let two CVSS-9.8
|
|
gitea RCEs be published as "none". Every FIXED is added to the count; pass 1's number is a floor. The
|
|
block also lists what pass 1 already decided — if a verdict looks wrong given its evidence, say so.
|
|
Record your verdicts in the per-recipe log so the number is auditable.
|
|
|
|
### 5. Classify severity and priority
|
|
For each recipe collect the CVE ids with **severities** (the scan gives them, with GHSA ids). Sort the
|
|
report rows by what an operator should deal with first:
|
|
1. recipes with a **critical**, then **high**, then anything else with CVEs (more CVEs higher within a band);
|
|
2. then `?` (a count that could not be established — investigate, do not ignore);
|
|
3. then recipes with an upgrade available but **0** CVEs;
|
|
4. then `UPTODATE`.
|
|
|
|
Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin advisories.
|
|
|
|
### 6. Write the report spec
|
|
`/tmp/cve-spec-<DATE>.json`, same shape as `/recipe-report` (see `recipe-report.py`'s header), with:
|
|
- **`"kind": "cve"`** — titles the page "The Recipe Report — CVE check" and files it as
|
|
`cve-<DATE>.html`. It appears in the SAME archive index as the weekly editions, suffixed
|
|
"— CVE check" so the two are told apart at a glance. Without this field you would overwrite that
|
|
date's weekly edition.
|
|
- `date`, `subtitle` "CVE check <human date>",
|
|
- `lead` — **one short paragraph**: fleet exposure in a sentence and what to do first.
|
|
- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
|
|
`1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
|
|
`PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
|
|
rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for
|
|
`external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
|
|
- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
|
|
established, a scan whose count is a floor with many undetermined advisories.
|
|
- `security[]` — one entry per **critical/high** finding: recipe · CVE id(s) + severity · what it fixes
|
|
· **which image** it is in. Name the image: `CVE-2025-49844` is a redis flaw, and an operator reading
|
|
"discourse" needs to know that.
|
|
- `changes[]` — **omit** (nothing changed; there are no PRs).
|
|
|
|
**`?` must stay RARE.** Use it only when a scan ran and reported genuinely failed sources, or the count
|
|
came back UNKNOWN and adjudication could not settle it. Never `none` for an unknown — a blank reads as
|
|
clean. A recipe with no upgrade available is `0`, not `?`. Many `?` is a bug for the Addendum.
|
|
|
|
### 7. Render and publish — via the script only
|
|
```
|
|
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py render /tmp/cve-spec-<DATE>.json /tmp/cve-<DATE>.html
|
|
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py publish /tmp/cve-<DATE>.html <DATE> cve
|
|
```
|
|
All layout is owned by `recipe-report.py`. Never hand-write or post-process HTML; if `render` errors,
|
|
fix the spec JSON and re-render. **Public page — no secrets, tokens, internal hostnames, raw logs, or
|
|
any billing/spend figures.**
|
|
|
|
### 8. Verify and stop
|
|
`curl -fsS https://report.ci.commoninternet.net/cve-<DATE>.html` renders and the index lists it. Print
|
|
the URL, a one-line summary (`N recipes swept · M with CVEs · K critical`), and `CVE CHECK COMPLETE`,
|
|
then go idle. One-shot — do not loop, and do not start upgrading anything.
|
|
|
|
## Guardrails
|
|
- **Read-only.** No PRs, no edits, no merges, no deploys, no CI runs. If a recipe looks urgent, say so
|
|
in the report — do not act on it. `/cve-check-and-upgrade` is the skill that acts.
|
|
- **Never report `0` for something you could not scan.** `0` means checked-and-clean; unknown is `?`.
|
|
- A count with undetermined advisories is a **floor** — say so in the notes rather than rounding away.
|
|
- **Public-safe output only.**
|