fix: fail closed on incomplete weekly image and CVE surveys

This commit is contained in:
notplants
2026-11-02 19:08:54 +00:00
parent e5cfa67e6e
commit b41eeaed75
9 changed files with 174 additions and 54 deletions
+14 -5
View File
@@ -95,8 +95,9 @@ For each recipe produce **one window per image**: `current pinned tag → newest
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
CRITICAL came from, and an image with no window is not counted at all.
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
nothing an upgrade could fix. Record it and move on.
- **No upgrade available** → the recipe is `UPTODATE`; its **CVEs fixed by an available upgrade**
count is `0`. This does not assert that the running image has no known CVEs. If advisory sources
failed or are blind, use `?` and say why.
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
@@ -111,9 +112,17 @@ python3 cc-ci-plan/audit-sources.py --security-sources
A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory
feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by
declaring an NVD CPE in their registry:
Do not assume that the fleet has zero blind recipes. Run the audit every sweep and inspect source
statuses; an HTTP error or rate limit is not evidence of no CVEs. `rustdesk-server` is currently
blind because its upstream GitHub repo publishes no security advisories and no other usable source
has been verified; mark it `?` until that changes.
Check `test -s /srv/cc-ci/.github-token` before a fleet scan. Without that read-only token,
GitHub's anonymous API limit can be exhausted partway through the fleet; HTTP 403 results are
failed sources, so the affected CVE counts remain unknown.
Two older blind spots — `mattermost-lts` and `mumble` — were addressed by declaring NVD CPEs in
their registries:
```
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
```
+10 -4
View File
@@ -30,7 +30,9 @@ keeps every weekly edition looking the same regardless of which model writes the
2. **Survey.** `python3 /srv/cc-ci/cc-ci-plan/recipe-report.py survey <DATE> > /tmp/survey.json`. This
gives you: the raw `/upgrade-all` summary; for every recipe mirror its open PRs + the `cc-ci/testme`
verdict; and each recipe's **per-recipe upgrade notes** (`upgrade_notes_md` — the breaking-change /
**CVE / security** analysis the upgrade did). Read it all.
**CVE / security** analysis the upgrade did). Read it all. Also run
`python3 /srv/cc-ci/cc-ci-plan/audit-sources.py --security-sources`; a recipe with no verified
security source or a failed source must be called out, not presented as a clean CVE scan.
3. **Review & classify.** The page order is: **short lead → the full wire table (priority-sorted) →
Addendum → Security Bulletin → per-recipe "What changed".** Same newspaper style; this order.
@@ -46,9 +48,13 @@ keeps every weekly edition looking the same regardless of which model writes the
lists **failed sources**, the count is **not** authoritative: render the cve cell as `?`
(unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea
RCEs were reported as "none" on 2026-08-07.
- **`?` must stay RARE — it means "we tried and could not tell", not "we didn't look".** Use it
ONLY when a scan ran and reported genuinely failed sources, **or when the scan block says
COUNT UNKNOWN**. In that case the scan's `0` means *not determined*: publish `?` and say so in
- The table's **CVEs fixed** column counts known CVEs addressed by the available recipe upgrade.
A zero for an up-to-date recipe means there is no upgrade window to credit; it does not
certify that the running images are free of vulnerabilities. If an upgrade exists but its
advisory scan is absent, a source failed, or an image window was unmatched, use `?`.
- **`?` means the count is not verified.** Use it when an upgrade has no scan, a source failed,
an image window was unmatched, or the scan block says COUNT UNKNOWN. In that case publish
`?` and say why in
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
+25 -6
View File
@@ -143,12 +143,31 @@ ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<r>; \
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
```
Build `RECIPES_TO_UPGRADE` = recipes with **≥1 available upgrade** (after the stash, so the untracked
overlay doesn't count as dirty). Others go to `SKIPPED_UPFRONT` with a reason (`dirty-worktree` only for
real TRACKED edits, `up-to-date`, `not-fetchable`). **A `tag and digest … not supported` parse FATA is
NOT `not-fetchable`** — per box item 4, catch it and resolve availability via the direct upstream-tag
check; only `not-fetchable` when the mirror genuinely can't be fetched (network/auth), never for a
digest-pinned image.
**Mandatory registry cross-check for EVERY recipe and EVERY image, even if abra says current:** after
reconciling the mirror and refreshing the recipe checkout, run
`python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --recipe-dir /root/.abra/recipes`
on cc-ci. Read the JSON `images`, `upgrades_available`, `unresolved`, and `all_resolved` fields.
The candidate list is the **union** of abra's upgrades and the resolver's upgrades. CryptPad's
`version-YYYY.M.D` app tag is a known counterexample: abra only reports a parse warning, while the
resolver finds the newer image. A warning is never evidence that an image is current. Inspect both
`newest_same_shape` and `newest_within_major`; a newer major is still a candidate for planning,
though it must not be applied without a compatibility check.
Build `RECIPES_TO_UPGRADE` from every recipe with at least one candidate. Only label a recipe
`up-to-date` if abra completed successfully **and** the resolver checked every relevant image
(`all_resolved: true`) **and** neither found a candidate. If an image is unresolved, templated,
floating, or lacks comparable tags, investigate it directly and record the evidence. Otherwise
record `unverified` with the image and reason; never fold it into `up-to-date`. `dirty-worktree`
applies only to real tracked edits. A tag+digest parse failure is not `not-fetchable`.
Run `python3 /srv/cc-ci/cc-ci-plan/audit-sources.py --security-sources` once for the full inventory
and include any `NO VERIFIED CVE SOURCE` recipes or failed advisory sources in the weekly summary.
Do not describe their CVE count as clean. This audit is a coverage check, not a substitute for
`advisory-scan.py` on each actual upgrade window.
Before the audit, check `test -s /srv/cc-ci/.github-token`. If it is missing, record that GitHub
advisory checks are subject to the anonymous 60-request/hour limit; any resulting HTTP 403 is a
failed source and must become `?` in the report. Never replace it with `0` or silently retry from
model memory. The file is a read-only GitHub token location; never print its contents.
**Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new