advisory-scan: tests, audit, and two real undercounts they found

Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions
against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives
every count with a SEPARATE semver implementation and its own release fetch and
diffs against the scanner. Both found real defects:

1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE
   patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the
   norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also
   keeps the upper bound conservative (18.5 stays out of a window ending at 18).

2. Advisories with no knowable fix version were silently counted as 'not fixed'.
   Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an
   open-ended range — six of them high severity. They are now INDETERMINATE:
   not counted, not dismissed, and surfaced in the output.

   All twelve turned out to be genuinely fixed: redis names each in the release
   notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10,
   7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from
   release notes, with the naming tags recorded as the citation. discourse's
   redis contribution goes 5 -> 17, and its total 128 -> 140.

Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot
settle: it hands over each open case's full evidence, plus every verdict pass 1
reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that
evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both
gitea CVSS-9.8 RCEs — now reach it instead of being dropped.

Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is
tested there is deterministic: which cases it selects, and that truncation is
announced rather than silent.

SPEC.md rewritten around the two passes.
This commit is contained in:
autonomic-bot
2026-08-11 01:29:11 +00:00
parent 46c4fff1a6
commit 44cb9b6704
6 changed files with 1288 additions and 57 deletions
+3
View File
@@ -52,6 +52,9 @@ keeps every weekly edition looking the same regardless of which model writes the
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
- **A count with undetermined advisories is a FLOOR.** If the scan block says N advisories
"could NOT be judged", report the number but say in the notes that it is a floor — those
advisories are neither fixed nor safe, they are unmeasured. Never round them away.
- **Counts span every image, each judged by its own window.** A scan block lists one line per
image with its version range and classification method; the headline is their union. So a
recipe's count legitimately includes **sidecar** CVEs (discourse's 128 = 123 app + 5 redis).
+10 -3
View File
@@ -176,9 +176,16 @@ e.g. discourse moving app 3.5.3→2026.7.1 *and* redis 7.4→8.10:
... --from 3.5.3 --to 2026.7.1 --image redis=7.4:8.10
```
→ 128 CVEs (123 app + 5 redis), where the redis five include a **critical** (CVE-2025-49844) that is
invisible if the sidecar is left out. `<name>` is substring-matched against source repo names, so make
it specific enough to hit exactly one.
→ 140 CVEs (123 app + 17 redis), where the redis seventeen include a **critical** (CVE-2025-49844)
that is invisible if the sidecar is left out. `<name>` is substring-matched against source repo names,
so make it specific enough to hit exactly one.
**If the scan reports advisories it could NOT judge, re-run it with `--adjudicate`.** That is a second
pass: it collects each open case's full evidence (advisory prose, references, affected ranges, every
release naming the CVE) and asks YOU to decide FIXED / NOT-FIXED / STILL-UNKNOWN with a reason citing
that evidence. The deterministic number is a **floor** — add every FIXED to the count. Say
STILL-UNKNOWN rather than inferring from memory, and never record an undecided CVE as unaffected.
It also shows what pass 1 already decided; if a verdict looks wrong given its evidence, say so.
It queries, per recipe: the **GitHub Security Advisories API** for every source repo in
`cc-ci-plan/upstream/<recipe>.md` (CVE + GHSA + severity + vulnerable/patched ranges, so