advisory-scan: tests, audit, and two real undercounts they found
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives every count with a SEPARATE semver implementation and its own release fetch and diffs against the scanner. Both found real defects: 1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also keeps the upper bound conservative (18.5 stays out of a window ending at 18). 2. Advisories with no knowable fix version were silently counted as 'not fixed'. Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an open-ended range — six of them high severity. They are now INDETERMINATE: not counted, not dismissed, and surfaced in the output. All twelve turned out to be genuinely fixed: redis names each in the release notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from release notes, with the naming tags recorded as the citation. discourse's redis contribution goes 5 -> 17, and its total 128 -> 140. Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot settle: it hands over each open case's full evidence, plus every verdict pass 1 reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both gitea CVSS-9.8 RCEs — now reach it instead of being dropped. Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is tested there is deterministic: which cases it selects, and that truncation is announced rather than silent. SPEC.md rewritten around the two passes.
This commit is contained in:
@@ -52,6 +52,9 @@ keeps every weekly edition looking the same regardless of which model writes the
|
||||
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
|
||||
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
|
||||
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
|
||||
- **A count with undetermined advisories is a FLOOR.** If the scan block says N advisories
|
||||
"could NOT be judged", report the number but say in the notes that it is a floor — those
|
||||
advisories are neither fixed nor safe, they are unmeasured. Never round them away.
|
||||
- **Counts span every image, each judged by its own window.** A scan block lists one line per
|
||||
image with its version range and classification method; the headline is their union. So a
|
||||
recipe's count legitimately includes **sidecar** CVEs (discourse's 128 = 123 app + 5 redis).
|
||||
|
||||
@@ -176,9 +176,16 @@ e.g. discourse moving app 3.5.3→2026.7.1 *and* redis 7.4→8.10:
|
||||
... --from 3.5.3 --to 2026.7.1 --image redis=7.4:8.10
|
||||
```
|
||||
|
||||
→ 128 CVEs (123 app + 5 redis), where the redis five include a **critical** (CVE-2025-49844) that is
|
||||
invisible if the sidecar is left out. `<name>` is substring-matched against source repo names, so make
|
||||
it specific enough to hit exactly one.
|
||||
→ 140 CVEs (123 app + 17 redis), where the redis seventeen include a **critical** (CVE-2025-49844)
|
||||
that is invisible if the sidecar is left out. `<name>` is substring-matched against source repo names,
|
||||
so make it specific enough to hit exactly one.
|
||||
|
||||
**If the scan reports advisories it could NOT judge, re-run it with `--adjudicate`.** That is a second
|
||||
pass: it collects each open case's full evidence (advisory prose, references, affected ranges, every
|
||||
release naming the CVE) and asks YOU to decide FIXED / NOT-FIXED / STILL-UNKNOWN with a reason citing
|
||||
that evidence. The deterministic number is a **floor** — add every FIXED to the count. Say
|
||||
STILL-UNKNOWN rather than inferring from memory, and never record an undecided CVE as unaffected.
|
||||
It also shows what pass 1 already decided; if a verdict looks wrong given its evidence, say so.
|
||||
|
||||
It queries, per recipe: the **GitHub Security Advisories API** for every source repo in
|
||||
`cc-ci-plan/upstream/<recipe>.md` (CVE + GHSA + severity + vulnerable/patched ranges, so
|
||||
|
||||
Reference in New Issue
Block a user