recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

advisory-scan: tests, audit, and two real undercounts they found

Browse Source 
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions
against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives
every count with a SEPARATE semver implementation and its own release fetch and
diffs against the scanner. Both found real defects:

1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE
   patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the
   norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also
   keeps the upper bound conservative (18.5 stays out of a window ending at 18).

2. Advisories with no knowable fix version were silently counted as 'not fixed'.
   Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an
   open-ended range — six of them high severity. They are now INDETERMINATE:
   not counted, not dismissed, and surfaced in the output.

   All twelve turned out to be genuinely fixed: redis names each in the release
   notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10,
   7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from
   release notes, with the naming tags recorded as the citation. discourse's
   redis contribution goes 5 -> 17, and its total 128 -> 140.

Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot
settle: it hands over each open case's full evidence, plus every verdict pass 1
reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that
evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both
gitea CVSS-9.8 RCEs — now reach it instead of being dropped.

Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is
tested there is deterministic: which cases it selects, and that truncation is
announced rather than silent.

SPEC.md rewritten around the two passes.
This commit is contained in:
autonomic-bot
2026-08-11 01:29:11 +00:00
parent 46c4fff1a6
commit 44cb9b6704
6 changed files with 1288 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cc-ci-plan/advisory-scan.SPEC.md
+159 -42
View File
@@ -1,12 +1,14 @@
# Advisory scan — specification
What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the
implementation as it stands (2026-08-10); if you change the code, change this file in the same commit.
implementation as it stands (2026-08-11); if you change the code, change this file in the same commit.
`cc-ci-plan/test-advisory-scan.py` is the executable half of this spec — every rule below is asserted
there.
**Role.** A deterministic, per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a).
It is **strictly additive**: it never replaces the release-note reading the upgrade agent already does.
The CVE count reported for a recipe is the **union** of what the agent read and what this scan found;
the scan may never *lower* a count established by reading.
**Role.** A per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a). It is
**strictly additive**: it never replaces the release-note reading the upgrade agent already does. The
CVE count reported for a recipe is the **union** of what the agent read and what this scan found; the
scan may never *lower* a count established by reading.
**Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly
report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release
@@ -16,18 +18,35 @@ Nothing in the pipeline queried an advisory source. This scan closes that hole.
---
## Two passes
| | Pass 1 — measure | Pass 2 — judge (`--adjudicate`) |
|---|---|---|
| Who | Pure Python, no model | The calling agent, a model |
| Does | Collects evidence and decides every case it can by arithmetic | Weighs the collected evidence on cases arithmetic cannot settle |
| Output | A count, or `UNKNOWN` | FIXED / NOT-FIXED / STILL-UNKNOWN per open case |
| Rule | Deterministic and reproducible | May only **raise** the count, never lower it |
**Prefer pass 1.** Every case pass 1 decides is one that reproduces identically next week. Pass 2 exists
for evidence that is prose rather than data — a fallback, not a co-equal stage. When a class of case
keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c is exactly that: it moved
12 redis advisories out of pass 2 and into arithmetic.
---
## Inputs
```
advisory-scan.py <recipe> [--from <version>] [--to <version>]
[--image <name>=<from>:<to>]... [--json] [--registry DIR]
[--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
```
| Input | Meaning |
|---|---|
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable). `NAME` is matched as a substring against source repo names, e.g. `--image redis=7.4:8.10`. Malformed values are warned about on stderr and skipped. Without it that image's advisories stay unclassified. |
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
| `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
@@ -35,6 +54,8 @@ Exit code is always 0 — this is informational. Failures are *reported*, never
---
# Pass 1 — deterministic
## Step 1 — Collect source URLs from the registry
Read `cc-ci-plan/upstream/<recipe>.md` and extract every `http(s)://…` URL.
@@ -59,8 +80,8 @@ with *"not checked"*.
For every `github.com/<owner>/<repo>` URL in the registry:
`GET /repos/<owner>/<repo>/security-advisories`.
Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, `published_at`, and **all**
`vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined with `;`).
Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, **`description`**, `published_at`,
and **all** `vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined `;`).
- **All entries, not just the first.** An advisory carries one entry **per patched release line** —
n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the
@@ -68,12 +89,11 @@ Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, `published_at
- **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns
at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy
projects. discourse has 286 advisories; a single page cannot even cover one upgrade window.
- **The description is kept from this response.** It is already present here, and pass 2 needs the
prose; re-fetching it per advisory would cost one request each.
- **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not**
a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal.
This source is primary because it carries **severity and version ranges**, making "fixed by *this*
upgrade" computable rather than guessed.
### 2b. Vendor release / security pages
Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160
@@ -83,7 +103,8 @@ URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as temp
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.
This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
page names neither.
page names neither. A CVE found **only** here carries no version data, so pass 1 cannot place it — it
goes to pass 2 (§6).
### 2c. OSV.dev — supplementary
@@ -96,7 +117,7 @@ Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a
## Step 3 — Union
All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched,
published_at, context}`. A CVE seen by several sources keeps them all.
description, published_at, url, cvss, context}`. A CVE seen by several sources keeps them all.
## Step 4 — Classify against the upgrade window
@@ -104,25 +125,33 @@ Two invariants govern this step, both learned from a wrong answer in production.
> **A. Every image is judged by its OWN versions.** The app repo uses `--from/--to`; each sidecar uses
> its own `--image NAME=FROM:TO`. **Pass them all in ONE invocation** — the count is a union across
> images, and the UNKNOWN guarantee in B only holds when a single run sees every one. An image with no window is **not** classified — its advisories are
> listed as unclassified so they stay visible without inflating the count. The reported count is the
> **union across windows**, and each window is classified independently (so one may use version
> ranges while another falls back to dates).
> images, and the UNKNOWN guarantee in B only holds when a single run sees every one. An image with no
> window is **not** classified; its advisories are listed as unclassified so they stay visible without
> inflating the count. Each window is classified independently, so one may use version ranges while
> another falls back to dates.
> *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including
> `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically
> inside discourse's `3.5.3 → 2026.7.1` range. The fix is not to ignore sidecars but to give each one
> the versions it actually moved through: with `--image redis=7.4:8.10`, discourse scores
> **128 = 123 (app, by date) + 5 (redis, by version range)** — and the redis five include
> `CVE-2025-49844`, **critical**, which was invisible while sidecars went uncounted.
> **140 = 123 (app, by date) + 17 (redis)**, and the redis seventeen include `CVE-2025-49844`,
> **critical**, invisible while sidecars went uncounted.
>
> **B. Never emit a number you cannot justify.** If neither method below can order the window, the
> count is `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety.
> **B. Never emit a number you cannot justify.** If no method can order a window, the count is
> `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
> cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".
### 4a. By patched version (preferred — exact)
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
satisfies `from < p <= to`, using a loose numeric key (leading integers per dot-part).
satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this
upgrade's doing), inclusive upper.
**Comparison is zero-padded to equal length**, so `18` == `18.0` == `18.0.0` as semver means it.
Without padding, plain tuple order makes `(18,) < (18,0)`, i.e. a fix in 18.0 falls *outside* a window
ending at 18 — and bare major tags are the norm for sidecars (`postgres:18`, `redis:8-alpine`). Padding
is permissive at the lower bound and conservative at the upper: with `to = 18`, a fix in `18.5` is
**not** counted, because nothing proves which 18.x a floating tag resolved to.
### 4b. By advisory publish date (fallback — temporal)
@@ -132,22 +161,50 @@ component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver
Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while
shipping earlier), but **release dates always order**. So:
1. Resolve `--from` and `--to` to **git tag dates** on the primary repo (tries `v<version>` then
1. Resolve `--from` and `--to` to **git tag dates** on that source (tries `v<version>` then
`<version>`; annotated tag → tagger date, else commit date).
2. An advisory counts as fixed when `date_from < published_at <= date_to`.
2. An advisory counts as fixed when `date_from < published_at <= date_to` — the same exclusive/inclusive
boundaries as 4a, so the two methods agree at the edges.
Both the method and the resolved window appear in the output. This reproduces, automatically, the hand
count that established discourse `3.5.3` (2025-12-30) → `2026.7.1` (2026-07-31) = **123 CVEs**.
This reproduces, automatically, the hand count that established discourse `3.5.3` (2025-12-30) →
`2026.7.1` (2026-07-31) = **123 CVEs**.
*Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes
**first-party plugin advisories** where the vendor files them on the same repo — which is why a
plugin-rich project scores far higher than a monolith, not a statement about relative security.
### 4c. Otherwise
### 4c. By release notes naming the CVE (rescue — for advisories with no fix version)
`count_known = false`, `cve_count_fixed = null`, and the markdown headline reads
**"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an explicit
*"This is NOT zero"*.
Applied to advisories 4a/4b could not decide, **before** giving up on them. Fetch the source repo's
GitHub **releases** (cached per repo, 4 pages) and find every tag whose notes **name the CVE id**. If
any such tag falls inside the window by 4a's rule, the advisory is fixed by this upgrade, and the
naming tags are recorded in `fix_versions_from_release_notes` as the citation.
> **Why this is not optional.** Vendors routinely publish an advisory with `patched_versions: "TBD"`
> and then name the CVE in the release notes of every branch that got the fix. **All 12** redis
> advisories crossed by discourse's redis bump are exactly this shape — `TBD`, or a placeholder like
> `7.4.X`, with an open-ended `vulnerable_version_range` (`All`, `>= 7.0.0`) — yet each is named in
> concrete releases (`CVE-2025-32023` → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Without this method
> discourse's redis contribution reads 5; with it, 17. Six of the twelve are high severity.
### 4d. Otherwise — indeterminate, not excluded
An advisory from a **windowed** source that none of 4a–4c could decide — no usable `patched_versions`,
an open-ended vulnerable range, and no release note naming it — is recorded as **indeterminate**. It is:
- **not** added to the count (nothing justifies counting it), and
- **not** treated as unaffected (nothing justifies dismissing it either).
It is listed prominently, the headline reads *"(at least — see undetermined below)"*, and it becomes an
input to pass 2. Silently excluding these is the same defect class as printing `0` for an unscanned
recipe, one level down.
### 4e. Unorderable window
If neither 4a nor 4b can order a window at all, `count_known = false`, `cve_count_fixed = null`, and the
headline reads **"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an
explicit *"This is NOT zero"*. If **any** requested window is unorderable the whole count is suppressed;
a partial number would understate a security figure.
## Step 5 — Output
@@ -155,16 +212,56 @@ Markdown (default) for pasting into the per-recipe upgrade log, or `--json`.
| Field | Meaning |
|---|---|
| `cve_count_fixed` | Union across all windows, or **`null`** if ANY requested window could not be ordered (a partial number would understate) |
| `cve_count_fixed` | Union across all windows, or **`null`** if any window was unorderable |
| `count_known` | Distinguishes "counted zero" from "could not count" |
| `windows` | Every source classified, with its from/to (the internal computed ranges) |
| `classified_by` | **Per source**: `patched version ranges` or `advisory publish date (version scheme changed)` |
| `cve_count_indeterminate` / `indeterminate[]` | Judged by nothing; a **floor marker** on the count |
| `resolved_by_release_notes` | CVE → the tags that named it (4c citations) |
| `windows` | Every source classified, with its from/to |
| `classified_by` | **Per source**: which method decided it |
| `date_window` | **Per source**, when 4b was used |
| `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id |
| `unclassified[]` | Seen but not attributable to this window (incl. other images) |
| `sources[]` | Every source with its own status |
| `sources_failed[]` | **Genuine** failures only |
| `sources_benign[]` | `no-advisories-published`, `skipped: template URL` |
| `unclassified[]` | Seen but not attributable (other images, or vendor-page-only) |
| `sources[]` / `sources_failed[]` / `sources_benign[]` | Per-source status; only genuine failures in `failed` |
---
# Pass 2 — adjudication (`--adjudicate`)
## Step 6 — What gets judged
Two kinds of open case, both real gaps rather than noise:
1. **Indeterminate** (§4d) — from an image *with* a window, but no fix version is knowable anywhere.
2. **Vendor-page-only** — a CVE seen only on a vendor security page, with no structured advisory
behind it. **gitea's two CVSS-9.8 RCEs are this shape.** They carry no version data, so no
arithmetic can place them, but the page prose usually states the fixed release.
## Step 7 — The evidence dossier
Pass 1 collects; pass 2 judges. Nothing in the dossier interprets — it assembles what was *measured*,
so the judgement is made against evidence rather than recollection. That distinction is the whole
point: the original failure was a report leaning on model knowledge that predated the CVEs, with no
source queried at all.
Per open case: severity, CVSS, sources, its window, why it is undecided, `patched_versions` and
`vulnerable_version_range` **as published**, summary, full description, references, affected ranges
with `first_patched_version`, and **every release tag naming the CVE** — whether or not in window, since
the model may reason about branch lines the arithmetic deliberately will not.
**Pass 2 also sees every decision pass 1 made** — a compact table of counted and excluded advisories
with the evidence behind each verdict. A deterministic verdict can still be wrong (a mis-parsed range,
a release note that mentions a CVE without fixing it), and only a reader with the evidence in front of
it can catch that. Silence means agreement.
## Step 8 — The verdict contract
For each open case: **FIXED** / **NOT-FIXED** / **STILL-UNKNOWN**, each with a one-line reason
**citing the evidence shown**. Every FIXED is added to the recipe's count — pass 1's number is a floor,
not a total. If the evidence does not settle it, **STILL-UNKNOWN**: do not infer from memory of the
project, and never record an undecided CVE as unaffected.
**No silent caps.** `MAX_ADJUDICATE` (25) and `MAX_REVIEW_ROWS` (400) bound the output; whenever either
truncates, the block says how many were dropped and that the unshown remain undetermined.
---
@@ -176,21 +273,41 @@ Markdown (default) for pasting into the per-recipe upgrade log, or `--json`.
- **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly
how two CVSS-9.8 gitea RCEs were published as `none`;
- **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed;
- a count with **indeterminate advisories** → the number is a floor; say so in the notes;
- benign notes → never `?`.
`?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is
a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect
in this tool or stale registry data.
## Testing
`test-advisory-scan.py` — 58 offline tests (fixtures, no network) plus 6 live regressions against the
counts published in week-2026-08-07. **The offline tier covers pass 1 only, by design**: pass 2's
judgement is a model's and cannot be asserted deterministically. What *is* tested about pass 2 is the
part that stays deterministic — which cases it selects, and that truncation is always announced.
```
python3 test-advisory-scan.py # offline
python3 test-advisory-scan.py --live # + historic report numbers
```
`audit-advisory-scan.py` re-derives the counts with a **separate** semver implementation and its own
release fetch, then diffs against the scanner. Run it after changing classification; it is what caught
the 12 undercounted redis CVEs.
## Known limits
1. **Versions must be supplied per image.** An image with no `--image` is not counted — the scan will not
guess a version range it was not told. `/recipe-upgrade` passes one per image it bumped.
1. **Versions must be supplied per image.** An image with no `--image` is not counted — the scan will
not guess a version range it was not given. `/recipe-upgrade` passes one per image it bumped.
2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time.
3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was
never pointed at.
4. **`NAME` matching is substring-against-source-name**, so a short or generic name can attach to more
than one repo (`postgres` matches `discourse/discourse-postgres`). The primary source is claimed
first and cannot be stolen; be specific enough to hit one repo.
5. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed
first and cannot be stolen. A name matching **nothing** is silently ignored — a typo costs coverage
without warning.
5. **Release-note rescue (4c) trusts that naming implies fixing.** A release note that merely mentions a
CVE would be read as fixing it. Pass 2's review table exists partly to catch this.
6. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed
sources (visibly, but degraded).