# Advisory scan — specification What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the implementation as it stands (2026-08-11); if you change the code, change this file in the same commit. `cc-ci-plan/test-advisory-scan.py` is the executable half of this spec — every rule below is asserted there. **Role.** A per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a). It is **strictly additive**: it never replaces the release-note reading the upgrade agent already does. The CVE count reported for a recipe is the **union** of what the agent read and what this scan found; the scan may never *lower* a count established by reading. **Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release notes*, which name neither — both were announced only in the vendor's blog security section — and the report then derived security content from those notes plus model knowledge, which predates the CVEs. Nothing in the pipeline queried an advisory source. This scan closes that hole. --- ## Two passes | | Pass 1 — measure | Pass 2 — judge (`--adjudicate`) | |---|---|---| | Who | Pure Python, no model | The calling agent, a model | | Does | Collects evidence and decides every case it can by arithmetic | Weighs the collected evidence on cases arithmetic cannot settle | | Output | A count, or `UNKNOWN` | FIXED / NOT-FIXED / STILL-UNKNOWN per open case | | Rule | Deterministic and reproducible | May only **raise** the count, never lower it | **Prefer pass 1.** Every case pass 1 decides is one that reproduces identically next week. Pass 2 exists for evidence that is prose rather than data — a fallback, not a co-equal stage. When a class of case keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c is exactly that: it moved 12 redis advisories out of pass 2 and into arithmetic. --- ## Inputs ``` advisory-scan.py [--from ] [--to ] [--image =:]... [--adjudicate] [--json] [--registry DIR] ``` | Input | Meaning | |---|---| | `` | Recipe name; selects `cc-ci-plan/upstream/.md` (the per-recipe URL registry) | | `--from` / `--to` | The **primary app image's** version window being upgraded across | | `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. | | `--adjudicate` | Run pass 2: append the evidence dossier for judgement | | `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` | | `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. | Exit code is always 0 — this is informational. Failures are *reported*, never raised. --- # Pass 1 — deterministic ## Step 1 — Collect source URLs from the registry Read `cc-ci-plan/upstream/.md` and extract every `http(s)://…` URL. **Trailing markdown punctuation is stripped** (`` ` `` `'` `"` `*` `.` `,` `;` `:` `>` `)`). The registry is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of `https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render `?` for no real reason. > **Registry hygiene matters.** The scan can only look where the registry points. Two classes of defect > have been found and fixed by running it: a **wrong URL** (`pgautoupgrade/pgautoupgrade`, which 404s — > the repo is `pgautoupgrade/docker-pgautoupgrade`) and a **missing** one (gitea's CVEs are announced at > `blog.gitea.com`, which the registry didn't list). When a vendor publishes security notes somewhere > the registry lacks, add it. ## Step 2 — Query the sources Three source classes, each recording **its own status** so *"checked, none found"* is never confused with *"not checked"*. ### 2a. GitHub Security Advisories — PRIMARY For every `github.com//` URL in the registry: `GET /repos///security-advisories`. Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, **`description`**, `published_at`, and **all** `vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined `;`). - **All entries, not just the first.** An advisory carries one entry **per patched release line** — n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window. - **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy projects. discourse has 286 advisories; a single page cannot even cover one upgrade window. - **The description is kept from this response.** It is already present here, and pass 2 needs the prose; re-fetching it per advisory would cost one request each. - **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not** a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal. ### 2b. Vendor release / security pages Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160 characters of context per hit. URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as templates** — they are human documentation (`…/changelog/v/`), not fetchable, and counting them as failures is wrong. This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release page names neither. A CVE found **only** here carries no version data, so pass 1 cannot place it — it goes to pass 2 (§6). ### 2c. OSV.dev — supplementary Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given. > **Measured, not assumed.** For the two gitea CVEs, OSV **404'd on both** and returned only Go > *dependency* advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword. > **Advisory databases lag the vendor**, which is why 2a and 2b lead and this is supplementary. ## Step 3 — Union All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched, description, published_at, url, cvss, context}`. A CVE seen by several sources keeps them all. ## Step 4 — Classify against the upgrade window Two invariants govern this step, both learned from a wrong answer in production. > **A. Every image is judged by its OWN versions.** The app repo uses `--from/--to`; each sidecar uses > its own `--image NAME=FROM:TO`. **Pass them all in ONE invocation** — the count is a union across > images, and the UNKNOWN guarantee in B only holds when a single run sees every one. An image with no > window is **not** classified; its advisories are listed as unclassified so they stay visible without > inflating the count. Each window is classified independently, so one may use version ranges while > another falls back to dates. > *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including > `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically > inside discourse's `3.5.3 → 2026.7.1` range. The fix is not to ignore sidecars but to give each one > the versions it actually moved through: with `--image redis=7.4:8.10`, discourse scores > **140 = 123 (app, by date) + 17 (redis)**, and the redis seventeen include `CVE-2025-49844`, > **critical**, invisible while sidecars went uncounted. > > **B. Never emit a number you cannot justify.** If no method can order a window, the count is > `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that > cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed". ### 4a. By patched version (preferred — exact) `patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p` satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this upgrade's doing), inclusive upper. **Comparison is zero-padded to equal length**, so `18` == `18.0` == `18.0.0` as semver means it. Without padding, plain tuple order makes `(18,) < (18,0)`, i.e. a fix in 18.0 falls *outside* a window ending at 18 — and bare major tags are the norm for sidecars (`postgres:18`, `redis:8-alpine`). Padding is permissive at the lower bound and conservative at the upper: with `to = 18`, a fix in `18.5` is **not** counted, because nothing proves which 18.x a floating tag resolved to. ### 4b. By advisory publish date (fallback — temporal) Used **only** when 4a cannot be trusted: a **version-scheme change**, detected as the leading version component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver `2026.7.1`. Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while shipping earlier), but **release dates always order**. So: 1. Resolve `--from` and `--to` to **git tag dates** on that source (tries `v` then ``; annotated tag → tagger date, else commit date). 2. An advisory counts as fixed when `date_from < published_at <= date_to` — the same exclusive/inclusive boundaries as 4a, so the two methods agree at the edges. This reproduces, automatically, the hand count that established discourse `3.5.3` (2025-12-30) → `2026.7.1` (2026-07-31) = **123 CVEs**. *Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes **first-party plugin advisories** where the vendor files them on the same repo — which is why a plugin-rich project scores far higher than a monolith, not a statement about relative security. ### 4c. By release notes naming the CVE (rescue — for advisories with no fix version) Applied to advisories 4a/4b could not decide, **before** giving up on them. Fetch the source repo's GitHub **releases** (cached per repo, 4 pages) and find every tag whose notes **name the CVE id**. If any such tag falls inside the window by 4a's rule, the advisory is fixed by this upgrade, and the naming tags are recorded in `fix_versions_from_release_notes` as the citation. > **Why this is not optional.** Vendors routinely publish an advisory with `patched_versions: "TBD"` > and then name the CVE in the release notes of every branch that got the fix. **All 12** redis > advisories crossed by discourse's redis bump are exactly this shape — `TBD`, or a placeholder like > `7.4.X`, with an open-ended `vulnerable_version_range` (`All`, `>= 7.0.0`) — yet each is named in > concrete releases (`CVE-2025-32023` → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Without this method > discourse's redis contribution reads 5; with it, 17. Six of the twelve are high severity. ### 4d. Otherwise — indeterminate, not excluded An advisory from a **windowed** source that none of 4a–4c could decide — no usable `patched_versions`, an open-ended vulnerable range, and no release note naming it — is recorded as **indeterminate**. It is: - **not** added to the count (nothing justifies counting it), and - **not** treated as unaffected (nothing justifies dismissing it either). It is listed prominently, the headline reads *"(at least — see undetermined below)"*, and it becomes an input to pass 2. Silently excluding these is the same defect class as printing `0` for an unscanned recipe, one level down. ### 4e. Unorderable window If neither 4a nor 4b can order a window at all, `count_known = false`, `cve_count_fixed = null`, and the headline reads **"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an explicit *"This is NOT zero"*. If **any** requested window is unorderable the whole count is suppressed; a partial number would understate a security figure. ## Step 5 — Output Markdown (default) for pasting into the per-recipe upgrade log, or `--json`. | Field | Meaning | |---|---| | `cve_count_fixed` | Union across all windows, or **`null`** if any window was unorderable | | `count_known` | Distinguishes "counted zero" from "could not count" | | `cve_count_indeterminate` / `indeterminate[]` | Judged by nothing; a **floor marker** on the count | | `resolved_by_release_notes` | CVE → the tags that named it (4c citations) | | `windows` | Every source classified, with its from/to | | `classified_by` | **Per source**: which method decided it | | `date_window` | **Per source**, when 4b was used | | `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id | | `unclassified[]` | Seen but not attributable (other images, or vendor-page-only) | | `sources[]` / `sources_failed[]` / `sources_benign[]` | Per-source status; only genuine failures in `failed` | --- # Pass 2 — adjudication (`--adjudicate`) ## Step 6 — What gets judged Two kinds of open case, both real gaps rather than noise: 1. **Indeterminate** (§4d) — from an image *with* a window, but no fix version is knowable anywhere. 2. **Vendor-page-only** — a CVE seen only on a vendor security page, with no structured advisory behind it. **gitea's two CVSS-9.8 RCEs are this shape.** They carry no version data, so no arithmetic can place them, but the page prose usually states the fixed release. ## Step 7 — The evidence dossier Pass 1 collects; pass 2 judges. Nothing in the dossier interprets — it assembles what was *measured*, so the judgement is made against evidence rather than recollection. That distinction is the whole point: the original failure was a report leaning on model knowledge that predated the CVEs, with no source queried at all. Per open case: severity, CVSS, sources, its window, why it is undecided, `patched_versions` and `vulnerable_version_range` **as published**, summary, full description, references, affected ranges with `first_patched_version`, and **every release tag naming the CVE** — whether or not in window, since the model may reason about branch lines the arithmetic deliberately will not. **Pass 2 also sees every decision pass 1 made** — a compact table of counted and excluded advisories with the evidence behind each verdict. A deterministic verdict can still be wrong (a mis-parsed range, a release note that mentions a CVE without fixing it), and only a reader with the evidence in front of it can catch that. Silence means agreement. ## Step 8 — The verdict contract For each open case: **FIXED** / **NOT-FIXED** / **STILL-UNKNOWN**, each with a one-line reason **citing the evidence shown**. Every FIXED is added to the recipe's count — pass 1's number is a floor, not a total. If the evidence does not settle it, **STILL-UNKNOWN**: do not infer from memory of the project, and never record an undecided CVE as unaffected. **No silent caps.** `MAX_ADJUDICATE` (25) and `MAX_REVIEW_ROWS` (400) bound the output; whenever either truncates, the block says how many were dropped and that the unshown remain undetermined. --- ## How consumers must read it `/recipe-report` renders the `cve` column from the **union** of this scan and the agent's own reading: - a clean scan → its number, **including `0`**; - **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly how two CVSS-9.8 gitea RCEs were published as `none`; - **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed; - a count with **indeterminate advisories** → the number is a floor; say so in the notes; - benign notes → never `?`. `?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect in this tool or stale registry data. ## Testing `test-advisory-scan.py` — 58 offline tests (fixtures, no network) plus 6 live regressions against the counts published in week-2026-08-07. **The offline tier covers pass 1 only, by design**: pass 2's judgement is a model's and cannot be asserted deterministically. What *is* tested about pass 2 is the part that stays deterministic — which cases it selects, and that truncation is always announced. ``` python3 test-advisory-scan.py # offline python3 test-advisory-scan.py --live # + historic report numbers ``` `audit-advisory-scan.py` re-derives the counts with a **separate** semver implementation and its own release fetch, then diffs against the scanner. Run it after changing classification; it is what caught the 12 undercounted redis CVEs. ## Known limits 1. **Versions must be supplied per image.** An image with no `--image` is not counted — the scan will not guess a version range it was not given. `/recipe-upgrade` passes one per image it bumped. 2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time. 3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was never pointed at. 4. **`NAME` matching is substring-against-source-name**, so a short or generic name can attach to more than one repo (`postgres` matches `discourse/discourse-postgres`). The primary source is claimed first and cannot be stolen. A name matching **nothing** is silently ignored — a typo costs coverage without warning. 5. **Release-note rescue (4c) trusts that naming implies fixing.** A release note that merely mentions a CVE would be read as fixing it. Pass 2's review table exists partly to catch this. 6. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed sources (visibly, but degraded).