cc-ci-orchestrator/cc-ci-plan/advisory-scan.SPEC.md

Advisory scan — specification

What cc-ci-plan/advisory-scan.py does, step by step, and why each step exists. This documents the implementation as it stands (2026-08-11); if you change the code, change this file in the same commit. cc-ci-plan/test-advisory-scan.py is the executable half of this spec — every rule below is asserted there.

Role. A per-recipe CVE detector run as a pre-step of /recipe-upgrade (step 2a). It is strictly additive: it never replaces the release-note reading the upgrade agent already does. The CVE count reported for a recipe is the union of what the agent read and what this scan found; the scan may never lower a count established by reading.

Why it exists. gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly report printed gitea's CVE count as 1, then none. The upgrade agent had read the GitHub release notes, which name neither — both were announced only in the vendor's blog security section — and the report then derived security content from those notes plus model knowledge, which predates the CVEs. Nothing in the pipeline queried an advisory source. This scan closes that hole.

---

Two passes

	Pass 1 — measure	Pass 2 — judge (--adjudicate)
Who	Pure Python, no model	The calling agent, a model
Does	Collects evidence and decides every case it can by arithmetic	Weighs the collected evidence on cases arithmetic cannot settle
Output	A count, or UNKNOWN	FIXED / NOT-FIXED / STILL-UNKNOWN per open case
Rule	Deterministic and reproducible	May only raise the count, never lower it

Prefer pass 1. Every case pass 1 decides is one that reproduces identically next week. Pass 2 exists for evidence that is prose rather than data — a fallback, not a co-equal stage. When a class of case keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c is exactly that: it moved 12 redis advisories out of pass 2 and into arithmetic.

---

Inputs

advisory-scan.py <recipe> [--from <version>] [--to <version>]
                          [--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]

advisory-scan.py <recipe> --compose-to <URL> [--compose-from <URL>]   # windows derived, not typed

Input	Meaning
<recipe>	Recipe name; selects cc-ci-plan/upstream/<recipe>.md (the per-recipe URL registry)
--from / --to	The primary app image's version window being upgraded across
--image NAME=FROM:TO	A sidecar image and the versions it moved between (repeatable, all in ONE call). NAME is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified.
--compose-to URL	Derive every window by diffing this compose against its baseline, instead of typing --from/--to/--image. Point it at a PR's compose.yml.
--compose-from URL	Baseline for the above. Default: the same repo's default branch, resolved from the API — never assumed to be main.
--adjudicate	Run pass 2: append the evidence dossier for judgement
--registry	Registry dir; also CCCI_UPSTREAM_REGISTRY
GITHUB_TOKEN / GITHUB_TOKEN_FILE	Read-only token; rate limit only (60/hr anonymous → 5000/hr). Default file /srv/cc-ci/.github-token, mode 600. Public advisories need no scopes.

Exit code is always 0 — this is informational. Failures are reported, never raised.

---

Pass 1 — deterministic

Step 1 — Collect source URLs from the registry

Read cc-ci-plan/upstream/<recipe>.md and extract every http(s)://… URL.

Trailing markdown punctuation is stripped (` ' " * . , ; : > )). The registry is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render ?` for no real reason.

Registry hygiene matters. The scan can only look where the registry points. Two classes of defect have been found and fixed by running it: a wrong URL (pgautoupgrade/pgautoupgrade, which 404s — the repo is pgautoupgrade/docker-pgautoupgrade) and a missing one (gitea's CVEs are announced at blog.gitea.com, which the registry didn't list). When a vendor publishes security notes somewhere the registry lacks, add it.

Step 2 — Query the sources

Three source classes, each recording its own status so "checked, none found" is never confused with "not checked".

2a. GitHub Security Advisories — PRIMARY

For every github.com/<owner>/<repo> URL in the registry: GET /repos/<owner>/<repo>/security-advisories.

Captured per advisory: cve_id, ghsa_id, severity, summary, description, published_at, and all vulnerabilities[] entries' vulnerable_version_range + patched_versions (joined ;).

• All entries, not just the first. An advisory carries one entry per patched release line — n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only vulnerabilities[0] silently dropped the line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window.
• Pagination via the Link rel="next" cursor, to exhaustion (cap 20 hops). This endpoint returns at most 100 rows and ignores ?page= — it re-returns the same rows, which silently truncates busy projects. discourse has 286 advisories; a single page cannot even cover one upgrade window.
• The description is kept from this response. It is already present here, and pass 2 needs the prose; re-fetching it per advisory would cost one request each.
• HTTP 404 ⇒ no-advisories-published — a benign absence (many sidecar images publish none), not a failure. Conflating the two pushed nearly every recipe to ? and destroyed the signal.

2b. Vendor release / security pages

Every other registry URL is fetched, HTML-stripped, and scanned for CVE-\d{4}-\d{4,7}, keeping ±160 characters of context per hit.

URLs containing <, >, {, }, VERSION, or vX.Y.Z are skipped as templates — they are human documentation (…/changelog/v<VERSION>/), not fetchable, and counting them as failures is wrong.

This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release page names neither.

When the page is a changelog organised by release, each CVE is attributed to the release heading it appears under (Changes with nginx 1.31.3, ## v1.31.3, …) and that becomes its fixed-in version. Without this, a project that publishes no advisory feed can never contribute a CVE:

nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes from nginx.org/en/CHANGES. Scraping ids out of it without attributing them to a release left them with no patched version, so they were never classifiable — and every nginx bump in the fleet reported 0 forever. nginx is a sidecar in most recipes. Measured: 1.31.1 → 1.31.3 fixes six CVEs (three in .2, three in .3); lasuite-docs#7 went 0 → 6 and lasuite-drive#6 went 0 → 3 on this alone.

A changelog CVE is tied to a window by the image name appearing in the page URL (window nginx ↔ nginx.org/...). A CVE found on a vendor page with no attributable release still has no version data, so pass 1 cannot place it — it goes to pass 2 (§6).

2c. OSV.dev — supplementary

Only when the recipe has an entry in OSV_PACKAGES (ecosystem + package) and a version is given.

Measured, not assumed. For the two gitea CVEs, OSV 404'd on both and returned only Go dependency advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword. Advisory databases lag the vendor, which is why 2a and 2b lead and this is supplementary.

Step 3 — Union

All findings merge into one CVE map: id → {sources[], severity, ghsa, vulnerable_range, patched, description, published_at, url, cvss, context}. A CVE seen by several sources keeps them all.

Step 4 — Classify against the upgrade window

Two invariants govern this step, both learned from a wrong answer in production.

A. Every image is judged by its OWN versions. The app repo uses --from/--to; each sidecar uses its own --image NAME=FROM:TO. Pass them all in ONE invocation — the count is a union across images, and the UNKNOWN guarantee in B only holds when a single run sees every one. An image with no window is not classified; its advisories are listed as unclassified so they stay visible without inflating the count. Each window is classified independently, so one may use version ranges while another falls back to dates. Why: discourse once reported 133, of which 34 were redis CVEs — including CVE-2021-21309, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically inside discourse's 3.5.3 → 2026.7.1 range. The fix is not to ignore sidecars but to give each one the versions it actually moved through: with --image redis=7.4:8.10, discourse scores 140 = 123 (app, by date) + 17 (redis), and the redis seventeen include CVE-2025-49844, critical, invisible while sidecars went uncounted.

B. Never emit a number you cannot justify. If no method can order a window, the count is null / UNKNOWN, never 0. A 0 in a security column asserts safety. Equally, an advisory that cannot be judged is indeterminate (§4d) — never silently counted as "not fixed".

3b. Deriving the windows from a compose diff (--compose-to)

Typing --from/--to/--image by hand means someone has to remember that the recipe also bumped its redis. That is how sidecar CVEs went uncounted for months. This mode reads the windows off the diff:

1. Fetch both compose files (baseline = the repo's default branch from the API, since several recipes keep a stale main beside a live master).
2. Parse {service: (image-repo, tag)} — keyed by service, not image repo, because an upgrade may change the repo itself (plausible moved plausible/analytics → ghcr.io/plausible/community-edition; keyed by repo that reads as one image vanishing and an unrelated one appearing, losing the app window entirely).
3. Every service whose tag or repo changed becomes a window. The app service drives --from/--to (coop-cloud convention: it is the recipe's primary image); the rest become --image windows. Unchanged images produce no window — inventing one would be a false count.
4. The derived windows are printed to stderr before the scan, so the inputs are auditable.

Image names are matched against advisory sources both ways — an image name is often longer than its source repo (clickhouse/clickhouse-server vs ClickHouse/ClickHouse) and sometimes shorter (redis vs redis/redis).

Verified on plausible PR #5: from the compose URL alone it derives v2.0.0 → v3.2.1 plus clickhouse-server 23.4.2.11-alpine → 24.12-alpine, and reports 6 — identical to the hand-specified args.

--from/--to/--image remain available for finer-grained checks (scanning a window that is not a literal compose diff, e.g. "what would the compatibility-safe target fix?").

4a. By patched version (preferred — exact)

patched_versions is a range expression (">= 2.18.1"), possibly several joined by ;. Extract every version-looking token; the advisory is fixed-by-this-upgrade if any patched version p satisfies from < p <= to — exclusive lower (a fix already in the version you were on is not this upgrade's doing), inclusive upper.

Comparison is zero-padded to equal length, so 18 == 18.0 == 18.0.0 as semver means it. Without padding, plain tuple order makes (18,) < (18,0), i.e. a fix in 18.0 falls outside a window ending at 18 — and bare major tags are the norm for sidecars (postgres:18, redis:8-alpine). Padding is permissive at the lower bound and conservative at the upper: with to = 18, a fix in 18.5 is not counted, because nothing proves which 18.x a floating tag resolved to.

4b. By advisory publish date (fallback — temporal)

Used only when 4a cannot be trusted: a version-scheme change, detected as the leading version component jumping by ≥ SCHEME_JUMP (100) — e.g. semver 3.5.3 → calver 2026.7.1.

Version strings are unorderable across such a jump (2025.12.2 compares "newer" than 3.5.3 while shipping earlier), but release dates always order. So:

1. Resolve --from and --to to git tag dates on that source (tries v<version> then <version>; annotated tag → tagger date, else commit date).
2. An advisory counts as fixed when date_from < published_at <= date_to — the same exclusive/inclusive boundaries as 4a, so the two methods agree at the edges.

This reproduces, automatically, the hand count that established discourse 3.5.3 (2025-12-30) → 2026.7.1 (2026-07-31) = 123 CVEs.

Assumption: the vendor publishes advisories at fix time (true for discourse). The count includes first-party plugin advisories where the vendor files them on the same repo — which is why a plugin-rich project scores far higher than a monolith, not a statement about relative security.

4c. By release notes naming the CVE (rescue — for advisories with no fix version)

Applied to advisories 4a/4b could not decide, before giving up on them. Fetch the source repo's GitHub releases (cached per repo, 4 pages) and find every tag whose notes name the CVE id. If any such tag falls inside the window by 4a's rule, the advisory is fixed by this upgrade, and the naming tags are recorded in fix_versions_from_release_notes as the citation.

Why this is not optional. Vendors routinely publish an advisory with patched_versions: "TBD" and then name the CVE in the release notes of every branch that got the fix. All 12 redis advisories crossed by discourse's redis bump are exactly this shape — TBD, or a placeholder like 7.4.X, with an open-ended vulnerable_version_range (All, >= 7.0.0) — yet each is named in concrete releases (CVE-2025-32023 → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Without this method discourse's redis contribution reads 5; with it, 17. Six of the twelve are high severity.

4d. Otherwise — indeterminate, not excluded

An advisory from a windowed source that none of 4a–4c could decide — no usable patched_versions, an open-ended vulnerable range, and no release note naming it — is recorded as indeterminate. It is:

• not added to the count (nothing justifies counting it), and
• not treated as unaffected (nothing justifies dismissing it either).

It is listed prominently, the headline reads "(at least — see undetermined below)", and it becomes an input to pass 2. Silently excluding these is the same defect class as printing 0 for an unscanned recipe, one level down.

4e. Unorderable window

If neither 4a nor 4b can order a window at all, count_known = false, cve_count_fixed = null, and the headline reads "CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count" with an explicit "This is NOT zero". If any requested window is unorderable the whole count is suppressed; a partial number would understate a security figure.

Step 5 — Output

Markdown (default) for pasting into the per-recipe upgrade log, or --json.

Field	Meaning
cve_count_fixed	Union across all windows, or null if any window was unorderable
count_known	Distinguishes "counted zero" from "could not count"
cve_count_indeterminate / indeterminate[]	Judged by nothing; a floor marker on the count
resolved_by_release_notes	CVE → the tags that named it (4c citations)
windows	Every source classified, with its from/to
classified_by	Per source: which method decided it
date_window	Per source, when 4b was used
fixed_by_this_upgrade[]	CVE ids, with severity / GHSA / fixed-in per id
unclassified[]	Seen but not attributable (other images, or vendor-page-only)
sources[] / sources_failed[] / sources_benign[]	Per-source status; only genuine failures in failed

---

Pass 2 — adjudication (--adjudicate)

Step 6 — What gets judged

Two kinds of open case, both real gaps rather than noise:

1. Indeterminate (§4d) — from an image with a window, but no fix version is knowable anywhere.
2. Vendor-page-only — a CVE seen only on a vendor security page, with no structured advisory behind it. gitea's two CVSS-9.8 RCEs are this shape. They carry no version data, so no arithmetic can place them, but the page prose usually states the fixed release.

Step 7 — The evidence dossier

Pass 1 collects; pass 2 judges. Nothing in the dossier interprets — it assembles what was measured, so the judgement is made against evidence rather than recollection. That distinction is the whole point: the original failure was a report leaning on model knowledge that predated the CVEs, with no source queried at all.

Per open case: severity, CVSS, sources, its window, why it is undecided, patched_versions and vulnerable_version_range as published, summary, full description, references, affected ranges with first_patched_version, and every release tag naming the CVE — whether or not in window, since the model may reason about branch lines the arithmetic deliberately will not.

Pass 2 also sees every decision pass 1 made — a compact table of counted and excluded advisories with the evidence behind each verdict. A deterministic verdict can still be wrong (a mis-parsed range, a release note that mentions a CVE without fixing it), and only a reader with the evidence in front of it can catch that. Silence means agreement.

Step 8 — The verdict contract

For each open case: FIXED / NOT-FIXED / STILL-UNKNOWN, each with a one-line reason citing the evidence shown. Every FIXED is added to the recipe's count — pass 1's number is a floor, not a total. If the evidence does not settle it, STILL-UNKNOWN: do not infer from memory of the project, and never record an undecided CVE as unaffected.

No silent caps. MAX_ADJUDICATE (25) and MAX_REVIEW_ROWS (400) bound the output; whenever either truncates, the block says how many were dropped and that the unshown remain undetermined.

---

How consumers must read it

/recipe-report renders the cve column from the union of this scan and the agent's own reading:

• a clean scan → its number, including 0;
• failed sources, or UNKNOWN → ?, never none — a blank reads as "clean", which is exactly how two CVSS-9.8 gitea RCEs were published as none;
• no upgrade this run → 0, not ? — nothing an upgrade could have fixed;
• a count with indeterminate advisories → the number is a floor; say so in the notes;
• benign notes → never ?.

? must stay rare: it means we tried and could not tell, not we did not look. A rash of ? is a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect in this tool or stale registry data.

Testing

test-advisory-scan.py — 58 offline tests (fixtures, no network) plus 6 live regressions against the counts published in week-2026-08-07. The offline tier covers pass 1 only, by design: pass 2's judgement is a model's and cannot be asserted deterministically. What is tested about pass 2 is the part that stays deterministic — which cases it selects, and that truncation is always announced.

python3 test-advisory-scan.py          # offline
python3 test-advisory-scan.py --live   # + historic report numbers

audit-advisory-scan.py re-derives the counts with a separate semver implementation and its own release fetch, then diffs against the scanner. Run it after changing classification; it is what caught the 12 undercounted redis CVEs.

Known limits

1. Versions must be supplied per image. An image with no --image is not counted — the scan will not guess a version range it was not given. /recipe-upgrade passes one per image it bumped.
2. Date-based counts are temporal, not exact — they assume publish-at-fix-time.
3. Registry-bound. Unlisted vendor security pages are invisible; the scan cannot know what it was never pointed at.
4. NAME matching is substring-against-source-name, so a short or generic name can attach to more than one repo (postgres matches discourse/discourse-postgres). The primary source is claimed first and cannot be stolen. A name matching nothing is silently ignored — a typo costs coverage without warning.
5. Release-note rescue (4c) trusts that naming implies fixing. A release note that merely mentions a CVE would be read as fixing it. Pass 2's review table exists partly to catch this.
6. Rate limit without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed sources (visibly, but degraded).