cc-ci-orchestrator/cc-ci-plan/advisory-scan.SPEC.md

# Advisory scan — specification

What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the
implementation as it stands (2026-08-11); if you change the code, change this file in the same commit.
`cc-ci-plan/test-advisory-scan.py` is the executable half of this spec — every rule below is asserted
there.

**Role.** A per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a). It is
**strictly additive**: it never replaces the release-note reading the upgrade agent already does. The
CVE count reported for a recipe is the **union** of what the agent read and what this scan found; the
scan may never *lower* a count established by reading.

**Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly
report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release
notes*, which name neither — both were announced only in the vendor's blog security section — and the
report then derived security content from those notes plus model knowledge, which predates the CVEs.
Nothing in the pipeline queried an advisory source. This scan closes that hole.

---

## Two passes

| | Pass 1 — measure | Pass 2 — judge (`--adjudicate`) |
|---|---|---|
| Who | Pure Python, no model | The calling agent, a model |
| Does | Collects evidence and decides every case it can by arithmetic | Weighs the collected evidence on cases arithmetic cannot settle |
| Output | A count, or `UNKNOWN` | FIXED / NOT-FIXED / STILL-UNKNOWN per open case |
| Rule | Deterministic and reproducible | May only **raise** the count, never lower it |

**Prefer pass 1.** Every case pass 1 decides is one that reproduces identically next week. Pass 2 exists
for evidence that is prose rather than data — a fallback, not a co-equal stage. When a class of case
keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c is exactly that: it moved
12 redis advisories out of pass 2 and into arithmetic.

---

## Inputs

```
advisory-scan.py <recipe> [--from <version>] [--to <version>]
                          [--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
```

| Input | Meaning |
|---|---|
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
| `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |

Exit code is always 0 — this is informational. Failures are *reported*, never raised.

---

# Pass 1 — deterministic

## Step 1 — Collect source URLs from the registry

Read `cc-ci-plan/upstream/<recipe>.md` and extract every `http(s)://…` URL.

**Trailing markdown punctuation is stripped** (`` ` `` `'` `"` `*` `.` `,` `;` `:` `>` `)`). The registry
is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of
`https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render `?` for no real reason.

> **Registry hygiene matters.** The scan can only look where the registry points. Two classes of defect
> have been found and fixed by running it: a **wrong URL** (`pgautoupgrade/pgautoupgrade`, which 404s —
> the repo is `pgautoupgrade/docker-pgautoupgrade`) and a **missing** one (gitea's CVEs are announced at
> `blog.gitea.com`, which the registry didn't list). When a vendor publishes security notes somewhere
> the registry lacks, add it.

## Step 2 — Query the sources

Three source classes, each recording **its own status** so *"checked, none found"* is never confused
with *"not checked"*.

### 2a. GitHub Security Advisories — PRIMARY

For every `github.com/<owner>/<repo>` URL in the registry:
`GET /repos/<owner>/<repo>/security-advisories`.

Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, **`description`**, `published_at`,
and **all** `vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined `;`).

- **All entries, not just the first.** An advisory carries one entry **per patched release line** —
  n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the
  line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window.
- **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns
  at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy
  projects. discourse has 286 advisories; a single page cannot even cover one upgrade window.
- **The description is kept from this response.** It is already present here, and pass 2 needs the
  prose; re-fetching it per advisory would cost one request each.
- **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not**
  a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal.

### 2b. Vendor release / security pages

Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160
characters of context per hit.

URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as templates** — they are
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.

This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
page names neither. A CVE found **only** here carries no version data, so pass 1 cannot place it — it
goes to pass 2 (§6).

### 2c. OSV.dev — supplementary

Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.

> **Measured, not assumed.** For the two gitea CVEs, OSV **404'd on both** and returned only Go
> *dependency* advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword.
> **Advisory databases lag the vendor**, which is why 2a and 2b lead and this is supplementary.

## Step 3 — Union

All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched,
description, published_at, url, cvss, context}`. A CVE seen by several sources keeps them all.

## Step 4 — Classify against the upgrade window

Two invariants govern this step, both learned from a wrong answer in production.

> **A. Every image is judged by its OWN versions.** The app repo uses `--from/--to`; each sidecar uses
> its own `--image NAME=FROM:TO`. **Pass them all in ONE invocation** — the count is a union across
> images, and the UNKNOWN guarantee in B only holds when a single run sees every one. An image with no
> window is **not** classified; its advisories are listed as unclassified so they stay visible without
> inflating the count. Each window is classified independently, so one may use version ranges while
> another falls back to dates.
> *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including
> `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically
> inside discourse's `3.5.3 → 2026.7.1` range. The fix is not to ignore sidecars but to give each one
> the versions it actually moved through: with `--image redis=7.4:8.10`, discourse scores
> **140 = 123 (app, by date) + 17 (redis)**, and the redis seventeen include `CVE-2025-49844`,
> **critical**, invisible while sidecars went uncounted.
>
> **B. Never emit a number you cannot justify.** If no method can order a window, the count is
> `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
> cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".

### 4a. By patched version (preferred — exact)

`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this
upgrade's doing), inclusive upper.

**Comparison is zero-padded to equal length**, so `18` == `18.0` == `18.0.0` as semver means it.
Without padding, plain tuple order makes `(18,) < (18,0)`, i.e. a fix in 18.0 falls *outside* a window
ending at 18 — and bare major tags are the norm for sidecars (`postgres:18`, `redis:8-alpine`). Padding
is permissive at the lower bound and conservative at the upper: with `to = 18`, a fix in `18.5` is
**not** counted, because nothing proves which 18.x a floating tag resolved to.

### 4b. By advisory publish date (fallback — temporal)

Used **only** when 4a cannot be trusted: a **version-scheme change**, detected as the leading version
component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver `2026.7.1`.

Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while
shipping earlier), but **release dates always order**. So:

1. Resolve `--from` and `--to` to **git tag dates** on that source (tries `v<version>` then
   `<version>`; annotated tag → tagger date, else commit date).
2. An advisory counts as fixed when `date_from < published_at <= date_to` — the same exclusive/inclusive
   boundaries as 4a, so the two methods agree at the edges.

This reproduces, automatically, the hand count that established discourse `3.5.3` (2025-12-30) →
`2026.7.1` (2026-07-31) = **123 CVEs**.

*Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes
**first-party plugin advisories** where the vendor files them on the same repo — which is why a
plugin-rich project scores far higher than a monolith, not a statement about relative security.

### 4c. By release notes naming the CVE (rescue — for advisories with no fix version)

Applied to advisories 4a/4b could not decide, **before** giving up on them. Fetch the source repo's
GitHub **releases** (cached per repo, 4 pages) and find every tag whose notes **name the CVE id**. If
any such tag falls inside the window by 4a's rule, the advisory is fixed by this upgrade, and the
naming tags are recorded in `fix_versions_from_release_notes` as the citation.

> **Why this is not optional.** Vendors routinely publish an advisory with `patched_versions: "TBD"`
> and then name the CVE in the release notes of every branch that got the fix. **All 12** redis
> advisories crossed by discourse's redis bump are exactly this shape — `TBD`, or a placeholder like
> `7.4.X`, with an open-ended `vulnerable_version_range` (`All`, `>= 7.0.0`) — yet each is named in
> concrete releases (`CVE-2025-32023` → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Without this method
> discourse's redis contribution reads 5; with it, 17. Six of the twelve are high severity.

### 4d. Otherwise — indeterminate, not excluded

An advisory from a **windowed** source that none of 4a–4c could decide — no usable `patched_versions`,
an open-ended vulnerable range, and no release note naming it — is recorded as **indeterminate**. It is:

- **not** added to the count (nothing justifies counting it), and
- **not** treated as unaffected (nothing justifies dismissing it either).

It is listed prominently, the headline reads *"(at least — see undetermined below)"*, and it becomes an
input to pass 2. Silently excluding these is the same defect class as printing `0` for an unscanned
recipe, one level down.

### 4e. Unorderable window

If neither 4a nor 4b can order a window at all, `count_known = false`, `cve_count_fixed = null`, and the
headline reads **"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an
explicit *"This is NOT zero"*. If **any** requested window is unorderable the whole count is suppressed;
a partial number would understate a security figure.

## Step 5 — Output

Markdown (default) for pasting into the per-recipe upgrade log, or `--json`.

| Field | Meaning |
|---|---|
| `cve_count_fixed` | Union across all windows, or **`null`** if any window was unorderable |
| `count_known` | Distinguishes "counted zero" from "could not count" |
| `cve_count_indeterminate` / `indeterminate[]` | Judged by nothing; a **floor marker** on the count |
| `resolved_by_release_notes` | CVE → the tags that named it (4c citations) |
| `windows` | Every source classified, with its from/to |
| `classified_by` | **Per source**: which method decided it |
| `date_window` | **Per source**, when 4b was used |
| `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id |
| `unclassified[]` | Seen but not attributable (other images, or vendor-page-only) |
| `sources[]` / `sources_failed[]` / `sources_benign[]` | Per-source status; only genuine failures in `failed` |

---

# Pass 2 — adjudication (`--adjudicate`)

## Step 6 — What gets judged

Two kinds of open case, both real gaps rather than noise:

1. **Indeterminate** (§4d) — from an image *with* a window, but no fix version is knowable anywhere.
2. **Vendor-page-only** — a CVE seen only on a vendor security page, with no structured advisory
   behind it. **gitea's two CVSS-9.8 RCEs are this shape.** They carry no version data, so no
   arithmetic can place them, but the page prose usually states the fixed release.

## Step 7 — The evidence dossier

Pass 1 collects; pass 2 judges. Nothing in the dossier interprets — it assembles what was *measured*,
so the judgement is made against evidence rather than recollection. That distinction is the whole
point: the original failure was a report leaning on model knowledge that predated the CVEs, with no
source queried at all.

Per open case: severity, CVSS, sources, its window, why it is undecided, `patched_versions` and
`vulnerable_version_range` **as published**, summary, full description, references, affected ranges
with `first_patched_version`, and **every release tag naming the CVE** — whether or not in window, since
the model may reason about branch lines the arithmetic deliberately will not.

**Pass 2 also sees every decision pass 1 made** — a compact table of counted and excluded advisories
with the evidence behind each verdict. A deterministic verdict can still be wrong (a mis-parsed range,
a release note that mentions a CVE without fixing it), and only a reader with the evidence in front of
it can catch that. Silence means agreement.

## Step 8 — The verdict contract

For each open case: **FIXED** / **NOT-FIXED** / **STILL-UNKNOWN**, each with a one-line reason
**citing the evidence shown**. Every FIXED is added to the recipe's count — pass 1's number is a floor,
not a total. If the evidence does not settle it, **STILL-UNKNOWN**: do not infer from memory of the
project, and never record an undecided CVE as unaffected.

**No silent caps.** `MAX_ADJUDICATE` (25) and `MAX_REVIEW_ROWS` (400) bound the output; whenever either
truncates, the block says how many were dropped and that the unshown remain undetermined.

---

## How consumers must read it

`/recipe-report` renders the `cve` column from the **union** of this scan and the agent's own reading:

- a clean scan → its number, **including `0`**;
- **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly
  how two CVSS-9.8 gitea RCEs were published as `none`;
- **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed;
- a count with **indeterminate advisories** → the number is a floor; say so in the notes;
- benign notes → never `?`.

`?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is
a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect
in this tool or stale registry data.

## Testing

`test-advisory-scan.py` — 58 offline tests (fixtures, no network) plus 6 live regressions against the
counts published in week-2026-08-07. **The offline tier covers pass 1 only, by design**: pass 2's
judgement is a model's and cannot be asserted deterministically. What *is* tested about pass 2 is the
part that stays deterministic — which cases it selects, and that truncation is always announced.

```
python3 test-advisory-scan.py          # offline
python3 test-advisory-scan.py --live   # + historic report numbers
```

`audit-advisory-scan.py` re-derives the counts with a **separate** semver implementation and its own
release fetch, then diffs against the scanner. Run it after changing classification; it is what caught
the 12 undercounted redis CVEs.

## Known limits

1. **Versions must be supplied per image.** An image with no `--image` is not counted — the scan will
   not guess a version range it was not given. `/recipe-upgrade` passes one per image it bumped.
2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time.
3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was
   never pointed at.
4. **`NAME` matching is substring-against-source-name**, so a short or generic name can attach to more
   than one repo (`postgres` matches `discourse/discourse-postgres`). The primary source is claimed
   first and cannot be stolen. A name matching **nothing** is silently ignored — a typo costs coverage
   without warning.
5. **Release-note rescue (4c) trusts that naming implies fixing.** A release note that merely mentions a
   CVE would be read as fixing it. Pass 2's review table exists partly to catch this.
6. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed
   sources (visibly, but degraded).