docs: spec for the advisory scanner's CVE detection
Step-by-step specification of cc-ci-plan/advisory-scan.py: inputs, the three source classes and why each is ranked where it is, the union, both classification paths (patched-version ranges, and the advisory-publish-date fallback for version-scheme changes), the output contract, and how /recipe-report must read it. Each rule records the production wrong answer that motivated it — the false 133 from cross-image counting, the n8n misclassification from reading only vulnerabilities[0], the '?' sprawl from url punctuation and benign-404s, and the 'never emit 0 for an undetermined count' rule. Claims cross-checked against the implementation. Keep this file in the same commit as any behaviour change.
This commit is contained in:
autonomic-bot
@@ -0,0 +1,184 @@
|
||||
# Advisory scan — specification
|
||||
|
||||
What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the
|
||||
implementation as it stands (2026-08-10); if you change the code, change this file in the same commit.
|
||||
|
||||
**Role.** A deterministic, per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a).
|
||||
It is **strictly additive**: it never replaces the release-note reading the upgrade agent already does.
|
||||
The CVE count reported for a recipe is the **union** of what the agent read and what this scan found;
|
||||
the scan may never *lower* a count established by reading.
|
||||
|
||||
**Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly
|
||||
report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release
|
||||
notes*, which name neither — both were announced only in the vendor's blog security section — and the
|
||||
report then derived security content from those notes plus model knowledge, which predates the CVEs.
|
||||
Nothing in the pipeline queried an advisory source. This scan closes that hole.
|
||||
|
||||
---
|
||||
|
||||
## Inputs
|
||||
|
||||
```
|
||||
advisory-scan.py <recipe> [--from <version>] [--to <version>] [--json] [--registry DIR]
|
||||
```
|
||||
|
||||
| Input | Meaning |
|
||||
|---|---|
|
||||
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
|
||||
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
|
||||
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
|
||||
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
|
||||
|
||||
Exit code is always 0 — this is informational. Failures are *reported*, never raised.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Collect source URLs from the registry
|
||||
|
||||
Read `cc-ci-plan/upstream/<recipe>.md` and extract every `http(s)://…` URL.
|
||||
|
||||
**Trailing markdown punctuation is stripped** (`` ` `` `'` `"` `*` `.` `,` `;` `:` `>` `)`). The registry
|
||||
is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of
|
||||
`https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render `?` for no real reason.
|
||||
|
||||
> **Registry hygiene matters.** The scan can only look where the registry points. Two classes of defect
|
||||
> have been found and fixed by running it: a **wrong URL** (`pgautoupgrade/pgautoupgrade`, which 404s —
|
||||
> the repo is `pgautoupgrade/docker-pgautoupgrade`) and a **missing** one (gitea's CVEs are announced at
|
||||
> `blog.gitea.com`, which the registry didn't list). When a vendor publishes security notes somewhere
|
||||
> the registry lacks, add it.
|
||||
|
||||
## Step 2 — Query the sources
|
||||
|
||||
Three source classes, each recording **its own status** so *"checked, none found"* is never confused
|
||||
with *"not checked"*.
|
||||
|
||||
### 2a. GitHub Security Advisories — PRIMARY
|
||||
|
||||
For every `github.com/<owner>/<repo>` URL in the registry:
|
||||
`GET /repos/<owner>/<repo>/security-advisories`.
|
||||
|
||||
Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, `published_at`, and **all**
|
||||
`vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined with `;`).
|
||||
|
||||
- **All entries, not just the first.** An advisory carries one entry **per patched release line** —
|
||||
n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the
|
||||
line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window.
|
||||
- **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns
|
||||
at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy
|
||||
projects. discourse has 286 advisories; a single page cannot even cover one upgrade window.
|
||||
- **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not**
|
||||
a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal.
|
||||
|
||||
This source is primary because it carries **severity and version ranges**, making "fixed by *this*
|
||||
upgrade" computable rather than guessed.
|
||||
|
||||
### 2b. Vendor release / security pages
|
||||
|
||||
Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160
|
||||
characters of context per hit.
|
||||
|
||||
URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as templates** — they are
|
||||
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.
|
||||
|
||||
This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
|
||||
page names neither.
|
||||
|
||||
### 2c. OSV.dev — supplementary
|
||||
|
||||
Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.
|
||||
|
||||
> **Measured, not assumed.** For the two gitea CVEs, OSV **404'd on both** and returned only Go
|
||||
> *dependency* advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword.
|
||||
> **Advisory databases lag the vendor**, which is why 2a and 2b lead and this is supplementary.
|
||||
|
||||
## Step 3 — Union
|
||||
|
||||
All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched,
|
||||
published_at, context}`. A CVE seen by several sources keeps them all.
|
||||
|
||||
## Step 4 — Classify against the upgrade window
|
||||
|
||||
Two invariants govern this step, both learned from a wrong answer in production.
|
||||
|
||||
> **A. The window belongs to ONE image.** Only advisories from the **primary** source (the first
|
||||
> `github-advisories:` source, i.e. the app repo the registry lists first) are classified. Sidecar
|
||||
> advisories are recorded as unclassified so they stay visible without inflating the count.
|
||||
> *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including
|
||||
> `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically
|
||||
> inside discourse's `3.5.3 → 2026.7.1` range.
|
||||
>
|
||||
> **B. Never emit a number you cannot justify.** If neither method below can order the window, the
|
||||
> count is `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety.
|
||||
|
||||
### 4a. By patched version (preferred — exact)
|
||||
|
||||
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
|
||||
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
|
||||
satisfies `from < p <= to`, using a loose numeric key (leading integers per dot-part).
|
||||
|
||||
### 4b. By advisory publish date (fallback — temporal)
|
||||
|
||||
Used **only** when 4a cannot be trusted: a **version-scheme change**, detected as the leading version
|
||||
component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver `2026.7.1`.
|
||||
|
||||
Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while
|
||||
shipping earlier), but **release dates always order**. So:
|
||||
|
||||
1. Resolve `--from` and `--to` to **git tag dates** on the primary repo (tries `v<version>` then
|
||||
`<version>`; annotated tag → tagger date, else commit date).
|
||||
2. An advisory counts as fixed when `date_from < published_at <= date_to`.
|
||||
|
||||
Both the method and the resolved window appear in the output. This reproduces, automatically, the hand
|
||||
count that established discourse `3.5.3` (2025-12-30) → `2026.7.1` (2026-07-31) = **123 CVEs**.
|
||||
|
||||
*Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes
|
||||
**first-party plugin advisories** where the vendor files them on the same repo — which is why a
|
||||
plugin-rich project scores far higher than a monolith, not a statement about relative security.
|
||||
|
||||
### 4c. Otherwise
|
||||
|
||||
`count_known = false`, `cve_count_fixed = null`, and the markdown headline reads
|
||||
**"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an explicit
|
||||
*"This is NOT zero"*.
|
||||
|
||||
## Step 5 — Output
|
||||
|
||||
Markdown (default) for pasting into the per-recipe upgrade log, or `--json`.
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `cve_count_fixed` | Count, or **`null`** when undeterminable |
|
||||
| `count_known` | Distinguishes "counted zero" from "could not count" |
|
||||
| `classified_by` | `patched version ranges` or `advisory publish date (version scheme changed)` |
|
||||
| `date_window` | The resolved date window, when 4b was used |
|
||||
| `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id |
|
||||
| `unclassified[]` | Seen but not attributable to this window (incl. other images) |
|
||||
| `sources[]` | Every source with its own status |
|
||||
| `sources_failed[]` | **Genuine** failures only |
|
||||
| `sources_benign[]` | `no-advisories-published`, `skipped: template URL` |
|
||||
|
||||
---
|
||||
|
||||
## How consumers must read it
|
||||
|
||||
`/recipe-report` renders the `cve` column from the **union** of this scan and the agent's own reading:
|
||||
|
||||
- a clean scan → its number, **including `0`**;
|
||||
- **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly
|
||||
how two CVSS-9.8 gitea RCEs were published as `none`;
|
||||
- **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed;
|
||||
- benign notes → never `?`.
|
||||
|
||||
`?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is
|
||||
a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect
|
||||
in this tool or stale registry data.
|
||||
|
||||
## Known limits
|
||||
|
||||
1. **One window per scan.** Sidecar bumps (redis, postgres) are not counted; they appear as
|
||||
unclassified. Per-image windows would be the natural extension.
|
||||
2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time.
|
||||
3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was
|
||||
never pointed at.
|
||||
4. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed
|
||||
sources (visibly, but degraded).
|
||||
Reference in New Issue
Block a user