From 78ae2be8ae02e9e69e2be59dcfa72cbe46e283d3 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Mon, 10 Aug 2026 21:55:37 +0000 Subject: [PATCH] docs: spec for the advisory scanner's CVE detection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step-by-step specification of cc-ci-plan/advisory-scan.py: inputs, the three source classes and why each is ranked where it is, the union, both classification paths (patched-version ranges, and the advisory-publish-date fallback for version-scheme changes), the output contract, and how /recipe-report must read it. Each rule records the production wrong answer that motivated it — the false 133 from cross-image counting, the n8n misclassification from reading only vulnerabilities[0], the '?' sprawl from url punctuation and benign-404s, and the 'never emit 0 for an undetermined count' rule. Claims cross-checked against the implementation. Keep this file in the same commit as any behaviour change. --- cc-ci-plan/advisory-scan.SPEC.md | 184 +++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 cc-ci-plan/advisory-scan.SPEC.md diff --git a/cc-ci-plan/advisory-scan.SPEC.md b/cc-ci-plan/advisory-scan.SPEC.md new file mode 100644 index 0000000..7b665d9 --- /dev/null +++ b/cc-ci-plan/advisory-scan.SPEC.md @@ -0,0 +1,184 @@ +# Advisory scan — specification + +What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the +implementation as it stands (2026-08-10); if you change the code, change this file in the same commit. + +**Role.** A deterministic, per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a). +It is **strictly additive**: it never replaces the release-note reading the upgrade agent already does. +The CVE count reported for a recipe is the **union** of what the agent read and what this scan found; +the scan may never *lower* a count established by reading. + +**Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly +report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release +notes*, which name neither — both were announced only in the vendor's blog security section — and the +report then derived security content from those notes plus model knowledge, which predates the CVEs. +Nothing in the pipeline queried an advisory source. This scan closes that hole. + +--- + +## Inputs + +``` +advisory-scan.py [--from ] [--to ] [--json] [--registry DIR] +``` + +| Input | Meaning | +|---|---| +| `` | Recipe name; selects `cc-ci-plan/upstream/.md` (the per-recipe URL registry) | +| `--from` / `--to` | The **primary app image's** version window being upgraded across | +| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` | +| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. | + +Exit code is always 0 — this is informational. Failures are *reported*, never raised. + +--- + +## Step 1 — Collect source URLs from the registry + +Read `cc-ci-plan/upstream/.md` and extract every `http(s)://…` URL. + +**Trailing markdown punctuation is stripped** (`` ` `` `'` `"` `*` `.` `,` `;` `:` `>` `)`). The registry +is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of +`https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render `?` for no real reason. + +> **Registry hygiene matters.** The scan can only look where the registry points. Two classes of defect +> have been found and fixed by running it: a **wrong URL** (`pgautoupgrade/pgautoupgrade`, which 404s — +> the repo is `pgautoupgrade/docker-pgautoupgrade`) and a **missing** one (gitea's CVEs are announced at +> `blog.gitea.com`, which the registry didn't list). When a vendor publishes security notes somewhere +> the registry lacks, add it. + +## Step 2 — Query the sources + +Three source classes, each recording **its own status** so *"checked, none found"* is never confused +with *"not checked"*. + +### 2a. GitHub Security Advisories — PRIMARY + +For every `github.com//` URL in the registry: +`GET /repos///security-advisories`. + +Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, `published_at`, and **all** +`vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined with `;`). + +- **All entries, not just the first.** An advisory carries one entry **per patched release line** — + n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the + line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window. +- **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns + at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy + projects. discourse has 286 advisories; a single page cannot even cover one upgrade window. +- **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not** + a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal. + +This source is primary because it carries **severity and version ranges**, making "fixed by *this* +upgrade" computable rather than guessed. + +### 2b. Vendor release / security pages + +Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160 +characters of context per hit. + +URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as templates** — they are +human documentation (`…/changelog/v/`), not fetchable, and counting them as failures is wrong. + +This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release +page names neither. + +### 2c. OSV.dev — supplementary + +Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given. + +> **Measured, not assumed.** For the two gitea CVEs, OSV **404'd on both** and returned only Go +> *dependency* advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword. +> **Advisory databases lag the vendor**, which is why 2a and 2b lead and this is supplementary. + +## Step 3 — Union + +All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched, +published_at, context}`. A CVE seen by several sources keeps them all. + +## Step 4 — Classify against the upgrade window + +Two invariants govern this step, both learned from a wrong answer in production. + +> **A. The window belongs to ONE image.** Only advisories from the **primary** source (the first +> `github-advisories:` source, i.e. the app repo the registry lists first) are classified. Sidecar +> advisories are recorded as unclassified so they stay visible without inflating the count. +> *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including +> `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically +> inside discourse's `3.5.3 → 2026.7.1` range. +> +> **B. Never emit a number you cannot justify.** If neither method below can order the window, the +> count is `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. + +### 4a. By patched version (preferred — exact) + +`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract +every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p` +satisfies `from < p <= to`, using a loose numeric key (leading integers per dot-part). + +### 4b. By advisory publish date (fallback — temporal) + +Used **only** when 4a cannot be trusted: a **version-scheme change**, detected as the leading version +component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver `2026.7.1`. + +Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while +shipping earlier), but **release dates always order**. So: + +1. Resolve `--from` and `--to` to **git tag dates** on the primary repo (tries `v` then + ``; annotated tag → tagger date, else commit date). +2. An advisory counts as fixed when `date_from < published_at <= date_to`. + +Both the method and the resolved window appear in the output. This reproduces, automatically, the hand +count that established discourse `3.5.3` (2025-12-30) → `2026.7.1` (2026-07-31) = **123 CVEs**. + +*Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes +**first-party plugin advisories** where the vendor files them on the same repo — which is why a +plugin-rich project scores far higher than a monolith, not a statement about relative security. + +### 4c. Otherwise + +`count_known = false`, `cve_count_fixed = null`, and the markdown headline reads +**"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an explicit +*"This is NOT zero"*. + +## Step 5 — Output + +Markdown (default) for pasting into the per-recipe upgrade log, or `--json`. + +| Field | Meaning | +|---|---| +| `cve_count_fixed` | Count, or **`null`** when undeterminable | +| `count_known` | Distinguishes "counted zero" from "could not count" | +| `classified_by` | `patched version ranges` or `advisory publish date (version scheme changed)` | +| `date_window` | The resolved date window, when 4b was used | +| `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id | +| `unclassified[]` | Seen but not attributable to this window (incl. other images) | +| `sources[]` | Every source with its own status | +| `sources_failed[]` | **Genuine** failures only | +| `sources_benign[]` | `no-advisories-published`, `skipped: template URL` | + +--- + +## How consumers must read it + +`/recipe-report` renders the `cve` column from the **union** of this scan and the agent's own reading: + +- a clean scan → its number, **including `0`**; +- **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly + how two CVSS-9.8 gitea RCEs were published as `none`; +- **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed; +- benign notes → never `?`. + +`?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is +a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect +in this tool or stale registry data. + +## Known limits + +1. **One window per scan.** Sidecar bumps (redis, postgres) are not counted; they appear as + unclassified. Per-image windows would be the natural extension. +2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time. +3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was + never pointed at. +4. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed + sources (visibly, but degraded).