autonomic-bot
78ae2be8ae
Step-by-step specification of cc-ci-plan/advisory-scan.py: inputs, the three source classes and why each is ranked where it is, the union, both classification paths (patched-version ranges, and the advisory-publish-date fallback for version-scheme changes), the output contract, and how /recipe-report must read it. Each rule records the production wrong answer that motivated it — the false 133 from cross-image counting, the n8n misclassification from reading only vulnerabilities[0], the '?' sprawl from url punctuation and benign-404s, and the 'never emit 0 for an undetermined count' rule. Claims cross-checked against the implementation. Keep this file in the same commit as any behaviour change.
185 lines
9.6 KiB
Markdown
185 lines
9.6 KiB
Markdown
# Advisory scan — specification
|
|
|
|
What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the
|
|
implementation as it stands (2026-08-10); if you change the code, change this file in the same commit.
|
|
|
|
**Role.** A deterministic, per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a).
|
|
It is **strictly additive**: it never replaces the release-note reading the upgrade agent already does.
|
|
The CVE count reported for a recipe is the **union** of what the agent read and what this scan found;
|
|
the scan may never *lower* a count established by reading.
|
|
|
|
**Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly
|
|
report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release
|
|
notes*, which name neither — both were announced only in the vendor's blog security section — and the
|
|
report then derived security content from those notes plus model knowledge, which predates the CVEs.
|
|
Nothing in the pipeline queried an advisory source. This scan closes that hole.
|
|
|
|
---
|
|
|
|
## Inputs
|
|
|
|
```
|
|
advisory-scan.py <recipe> [--from <version>] [--to <version>] [--json] [--registry DIR]
|
|
```
|
|
|
|
| Input | Meaning |
|
|
|---|---|
|
|
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
|
|
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
|
|
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
|
|
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
|
|
|
|
Exit code is always 0 — this is informational. Failures are *reported*, never raised.
|
|
|
|
---
|
|
|
|
## Step 1 — Collect source URLs from the registry
|
|
|
|
Read `cc-ci-plan/upstream/<recipe>.md` and extract every `http(s)://…` URL.
|
|
|
|
**Trailing markdown punctuation is stripped** (`` ` `` `'` `"` `*` `.` `,` `;` `:` `>` `)`). The registry
|
|
is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of
|
|
`https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render `?` for no real reason.
|
|
|
|
> **Registry hygiene matters.** The scan can only look where the registry points. Two classes of defect
|
|
> have been found and fixed by running it: a **wrong URL** (`pgautoupgrade/pgautoupgrade`, which 404s —
|
|
> the repo is `pgautoupgrade/docker-pgautoupgrade`) and a **missing** one (gitea's CVEs are announced at
|
|
> `blog.gitea.com`, which the registry didn't list). When a vendor publishes security notes somewhere
|
|
> the registry lacks, add it.
|
|
|
|
## Step 2 — Query the sources
|
|
|
|
Three source classes, each recording **its own status** so *"checked, none found"* is never confused
|
|
with *"not checked"*.
|
|
|
|
### 2a. GitHub Security Advisories — PRIMARY
|
|
|
|
For every `github.com/<owner>/<repo>` URL in the registry:
|
|
`GET /repos/<owner>/<repo>/security-advisories`.
|
|
|
|
Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, `published_at`, and **all**
|
|
`vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined with `;`).
|
|
|
|
- **All entries, not just the first.** An advisory carries one entry **per patched release line** —
|
|
n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the
|
|
line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window.
|
|
- **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns
|
|
at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy
|
|
projects. discourse has 286 advisories; a single page cannot even cover one upgrade window.
|
|
- **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not**
|
|
a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal.
|
|
|
|
This source is primary because it carries **severity and version ranges**, making "fixed by *this*
|
|
upgrade" computable rather than guessed.
|
|
|
|
### 2b. Vendor release / security pages
|
|
|
|
Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160
|
|
characters of context per hit.
|
|
|
|
URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as templates** — they are
|
|
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.
|
|
|
|
This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
|
|
page names neither.
|
|
|
|
### 2c. OSV.dev — supplementary
|
|
|
|
Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.
|
|
|
|
> **Measured, not assumed.** For the two gitea CVEs, OSV **404'd on both** and returned only Go
|
|
> *dependency* advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword.
|
|
> **Advisory databases lag the vendor**, which is why 2a and 2b lead and this is supplementary.
|
|
|
|
## Step 3 — Union
|
|
|
|
All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched,
|
|
published_at, context}`. A CVE seen by several sources keeps them all.
|
|
|
|
## Step 4 — Classify against the upgrade window
|
|
|
|
Two invariants govern this step, both learned from a wrong answer in production.
|
|
|
|
> **A. The window belongs to ONE image.** Only advisories from the **primary** source (the first
|
|
> `github-advisories:` source, i.e. the app repo the registry lists first) are classified. Sidecar
|
|
> advisories are recorded as unclassified so they stay visible without inflating the count.
|
|
> *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including
|
|
> `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically
|
|
> inside discourse's `3.5.3 → 2026.7.1` range.
|
|
>
|
|
> **B. Never emit a number you cannot justify.** If neither method below can order the window, the
|
|
> count is `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety.
|
|
|
|
### 4a. By patched version (preferred — exact)
|
|
|
|
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
|
|
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
|
|
satisfies `from < p <= to`, using a loose numeric key (leading integers per dot-part).
|
|
|
|
### 4b. By advisory publish date (fallback — temporal)
|
|
|
|
Used **only** when 4a cannot be trusted: a **version-scheme change**, detected as the leading version
|
|
component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver `2026.7.1`.
|
|
|
|
Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while
|
|
shipping earlier), but **release dates always order**. So:
|
|
|
|
1. Resolve `--from` and `--to` to **git tag dates** on the primary repo (tries `v<version>` then
|
|
`<version>`; annotated tag → tagger date, else commit date).
|
|
2. An advisory counts as fixed when `date_from < published_at <= date_to`.
|
|
|
|
Both the method and the resolved window appear in the output. This reproduces, automatically, the hand
|
|
count that established discourse `3.5.3` (2025-12-30) → `2026.7.1` (2026-07-31) = **123 CVEs**.
|
|
|
|
*Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes
|
|
**first-party plugin advisories** where the vendor files them on the same repo — which is why a
|
|
plugin-rich project scores far higher than a monolith, not a statement about relative security.
|
|
|
|
### 4c. Otherwise
|
|
|
|
`count_known = false`, `cve_count_fixed = null`, and the markdown headline reads
|
|
**"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an explicit
|
|
*"This is NOT zero"*.
|
|
|
|
## Step 5 — Output
|
|
|
|
Markdown (default) for pasting into the per-recipe upgrade log, or `--json`.
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `cve_count_fixed` | Count, or **`null`** when undeterminable |
|
|
| `count_known` | Distinguishes "counted zero" from "could not count" |
|
|
| `classified_by` | `patched version ranges` or `advisory publish date (version scheme changed)` |
|
|
| `date_window` | The resolved date window, when 4b was used |
|
|
| `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id |
|
|
| `unclassified[]` | Seen but not attributable to this window (incl. other images) |
|
|
| `sources[]` | Every source with its own status |
|
|
| `sources_failed[]` | **Genuine** failures only |
|
|
| `sources_benign[]` | `no-advisories-published`, `skipped: template URL` |
|
|
|
|
---
|
|
|
|
## How consumers must read it
|
|
|
|
`/recipe-report` renders the `cve` column from the **union** of this scan and the agent's own reading:
|
|
|
|
- a clean scan → its number, **including `0`**;
|
|
- **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly
|
|
how two CVSS-9.8 gitea RCEs were published as `none`;
|
|
- **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed;
|
|
- benign notes → never `?`.
|
|
|
|
`?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is
|
|
a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect
|
|
in this tool or stale registry data.
|
|
|
|
## Known limits
|
|
|
|
1. **One window per scan.** Sidecar bumps (redis, postgres) are not counted; they appear as
|
|
unclassified. Per-image windows would be the natural extension.
|
|
2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time.
|
|
3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was
|
|
never pointed at.
|
|
4. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed
|
|
sources (visibly, but degraded).
|