autonomic-bot
44cb9b6704
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives every count with a SEPARATE semver implementation and its own release fetch and diffs against the scanner. Both found real defects: 1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also keeps the upper bound conservative (18.5 stays out of a window ending at 18). 2. Advisories with no knowable fix version were silently counted as 'not fixed'. Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an open-ended range — six of them high severity. They are now INDETERMINATE: not counted, not dismissed, and surfaced in the output. All twelve turned out to be genuinely fixed: redis names each in the release notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from release notes, with the naming tags recorded as the citation. discourse's redis contribution goes 5 -> 17, and its total 128 -> 140. Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot settle: it hands over each open case's full evidence, plus every verdict pass 1 reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both gitea CVSS-9.8 RCEs — now reach it instead of being dropped. Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is tested there is deterministic: which cases it selects, and that truncation is announced rather than silent. SPEC.md rewritten around the two passes.
123 lines
9.9 KiB
Markdown
123 lines
9.9 KiB
Markdown
---
|
|
name: recipe-report
|
|
description: Generate the weekly public "Recipe Report" after the cc-ci /upgrade-all run. Reviews how the upgrade went and the state of every recipe + open PR, and publishes a self-contained HTML page to report.ci.commoninternet.net (one page per week + a home index). Page order: short lead → the full wire table (priority-sorted, CVE-count column, CVE recipes first) → Addendum of special issues → Security Bulletin → per-recipe "what changed". Read-only — reports, never merges or edits PRs. Runs as its own agent (model separately overridable via REPORT_BACKEND/REPORT_MODEL; defaults to the upgrader's backend+model, opencode-go/glm-5.2). Invoke as /recipe-report [YYYY-MM-DD].
|
|
---
|
|
|
|
# recipe-report
|
|
|
|
Produce the weekly **"The Recipe Report"** — a public HTML page reviewing the latest `/upgrade-all` run
|
|
and the live state of every recipe + open PR. Served at **https://report.ci.commoninternet.net** (one
|
|
`week-<date>.html` per run + a home-page index). You are **read-only**: you report, you never merge,
|
|
edit, or comment on PRs. The page is **public + unauthenticated** — include only public-safe data
|
|
(recipe names, version bumps, PR titles/links, CI verdicts, concise error summaries); **never** secrets,
|
|
tokens, internal IPs, or raw logs.
|
|
|
|
Helper: `python3 /srv/cc-ci/cc-ci-plan/recipe-report.py {survey|render|publish}` (see its header).
|
|
|
|
**Deterministic formatting — spec in, page out (NON-NEGOTIABLE).** ALL layout, CSS, and HTML is owned by
|
|
`recipe-report.py`'s `render()`. Your ONLY output artifact is the **spec JSON** (data + editorial text).
|
|
You **never** write, edit, post-process, or templatize HTML yourself; you never put markup/inline styles
|
|
in spec text fields (they are HTML-escaped on purpose); the page is produced **solely** by
|
|
`render <spec> <out>`. Same spec shape every week → a structurally identical page. If the layout itself
|
|
must change, edit `recipe-report.py` (and `example-spec.json`) — never the generated page. This is what
|
|
keeps every weekly edition looking the same regardless of which model writes the spec.
|
|
|
|
## Procedure
|
|
|
|
1. **Pick the date.** `DATE` = the argument, else today (UTC). The `/upgrade-all` summary for that run is
|
|
`/srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-<DATE>.md`.
|
|
|
|
2. **Survey.** `python3 /srv/cc-ci/cc-ci-plan/recipe-report.py survey <DATE> > /tmp/survey.json`. This
|
|
gives you: the raw `/upgrade-all` summary; for every recipe mirror its open PRs + the `cc-ci/testme`
|
|
verdict; and each recipe's **per-recipe upgrade notes** (`upgrade_notes_md` — the breaking-change /
|
|
**CVE / security** analysis the upgrade did). Read it all.
|
|
|
|
3. **Review & classify.** The page order is: **short lead → the full wire table (priority-sorted) →
|
|
Addendum → Security Bulletin → per-recipe "What changed".** Same newspaper style; this order.
|
|
- **Security analysis.** Scan the per-recipe `upgrade_notes_md` + the summary (and use your own
|
|
knowledge of the version bumps) for upgrades that fix **CVEs / security issues**. For each recipe,
|
|
**count the CVEs** the PR fixes — this drives both the table's `cve` column and the priority sort.
|
|
- **ADDITIONALLY, and never instead:** each per-recipe log carries an `### Advisory scan
|
|
(deterministic pre-step)` block (from `cc-ci-plan/advisory-scan.py` — GitHub Security
|
|
Advisories + vendor security pages + OSV, with severities and fixed-in versions). Treat its
|
|
CVE list as a **further source** and report the **UNION** of it and what you found by reading.
|
|
Its entries are machine-derived with advisory IDs, so prefer them for CVE ids / severities /
|
|
fixed-in versions, and cite the GHSA where present in the Security Bulletin. If the block
|
|
lists **failed sources**, the count is **not** authoritative: render the cve cell as `?`
|
|
(unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea
|
|
RCEs were reported as "none" on 2026-08-07.
|
|
- **`?` must stay RARE — it means "we tried and could not tell", not "we didn't look".** Use it
|
|
ONLY when a scan ran and reported genuinely failed sources, **or when the scan block says
|
|
COUNT UNKNOWN**. In that case the scan's `0` means *not determined*: publish `?` and say so in
|
|
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
|
|
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
|
|
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
|
|
- **A count with undetermined advisories is a FLOOR.** If the scan block says N advisories
|
|
"could NOT be judged", report the number but say in the notes that it is a floor — those
|
|
advisories are neither fixed nor safe, they are unmeasured. Never round them away.
|
|
- **Counts span every image, each judged by its own window.** A scan block lists one line per
|
|
image with its version range and classification method; the headline is their union. So a
|
|
recipe's count legitimately includes **sidecar** CVEs (discourse's 128 = 123 app + 5 redis).
|
|
When a sidecar contributes a critical/high, name the image in the bulletin — CVE-2025-49844 is
|
|
a redis flaw, not a discourse one, and an operator reading "discourse" needs to know that.
|
|
(The scan headline itself now says `UNKNOWN` rather than a number in that case.)
|
|
In particular: a recipe with **no upgrade this run** (up-to-date/skipped) has nothing an
|
|
upgrade could have fixed — report `0`, not `?`. A recipe with a clean scan reports its number (including `0`). Benign notes in a scan
|
|
block (`no-advisories-published`, `skipped: template URL`) are NOT failures and must not
|
|
trigger `?`. If you find yourself rendering `?` for many recipes, that is a bug to report in
|
|
the Addendum, not a normal outcome.
|
|
Anything **critical/high** also gets a `security` bulletin entry (recipe · CVE id(s) + severity ·
|
|
what it fixes · PR link); be specific about severity and what's exposed if not merged.
|
|
- **Lead — ONE short paragraph.** A tight, concrete opener in opus's voice: fleet state in a sentence
|
|
(how healthy, what moved) and what to address first. Lead with the single most important thing; the
|
|
table and sections carry all the detail. Do NOT write multiple paragraphs.
|
|
- **Priority sort for the table.** Order rows by **recommended priority to address**:
|
|
1. recipes **with CVEs** first (more/higher-severity CVEs higher),
|
|
2. then **FAILED** (RED) recipes,
|
|
3. then **STALE**-test recipes,
|
|
4. then routine **GREEN** bumps,
|
|
5. then **UPTODATE / SKIPPED** last.
|
|
- **Addendum — special issues to look into.** A bullet list of real anomalies worth an operator's
|
|
eye: a recipe carrying **two open PRs** to reconcile, a **CI-run oddity** (flaky tier, needed
|
|
re-runs), drift between the summary and reality, a **misleading branch name**, a tooling snag that
|
|
silently drops a recipe, leftover artifact PRs, or **something in the test run that could be
|
|
improved**. **Only real issues — if everything looks clean, leave it empty; do NOT invent items.**
|
|
- **What changed — per recipe with a PR.** One short section per recipe that has a PR this week
|
|
describing what the upgrade actually changes (versions, notable deps, migrations/gotchas, reconcile
|
|
notes). Skip recipes with no PR.
|
|
|
|
4. **Write the spec** `/tmp/report-spec.json` (shape in the helper header):
|
|
- `date`, `subtitle` ("Week of <human date>"),
|
|
- `lead` — **one short paragraph**,
|
|
- `table[]` — EVERY recipe, **sorted by the priority above**. Fields: `recipe`, `change` "a → b",
|
|
`status` GREEN/FAILED/STALE/SKIPPED/UPTODATE, **`cve`** (integer count of CVEs this PR fixes; `0`
|
|
or omit for none), `ci` as a level/result number+info string e.g. `build 154 ✓` or `RED 200 ·
|
|
install` with `ci_url`, `pr`/`pr_url`, `notes`. **CI = link + number/info only; no images.**
|
|
- The table renders **two** status columns: **TESTS** (the `status` field — the CI/test verdict)
|
|
and **STATUS** (the PR's LIVE state, fetched client-side). You only supply `status`; the live
|
|
STATUS column is derived automatically from `recipe` + `pr` and needs no spec field — it shows
|
|
`open` vs a ✓ (any not-open state), refreshing ~30s via the same-origin `/pr/<recipe>/<n>` proxy
|
|
(cc-ci `nix/modules/reports.nix`); it degrades to a muted "?" if the proxy/repo is unreachable.
|
|
- `addendum[]` — list of bullet strings (may be empty),
|
|
- `security[]` — critical-CVE bulletin entries (may be empty),
|
|
- `changes[]` — `{recipe, body, links[]}`, one per recipe that has a PR.
|
|
|
|
**Template — match this.** `.claude/skills/recipe-report/example-spec.json` is the gold-standard
|
|
example (operator-approved style/format/voice). **Match its editorial voice, field shapes, and level
|
|
of specificity**, following the section order above. Read it before writing your spec.
|
|
|
|
5. **Render & publish — via the script only; never hand-edit the HTML.**
|
|
`python3 .../recipe-report.py render /tmp/report-spec.json /tmp/week-<DATE>.html`
|
|
`python3 .../recipe-report.py publish /tmp/week-<DATE>.html <DATE>`
|
|
(publish copies the page to `cc-ci:/var/lib/cc-ci-reports/` and regenerates the index.)
|
|
If `render` errors, **fix the spec JSON** (a malformed field) and re-run render — do NOT work around
|
|
it by writing HTML by hand. The rendered file is an artifact of the spec, never edited directly.
|
|
|
|
6. **Verify & stop.** `curl -fsS https://report.ci.commoninternet.net/week-<DATE>.html` renders and the
|
|
index lists it. Print the report URL and `RECIPE REPORT COMPLETE`, then go idle (one-shot — do not loop).
|
|
|
|
## Guardrails
|
|
- **Read-only.** Never merge/edit/comment on PRs or touch recipes/tests. You only describe + link.
|
|
- **Public-safe.** No secrets/tokens/internal hostnames/raw logs on the page. Summarize errors.
|
|
- One page per run; always regenerate the index so all weeks are listed (newest first).
|