recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
55ce8e3830dd81300a8c3103da8a40a4e5d56c5f
cc-ci-orchestrator/.claude/skills/recipe-report/SKILL.md
T
autonomic-bot 44cb9b6704 advisory-scan: tests, audit, and two real undercounts they found 
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions
against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives
every count with a SEPARATE semver implementation and its own release fetch and
diffs against the scanner. Both found real defects:

1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE
   patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the
   norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also
   keeps the upper bound conservative (18.5 stays out of a window ending at 18).

2. Advisories with no knowable fix version were silently counted as 'not fixed'.
   Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an
   open-ended range — six of them high severity. They are now INDETERMINATE:
   not counted, not dismissed, and surfaced in the output.

   All twelve turned out to be genuinely fixed: redis names each in the release
   notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10,
   7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from
   release notes, with the naming tags recorded as the citation. discourse's
   redis contribution goes 5 -> 17, and its total 128 -> 140.

Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot
settle: it hands over each open case's full evidence, plus every verdict pass 1
reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that
evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both
gitea CVSS-9.8 RCEs — now reach it instead of being dropped.

Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is
tested there is deterministic: which cases it selects, and that truncation is
announced rather than silent.

SPEC.md rewritten around the two passes.
2026-08-11 01:29:11 +00:00

9.9 KiB
Raw Blame History

name: recipe-report description: Generate the weekly public "Recipe Report" after the cc-ci /upgrade-all run. Reviews how the upgrade went and the state of every recipe + open PR, and publishes a self-contained HTML page to report.ci.commoninternet.net (one page per week + a home index). Page order: short lead → the full wire table (priority-sorted, CVE-count column, CVE recipes first) → Addendum of special issues → Security Bulletin → per-recipe "what changed". Read-only — reports, never merges or edits PRs. Runs as its own agent (model separately overridable via REPORT_BACKEND/REPORT_MODEL; defaults to the upgrader's backend+model, opencode-go/glm-5.2). Invoke as /recipe-report [YYYY-MM-DD].

recipe-report

Produce the weekly "The Recipe Report" — a public HTML page reviewing the latest /upgrade-all run and the live state of every recipe + open PR. Served at https://report.ci.commoninternet.net (one week-<date>.html per run + a home-page index). You are read-only: you report, you never merge, edit, or comment on PRs. The page is public + unauthenticated — include only public-safe data (recipe names, version bumps, PR titles/links, CI verdicts, concise error summaries); never secrets, tokens, internal IPs, or raw logs.

Helper: python3 /srv/cc-ci/cc-ci-plan/recipe-report.py {survey|render|publish} (see its header).

Deterministic formatting — spec in, page out (NON-NEGOTIABLE). ALL layout, CSS, and HTML is owned by recipe-report.py's render(). Your ONLY output artifact is the spec JSON (data + editorial text). You never write, edit, post-process, or templatize HTML yourself; you never put markup/inline styles in spec text fields (they are HTML-escaped on purpose); the page is produced solely by render <spec> <out>. Same spec shape every week → a structurally identical page. If the layout itself must change, edit recipe-report.py (and example-spec.json) — never the generated page. This is what keeps every weekly edition looking the same regardless of which model writes the spec.

Procedure

  1. Pick the date. DATE = the argument, else today (UTC). The /upgrade-all summary for that run is /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-<DATE>.md.

  2. Survey. python3 /srv/cc-ci/cc-ci-plan/recipe-report.py survey <DATE> > /tmp/survey.json. This gives you: the raw /upgrade-all summary; for every recipe mirror its open PRs + the cc-ci/testme verdict; and each recipe's per-recipe upgrade notes (upgrade_notes_md — the breaking-change / CVE / security analysis the upgrade did). Read it all.

  3. Review & classify. The page order is: short lead → the full wire table (priority-sorted) → Addendum → Security Bulletin → per-recipe "What changed". Same newspaper style; this order.

    • Security analysis. Scan the per-recipe upgrade_notes_md + the summary (and use your own knowledge of the version bumps) for upgrades that fix CVEs / security issues. For each recipe, count the CVEs the PR fixes — this drives both the table's cve column and the priority sort.
      • ADDITIONALLY, and never instead: each per-recipe log carries an ### Advisory scan (deterministic pre-step) block (from cc-ci-plan/advisory-scan.py — GitHub Security Advisories + vendor security pages + OSV, with severities and fixed-in versions). Treat its CVE list as a further source and report the UNION of it and what you found by reading. Its entries are machine-derived with advisory IDs, so prefer them for CVE ids / severities / fixed-in versions, and cite the GHSA where present in the Security Bulletin. If the block lists failed sources, the count is not authoritative: render the cve cell as ? (unknown), never none — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea RCEs were reported as "none" on 2026-08-07.
      • ? must stay RARE — it means "we tried and could not tell", not "we didn't look". Use it ONLY when a scan ran and reported genuinely failed sources, or when the scan block says COUNT UNKNOWN. In that case the scan's 0 means not determined: publish ? and say so in the notes; publishing 0 would assert a clean bill of health nothing supports. Note a version-scheme change is no longer a reason for ? — the scan resolves semver→calver jumps (discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
      • A count with undetermined advisories is a FLOOR. If the scan block says N advisories "could NOT be judged", report the number but say in the notes that it is a floor — those advisories are neither fixed nor safe, they are unmeasured. Never round them away.
      • Counts span every image, each judged by its own window. A scan block lists one line per image with its version range and classification method; the headline is their union. So a recipe's count legitimately includes sidecar CVEs (discourse's 128 = 123 app + 5 redis). When a sidecar contributes a critical/high, name the image in the bulletin — CVE-2025-49844 is a redis flaw, not a discourse one, and an operator reading "discourse" needs to know that. (The scan headline itself now says UNKNOWN rather than a number in that case.) In particular: a recipe with no upgrade this run (up-to-date/skipped) has nothing an upgrade could have fixed — report 0, not ?. A recipe with a clean scan reports its number (including 0). Benign notes in a scan block (no-advisories-published, skipped: template URL) are NOT failures and must not trigger ?. If you find yourself rendering ? for many recipes, that is a bug to report in the Addendum, not a normal outcome. Anything critical/high also gets a security bulletin entry (recipe · CVE id(s) + severity · what it fixes · PR link); be specific about severity and what's exposed if not merged.
    • Lead — ONE short paragraph. A tight, concrete opener in opus's voice: fleet state in a sentence (how healthy, what moved) and what to address first. Lead with the single most important thing; the table and sections carry all the detail. Do NOT write multiple paragraphs.
    • Priority sort for the table. Order rows by recommended priority to address:
      1. recipes with CVEs first (more/higher-severity CVEs higher),
      2. then FAILED (RED) recipes,
      3. then STALE-test recipes,
      4. then routine GREEN bumps,
      5. then UPTODATE / SKIPPED last.
    • Addendum — special issues to look into. A bullet list of real anomalies worth an operator's eye: a recipe carrying two open PRs to reconcile, a CI-run oddity (flaky tier, needed re-runs), drift between the summary and reality, a misleading branch name, a tooling snag that silently drops a recipe, leftover artifact PRs, or something in the test run that could be improved. Only real issues — if everything looks clean, leave it empty; do NOT invent items.
    • What changed — per recipe with a PR. One short section per recipe that has a PR this week describing what the upgrade actually changes (versions, notable deps, migrations/gotchas, reconcile notes). Skip recipes with no PR.

  4. Write the spec /tmp/report-spec.json (shape in the helper header):

    • date, subtitle ("Week of "),
    • lead — one short paragraph,
    • table[] — EVERY recipe, sorted by the priority above. Fields: recipe, change "a → b", status GREEN/FAILED/STALE/SKIPPED/UPTODATE, cve (integer count of CVEs this PR fixes; 0 or omit for none), ci as a level/result number+info string e.g. build 154 ✓ or RED 200 · install with ci_url, pr/pr_url, notes. CI = link + number/info only; no images.
      • The table renders two status columns: TESTS (the status field — the CI/test verdict) and STATUS (the PR's LIVE state, fetched client-side). You only supply status; the live STATUS column is derived automatically from recipe + pr and needs no spec field — it shows open vs a ✓ (any not-open state), refreshing ~30s via the same-origin /pr/<recipe>/<n> proxy (cc-ci nix/modules/reports.nix); it degrades to a muted "?" if the proxy/repo is unreachable.
    • addendum[] — list of bullet strings (may be empty),
    • security[] — critical-CVE bulletin entries (may be empty),
    • changes[] — {recipe, body, links[]}, one per recipe that has a PR.

    Template — match this. .claude/skills/recipe-report/example-spec.json is the gold-standard example (operator-approved style/format/voice). Match its editorial voice, field shapes, and level of specificity, following the section order above. Read it before writing your spec.

  5. Render & publish — via the script only; never hand-edit the HTML. python3 .../recipe-report.py render /tmp/report-spec.json /tmp/week-<DATE>.html python3 .../recipe-report.py publish /tmp/week-<DATE>.html <DATE> (publish copies the page to cc-ci:/var/lib/cc-ci-reports/ and regenerates the index.) If render errors, fix the spec JSON (a malformed field) and re-run render — do NOT work around it by writing HTML by hand. The rendered file is an artifact of the spec, never edited directly.

  6. Verify & stop. curl -fsS https://report.ci.commoninternet.net/week-<DATE>.html renders and the index lists it. Print the report URL and RECIPE REPORT COMPLETE, then go idle (one-shot — do not loop).

Guardrails

  • Read-only. Never merge/edit/comment on PRs or touch recipes/tests. You only describe + link.
  • Public-safe. No secrets/tokens/internal hostnames/raw logs on the page. Summarize errors.
  • One page per run; always regenerate the index so all weeks are listed (newest first).
Reference in New Issue View Git Blame Copy Permalink