diff --git a/.claude/skills/recipe-report/SKILL.md b/.claude/skills/recipe-report/SKILL.md index 73ba575..da02aa7 100644 --- a/.claude/skills/recipe-report/SKILL.md +++ b/.claude/skills/recipe-report/SKILL.md @@ -37,6 +37,15 @@ keeps every weekly edition looking the same regardless of which model writes the - **Security analysis.** Scan the per-recipe `upgrade_notes_md` + the summary (and use your own knowledge of the version bumps) for upgrades that fix **CVEs / security issues**. For each recipe, **count the CVEs** the PR fixes — this drives both the table's `cve` column and the priority sort. + - **ADDITIONALLY, and never instead:** each per-recipe log carries an `### Advisory scan + (deterministic pre-step)` block (from `cc-ci-plan/advisory-scan.py` — GitHub Security + Advisories + vendor security pages + OSV, with severities and fixed-in versions). Treat its + CVE list as a **further source** and report the **UNION** of it and what you found by reading. + Its entries are machine-derived with advisory IDs, so prefer them for CVE ids / severities / + fixed-in versions, and cite the GHSA where present in the Security Bulletin. If a recipe has + no scan block, or the block lists **failed sources**, the count is **not** authoritative: + render the cve cell as `?` (unknown), never `none` — a blank that reads as "clean" is exactly + how two CVSS-9.8 gitea RCEs were reported as "none" on 2026-08-07. Anything **critical/high** also gets a `security` bulletin entry (recipe · CVE id(s) + severity · what it fixes · PR link); be specific about severity and what's exposed if not merged. - **Lead — ONE short paragraph.** A tight, concrete opener in opus's voice: fleet state in a sentence diff --git a/.claude/skills/recipe-upgrade/SKILL.md b/.claude/skills/recipe-upgrade/SKILL.md index ce6fa55..4cd658b 100644 --- a/.claude/skills/recipe-upgrade/SKILL.md +++ b/.claude/skills/recipe-upgrade/SKILL.md @@ -157,6 +157,35 @@ On cc-ci's `~/.abra/recipes/` (wrap every abra call per the pseudo-TTY b `open-recipe-pr.sh`). Do **not** push to upstream; the version bump + tag + publish are the operator's final `abra recipe release` step. +### 2a. Advisory scan (deterministic; ADDITIVE — run it, never skip it) + +Run the deterministic scanner for the exact upgrade window and **paste its markdown block verbatim +into the per-recipe log**: + +``` +python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py --from --to +``` + +It queries, per recipe: the **GitHub Security Advisories API** for every source repo in +`cc-ci-plan/upstream/.md` (CVE + GHSA + severity + vulnerable/patched ranges, so +"fixed by THIS upgrade" is computed, not guessed), every **vendor release/security URL** in that +registry (fetched + regex-scanned for CVE ids), and **OSV** where a package mapping exists. + +**This does NOT replace your own release-note reading — it is an ADDITIONAL evidence source.** Do +exactly what you did before, then union the two: the CVE count you report is the union of the CVEs +you found in the notes and the CVEs the scan found. Never let the scan lower a count you established +by reading. + +Why it exists: gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). Both are named +only in the vendor's blog security section — the GitHub *release notes* mention neither — so the +release-note read found one unrelated minor item and the weekly report printed a CVE count of "1", +then "none". Advisory databases lagged too (OSV 404'd on both; NVD's API had neither by CPE, id, or +keyword), which is why the GitHub advisory API and the vendor pages lead. + +If the scanner reports **failed sources**, say so in the log — an incomplete scan must not read as +a clean one. If a vendor publishes security notes at a URL the registry lacks (gitea's +`blog.gitea.com`), **add it to `cc-ci-plan/upstream/.md`** so the next scan sees it. + ### 2b. Direct deploy + inspect on cc-ci — live feedback BEFORE CI (recipe-maintainer style) Before opening the PR / running `!testme`, deploy the WIP recipe **directly** on the cc-ci server and watch it converge — the way recipe-maintainer tests on `cctest`. This gives you **live logs + diff --git a/cc-ci-plan/advisory-scan.py b/cc-ci-plan/advisory-scan.py new file mode 100755 index 0000000..485ae16 --- /dev/null +++ b/cc-ci-plan/advisory-scan.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 +"""Deterministic per-recipe CVE/advisory scan — an ADDITIVE pre-step for /recipe-upgrade. + +WHY THIS EXISTS (2026-08-10): gitea 1.27.1 fixed two CVSS-9.8 RCEs (CVE-2026-60004, +CVE-2026-59774). Our weekly report showed gitea's CVE count as "1", then "none". The upgrade +subagent had scanned the GitHub *release notes*, which mention neither; the two CVEs were announced +only in the vendor's blog security section. The report generator then derived security content from +those notes plus model knowledge — and the model's training predates the CVEs. Nothing in the +pipeline ever queried an advisory source, so a critical CVE that is newer than the model and absent +from the changelog was invisible by construction. + +WHAT IT DOES NOT DO: it does not replace or alter any existing security analysis. It is a strictly +ADDITIONAL evidence source whose findings are unioned into the CVE count. + +SOURCES (measured against the gitea case before being chosen): + 1. GitHub Security Advisories API — repos///security-advisories. PRIMARY: carries + CVE id, GHSA id, severity AND vulnerable/patched version ranges, so "fixed by THIS upgrade" is + computable rather than guessed. Found both gitea CVEs. Derived from the source-repo URLs the + per-recipe registry already records — no new per-recipe config needed. + 2. Vendor release/security pages — every URL in cc-ci-plan/upstream/.md, fetched and + regex-scanned for CVE ids. This is what would have caught gitea: the vendor blog names both, + while the GitHub releases page names neither. Add vendor security/announcement URLs to the + registry to widen this. + 3. OSV.dev — supplementary, best-effort, only when the recipe declares an ecosystem/package + mapping below. NOTE: for gitea, OSV returned only Go *dependency* advisories and 404'd on both + application CVEs; NVD's API had them neither by CPE, CVE id, nor keyword. Advisory databases + lag the vendor — which is exactly why (1) and (2) lead. + +Every source reports its own status, so "checked, none found" is never confused with "not checked". + +Usage: + advisory-scan.py [--from ] [--to ] [--json] [--registry DIR] + + --from/--to are the app versions being upgraded between (e.g. 1.26.2 -> 1.27.1). When given, each + advisory is classified fixed-by-this-upgrade / still-open / older. Without them everything known + is listed unclassified. Exits 0 even when sources fail (informational; failures are reported). +""" + +from __future__ import annotations + +import argparse +import gzip +import json +import os +import re +import sys +import urllib.error +import urllib.request + +REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", "/srv/cc-ci/cc-ci-plan/upstream") +UA = "cc-ci-advisory-scan (+https://git.autonomic.zone/recipe-maintainers/cc-ci)" +TIMEOUT = int(os.environ.get("ADVISORY_SCAN_TIMEOUT", "45")) +CVE_RE = re.compile(r"CVE-\d{4}-\d{4,7}") + +# Optional OSV mappings: recipe -> (ecosystem, package). Supplementary only (see module docstring). +OSV_PACKAGES: dict[str, tuple[str, str]] = { + "gitea": ("Go", "code.gitea.io/gitea"), + "n8n": ("npm", "n8n"), +} + + +def _fetch(url: str, headers: dict | None = None) -> str: + h = {"User-Agent": UA, "Accept-Encoding": "gzip"} + h.update(headers or {}) + with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r: + raw = r.read() + if r.headers.get("Content-Encoding") == "gzip": + raw = gzip.decompress(raw) + return raw.decode(errors="replace") + + +def _vkey(v: str | None) -> tuple: + """Loose version ordering key: leading integers of each dot-part ('1.27.1-rootless' -> (1,27,1)).""" + if not v: + return () + v = v.strip().lstrip("vV").split("+")[0] + out = [] + for part in re.split(r"[.\-_]", v): + m = re.match(r"^\d+", part) + if m: + out.append(int(m.group())) + elif out: + break + return tuple(out) + + +def registry_urls(recipe: str, registry_dir: str) -> tuple[list[str], str | None]: + path = os.path.join(registry_dir, f"{recipe}.md") + try: + with open(path) as f: + text = f.read() + except OSError: + return [], None + urls = [] + for u in re.findall(r"https?://[^\s)|\]]+", text): + u = u.rstrip(".,;") + if u not in urls: + urls.append(u) + return urls, path + + +def github_advisories(urls: list[str]) -> list[dict]: + """Query GitHub Security Advisories for every github.com// in the registry.""" + seen, results = set(), [] + for u in urls: + m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u) + if not m: + continue + owner, repo = m.group(1), m.group(2).removesuffix(".git") + if (owner, repo) in seen: + continue + seen.add((owner, repo)) + api = f"https://api.github.com/repos/{owner}/{repo}/security-advisories?per_page=100" + hdrs = {"Accept": "application/vnd.github+json"} + tok = os.environ.get("GITHUB_TOKEN") + if tok: + hdrs["Authorization"] = f"Bearer {tok}" + entry = {"source": f"github-advisories:{owner}/{repo}", "status": "ok", "advisories": []} + try: + for a in json.loads(_fetch(api, hdrs)): + vulns = a.get("vulnerabilities") or [] + entry["advisories"].append( + { + "cve": a.get("cve_id"), + "ghsa": a.get("ghsa_id"), + "severity": a.get("severity"), + "summary": (a.get("summary") or "")[:200], + "vulnerable_range": vulns[0].get("vulnerable_version_range") if vulns else None, + "patched": vulns[0].get("patched_versions") if vulns else None, + "url": a.get("html_url"), + } + ) + except Exception as e: # noqa: BLE001 — a dead source must be REPORTED, never silent + entry["status"] = f"error: {type(e).__name__}: {e}" + results.append(entry) + return results + + +def vendor_pages(urls: list[str]) -> list[dict]: + """Fetch each registry URL and regex out CVE ids, with a little surrounding context.""" + out = [] + for u in urls: + if u.startswith("https://api.github.com"): + continue + entry = {"source": u, "status": "ok", "cves": [], "context": {}} + try: + text = _fetch(u) + plain = re.sub(r"<[^>]+>", " ", text) + for cve in sorted(set(CVE_RE.findall(plain))): + entry["cves"].append(cve) + i = plain.find(cve) + entry["context"][cve] = re.sub(r"\s+", " ", plain[max(0, i - 160) : i + 200]).strip() + except Exception as e: # noqa: BLE001 + entry["status"] = f"error: {type(e).__name__}: {e}" + out.append(entry) + return out + + +def osv(recipe: str, version: str | None) -> dict | None: + pkg = OSV_PACKAGES.get(recipe) + if not pkg or not version: + return None + eco, name = pkg + entry = {"source": f"osv:{eco}/{name}@{version}", "status": "ok", "cves": []} + try: + body = json.dumps({"package": {"name": name, "ecosystem": eco}, "version": version}).encode() + req = urllib.request.Request( + "https://api.osv.dev/v1/query", data=body, + headers={"Content-Type": "application/json", "User-Agent": UA}, method="POST", + ) + with urllib.request.urlopen(req, timeout=TIMEOUT) as r: + data = json.load(r) + ids = set() + for v in data.get("vulns", []): + for a in [v.get("id")] + (v.get("aliases") or []): + if a and a.startswith("CVE"): + ids.add(a) + entry["cves"] = sorted(ids) + except Exception as e: # noqa: BLE001 + entry["status"] = f"error: {type(e).__name__}: {e}" + return entry + + +def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str) -> dict: + urls, reg_path = registry_urls(recipe, registry_dir) + report: dict = { + "recipe": recipe, + "from": v_from, + "to": v_to, + "registry": reg_path, + "registry_urls": len(urls), + "sources": [], + "cves": {}, + } + if reg_path is None: + report["sources"].append( + {"source": f"registry:{recipe}.md", "status": "error: registry file not found"} + ) + + def record(cve: str, src: str, **extra): + e = report["cves"].setdefault(cve, {"sources": [], "severity": None, "ghsa": None, + "vulnerable_range": None, "patched": None, "context": None}) + if src not in e["sources"]: + e["sources"].append(src) + for k, v in extra.items(): + if v and not e.get(k): + e[k] = v + + for entry in github_advisories(urls): + report["sources"].append({"source": entry["source"], "status": entry["status"], + "found": len(entry.get("advisories", []))}) + for a in entry.get("advisories", []): + if a.get("cve"): + record(a["cve"], entry["source"], severity=a.get("severity"), ghsa=a.get("ghsa"), + vulnerable_range=a.get("vulnerable_range"), patched=a.get("patched"), + context=a.get("summary")) + + for entry in vendor_pages(urls): + report["sources"].append({"source": entry["source"], "status": entry["status"], + "found": len(entry.get("cves", []))}) + for cve in entry.get("cves", []): + record(cve, entry["source"], context=entry["context"].get(cve)) + + for version in filter(None, (v_from, v_to)): + o = osv(recipe, version) + if o: + report["sources"].append({"source": o["source"], "status": o["status"], + "found": len(o.get("cves", []))}) + for cve in o.get("cves", []): + record(cve, o["source"]) + + # Classify against the upgrade window when we know it: an advisory is "fixed by this upgrade" + # when its patched version is newer than `from` and no newer than `to`. + kf, kt = _vkey(v_from), _vkey(v_to) + fixed, unknown = [], [] + for cve, e in report["cves"].items(): + kp = _vkey((e.get("patched") or "").split(",")[0].strip() or None) + if kf and kt and kp and kf < kp <= kt: + e["classification"] = "fixed-by-this-upgrade" + fixed.append(cve) + else: + e["classification"] = "unclassified" if not (kf and kt and kp) else "outside-window" + if e["classification"] == "unclassified": + unknown.append(cve) + report["fixed_by_this_upgrade"] = sorted(fixed) + report["unclassified"] = sorted(unknown) + report["cve_count_fixed"] = len(fixed) + report["cve_count_total_seen"] = len(report["cves"]) + report["sources_failed"] = [s["source"] for s in report["sources"] if s["status"] != "ok"] + return report + + +def markdown(rep: dict) -> str: + """Human/agent-readable block for pasting into the per-recipe upgrade log.""" + L = [f"### Advisory scan (deterministic pre-step) — {rep['recipe']} " + f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"] + if rep["fixed_by_this_upgrade"]: + L.append(f"\n**CVEs fixed by this upgrade: {rep['cve_count_fixed']}**\n") + L.append("| CVE | severity | fixed in | advisory | source |") + L.append("|---|---|---|---|---|") + for cve in rep["fixed_by_this_upgrade"]: + e = rep["cves"][cve] + L.append(f"| {cve} | {e.get('severity') or '?'} | {e.get('patched') or '?'} | " + f"{e.get('ghsa') or '-'} | {e['sources'][0]} |") + else: + L.append("\n**CVEs fixed by this upgrade: 0 identified by the deterministic scan.**") + if rep["unclassified"]: + L.append(f"\nSeen but not version-classified ({len(rep['unclassified'])}): " + + ", ".join(rep["unclassified"][:12])) + if rep["sources_failed"]: + L.append(f"\n⚠ sources that FAILED (treat counts as incomplete): {', '.join(rep['sources_failed'])}") + L.append(f"\n_Sources checked: {len(rep['sources'])} " + f"({rep['registry_urls']} registry URLs + advisory APIs). This scan is ADDITIVE — it does " + f"not replace the release-note reading in the upgrade step._") + return "\n".join(L) + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("recipe") + ap.add_argument("--from", dest="v_from", default=None) + ap.add_argument("--to", dest="v_to", default=None) + ap.add_argument("--json", action="store_true", help="emit raw JSON instead of markdown") + ap.add_argument("--registry", default=REGISTRY_DIR) + a = ap.parse_args() + rep = scan(a.recipe, a.v_from, a.v_to, a.registry) + print(json.dumps(rep, indent=2) if a.json else markdown(rep)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/cc-ci-plan/upstream/gitea.md b/cc-ci-plan/upstream/gitea.md index 6103ef4..a522edc 100644 --- a/cc-ci-plan/upstream/gitea.md +++ b/cc-ci-plan/upstream/gitea.md @@ -3,6 +3,12 @@ | service | image | source repo | releases / changelog | |---------|-------|-------------|----------------------| | app | gitea/gitea | https://github.com/go-gitea/gitea | https://github.com/go-gitea/gitea/releases | + +**Security announcements: https://blog.gitea.com/ — per-release posts (e.g. +https://blog.gitea.com/release-of-1.27.1/) carry the CVE list; the GitHub release notes do NOT.** +This is where CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8, fixed in 1.27.1) were announced, +and why the 2026-08-03/07 reports under-counted gitea's CVEs. advisory-scan.py fetches every URL +in this file, so keep vendor security pages listed here. | db | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ | ## Standing notes