recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?'

Browse Source 
Two recipes could not see CVEs at all. mattermost-lts has an empty GitHub advisory
feed and renders its security bulletins client-side, so a text sweep finds nothing;
mumble publishes nothing anywhere the registry points. Both returned '?' - nothing
measured - which is honest but useless.

NVD is CPE-indexed and carries structured version ranges, so it answers where the
vendor does not. Declared per recipe as 'nvd-cpe: <image> = <cpe:2.3:...>'.

  mattermost-lts 10.5.0  -> 10.12.4   165 CVEs
  mattermost-lts 10.11.22 -> 10.12.4    0 CVEs  (measured, not unknown)
  mumble         1.3.0   -> 1.6.870      2 CVEs

Both NVD range forms are used: versionEndExcluding is a patched version;
versionEndIncluding means the fix version is unpublished but the upgrade delivers
it whenever it crosses X.

That 0 for the actual mattermost upgrade is the interesting one, and it needed a
new rule to be correct: a fix on the line you upgrade FROM was already yours.
mattermost patches every maintained line at once, so 10.11.22 -> 10.12.4 crosses
10.12.1 while 10.11.22 already had the 10.11.4 backport. Without the rule the scan
claimed 12 CVEs the upgrade did not deliver.

The rule is skipped for placeholders: '7.4.X' parses to a bare 7.4 and would read
as 'already fixed at 7.4', which silently dropped redis CVE-2024-46981 and took
discourse 140 -> 139 before I caught it.

79 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 / plausible 6 unchanged.
Fleet sweep: 0 recipes with no usable CVE source, down from 2.
This commit is contained in:
autonomic-bot
2026-08-11 22:16:37 +00:00
parent 4b9978ac02
commit 985dc06e47
6 changed files with 222 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cc-ci-plan/advisory-scan.py
+138 -2
View File
@@ -44,7 +44,9 @@ import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", "/srv/cc-ci/cc-ci-plan/upstream")
@@ -164,6 +166,27 @@ def _vkey(v: str | None) -> tuple:
return tuple(out)
def _already_fixed_on_from_line(kf: tuple, cands: list[tuple]) -> bool:
"""Was it ALREADY fixed on the line we are upgrading FROM?
The mirror image of _superseded_on_target_line, and just as necessary. mattermost fixes each CVE
across several maintained lines at once — CVE-2025-11794 is patched in 10.11.4, 10.12.1 and
10.5.12. Upgrading 10.11.22 -> 10.12.4 crosses 10.12.1, so a naive window test counts it; but
10.11.22 is already past 10.11.4, so the deployment HAD the fix before the upgrade. Counting it
credits the upgrade with work it did not do."""
if len(kf) < 2:
return False
line = kf[:2]
for c in cands:
if len(c) < 2 or c[:2] != line:
continue
n = max(len(kf), len(c))
pad = lambda z: z + (0,) * (n - len(z))
if pad(c) <= pad(kf):
return True
return False
def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool:
"""Does a patched version on the TARGET's own release line sit ABOVE the target?
@@ -344,6 +367,78 @@ def vendor_pages(urls: list[str]) -> list[dict]:
return out
NVD_API = "https://services.nvd.nist.gov/rest/json/cves/2.0"
NVD_CPE_RE = re.compile(r"^\s*[-*]?\s*nvd-cpe:\s*(\S+)\s*=\s*(cpe:2\.3:[^\s`]+)", re.M | re.I)
def registry_cpes(recipe: str, registry_dir: str) -> list[tuple[str, str]]:
"""[(image-key, cpe)] declared in the recipe's registry as `nvd-cpe: <key> = <cpe>`."""
path = os.path.join(registry_dir, f"{recipe}.md")
try:
return [(m.group(1), m.group(2)) for m in NVD_CPE_RE.finditer(open(path).read())]
except OSError:
return []
def nvd_advisories(cpe: str, key: str) -> dict:
"""CVEs for a CPE from NVD, with the version data the classifier needs.
THE FALLBACK FOR PROJECTS THAT PUBLISH NOTHING MACHINE-READABLE. mattermost's GitHub advisory
feed is empty and its security bulletins are client-side rendered; mumble publishes neither. Both
scanned as `?` — nothing measured — until here. NVD is CPE-indexed and carries structured ranges:
versionEndExcluding X -> fixed in X exactly (a patched version)
versionEndIncluding X -> affected up to and INCLUDING X, fixed in some later release. The
exact fix version is unknown, but the upgrade fixes it whenever it
crosses X — recorded as `affected_max` and judged in the classifier.
NVD LAGS the vendor (it had neither gitea CVSS-9.8 RCE at publication), so this is a fallback,
never a replacement for 2a/2b. Unauthenticated calls are rate-limited to ~5/30s, hence the retry.
"""
entry = {"source": f"nvd:{key}", "status": "ok", "advisories": []}
url = f"{NVD_API}?resultsPerPage=2000&virtualMatchString={urllib.parse.quote(cpe)}"
data = None
for attempt in range(3):
try:
data = json.loads(_fetch(url))
break
except Exception as e: # noqa: BLE001
if attempt == 2:
entry["status"] = f"error: {type(e).__name__}"
return entry
time.sleep(8)
for v in (data or {}).get("vulnerabilities", []):
c = v.get("cve") or {}
cid = c.get("id")
if not cid:
continue
fixed, affected_max = set(), set()
for cfg in c.get("configurations", []):
for node in cfg.get("nodes", []):
for m in node.get("cpeMatch", []):
if m.get("versionEndExcluding"):
fixed.add(m["versionEndExcluding"])
elif m.get("versionEndIncluding"):
affected_max.add(m["versionEndIncluding"])
sev = None
for mk in ("cvssMetricV31", "cvssMetricV30", "cvssMetricV2"):
got = (c.get("metrics") or {}).get(mk) or []
if got:
sev = (got[0].get("cvssData") or {}).get("baseSeverity")
break
entry["advisories"].append({
"cve": cid, "ghsa": None, "severity": (sev or "").lower() or None,
"summary": next((d.get("value") for d in c.get("descriptions", [])
if d.get("lang") == "en"), "")[:200],
"vulnerable_range": None,
"patched": "; ".join(sorted(fixed)) or None,
"affected_max": "; ".join(sorted(affected_max)) or None,
"url": f"https://nvd.nist.gov/vuln/detail/{cid}",
"published_at": c.get("published"), "description": None, "cvss": None,
})
return entry
def osv(recipe: str, version: str | None) -> dict | None:
pkg = OSV_PACKAGES.get(recipe)
if not pkg or not version:
@@ -642,7 +737,7 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
"vulnerable_range": None, "patched": None,
"context": None, "published_at": None,
"description": None, "url": None, "cvss": None,
"changelog_fixed_in": None})
"changelog_fixed_in": None, "affected_max": None})
if src not in e["sources"]:
e["sources"].append(src)
for k, v in extra.items():
@@ -659,6 +754,16 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
context=a.get("summary"), published_at=a.get("published_at"),
description=a.get("description"), url=a.get("url"), cvss=a.get("cvss"))
for key, cpe in registry_cpes(recipe, registry_dir):
entry = nvd_advisories(cpe, key)
report["sources"].append({"source": entry["source"], "status": entry["status"],
"found": len(entry.get("advisories") or [])})
for a in entry.get("advisories", []):
record(a["cve"], entry["source"], severity=a.get("severity"),
patched=a.get("patched"), affected_max=a.get("affected_max"),
context=a.get("summary"), published_at=a.get("published_at"),
url=a.get("url"))
for entry in vendor_pages(urls):
report["sources"].append({"source": entry["source"], "status": entry["status"],
"found": len(entry.get("cves", []))})
@@ -699,7 +804,8 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
#
# A source with no window is not classified: its advisories are listed as unclassified so they
# stay visible without inflating the count.
gh_sources = [x["source"] for x in report["sources"] if x["source"].startswith("github-advisories:")]
gh_sources = [x["source"] for x in report["sources"]
if x["source"].startswith(("github-advisories:", "nvd:"))]
primary = gh_sources[0] if (gh_sources and (v_from or v_to)) else None
report["primary_source"] = primary
@@ -708,6 +814,17 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
if primary:
windows[primary] = (v_from, v_to)
window_key[primary] = primary.split("/")[-1]
# The app's window must also cover its NVD entry. NVD sources are keyed by IMAGE name
# (`mattermost-team-edition`) while the advisory feed is keyed by REPO (`mattermost/
# mattermost`), so without this the fallback source that exists precisely because the feed
# is empty would itself go unwindowed — and mumble/mattermost would still report nothing.
pname = primary.split("/")[-1].lower()
for src in gh_sources:
if src.startswith("nvd:") and src not in windows:
k = src.split(":", 1)[1].lower()
if pname in k or k in pname:
windows[src] = (v_from, v_to)
window_key[src] = k
for key, wf, wt in (images or []):
for src in gh_sources:
if src in windows:
@@ -738,12 +855,31 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
continue
patched = e.get("patched") or ""
cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
# NEVER on a placeholder: "7.4.X" parses to the bare 7.4, which then reads as
# "already fixed at 7.4" and silently drops a real fix (redis CVE-2024-46981).
# A placeholder means the fix version is unknown — that is the indeterminate path.
if (kf and kt and not PLACEHOLDER_RE.search(patched)
and _already_fixed_on_from_line(kf, cands)):
# already had it before the upgrade
e.setdefault("classification", "outside-window")
continue
if kf and kt and _superseded_on_target_line(kt, cands):
# The target's own line got the fix LATER than the target: not fixed here.
e.setdefault("classification", "outside-window")
continue
if kf and kt and any(_within(kf, kt, c) for c in cands):
got.add(cve)
elif kf and kt and e.get("affected_max"):
# NVD's `versionEndIncluding X`: affected up to and INCLUDING X, fixed in some
# later release. The exact fix version is unpublished, but the upgrade delivers
# it whenever it crosses X — i.e. from <= X < to.
for t in re.findall(r"\d+(?:\.\d+)*", e["affected_max"]):
x = _vkey(t)
n = max(len(kf), len(kt), len(x))
pad = lambda z: z + (0,) * (n - len(z))
if x and pad(kf) <= pad(x) < pad(kt):
got.add(cve)
break
elif not patched or PLACEHOLDER_RE.search(patched):
# No fix version published ("TBD") or only a placeholder ("7.4.X" — which could
# be 7.4.1, inside the window). We cannot say either way, so say so.