Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 autonomic-bot
|
ef58e33102 | ||
|
autonomic-bot
|
94ce5c4de2 | ||
|
autonomic-bot
|
bb7ebb4a27 | ||
|
autonomic-bot
|
ecf126d98d | ||
|
autonomic-bot
|
ab88e59c21 | ||
|
autonomic-bot
|
e89da2d842 | ||
|
autonomic-bot
|
6c91373357 | ||
|
autonomic-bot
|
1db85a7e77 |
@@ -79,12 +79,29 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
|
||||
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
|
||||
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
|
||||
recipe.
|
||||
- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
|
||||
change must satisfy — set state up through the app's interface rather than its database, gate on
|
||||
version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
|
||||
the app's own telemetry before concluding a test is stale.
|
||||
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
|
||||
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
|
||||
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
|
||||
branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
|
||||
(`/cc-ci`, `/cc-ci-adv`) or their in-flight state.
|
||||
|
||||
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
|
||||
> ```
|
||||
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
|
||||
> ```
|
||||
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
|
||||
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
|
||||
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
|
||||
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
|
||||
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
|
||||
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
|
||||
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
|
||||
> two CVSS-9.8 RCEs" finding.
|
||||
|
||||
### 5. VERIFY each PR on the CI server (deterministic; still never merge)
|
||||
A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
|
||||
the bug. Verification is deterministic (the harness), not an AI judgement.
|
||||
|
||||
@@ -46,12 +46,19 @@ This is `/recipe-upgrade` step 1's research, stopping before it implements anyth
|
||||
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
|
||||
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
|
||||
> after. Only a genuinely dirty TRACKED tree is a skip.
|
||||
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). This is
|
||||
> **not** "not fetchable": enumerate the compose's `image:` refs yourself and check the upstream
|
||||
> registry directly for the ones abra could not read, picking the newest tag the app version
|
||||
> supports rather than the numerically highest.
|
||||
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
|
||||
> hand-check the registry; run the resolver, which is abra-independent and covers every image:
|
||||
> ```
|
||||
> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
|
||||
> ```
|
||||
> It reports, per image, `newest_within_major` (the compatibility-safe pick) and
|
||||
> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
|
||||
> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
|
||||
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
|
||||
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
|
||||
|
||||
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions.** This is the same reconcile
|
||||
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
|
||||
`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
|
||||
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
|
||||
research a stale checkout, and on the first real run that produced **two recipes with no survey output
|
||||
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
|
||||
@@ -66,6 +73,12 @@ nothing; it also auto-closes mirror PRs whose changes upstream has already merge
|
||||
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
|
||||
> `git reset --hard origin/main` on a checkout that tracks `master`.
|
||||
|
||||
**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
|
||||
nothing for images it cannot parse, and it reported "no new versions" for images that did have them
|
||||
(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
|
||||
the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
|
||||
(plausible's clickhouse) carried four CVEs.
|
||||
|
||||
Then read versions:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
|
||||
@@ -312,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**:
|
||||
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
|
||||
stale test; operator to decide).
|
||||
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
|
||||
0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
|
||||
it is written against the failures this pipeline has actually produced. The two that matter most
|
||||
here: **set state up through the app's own interface, never its database** (a plausible fixture
|
||||
that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
|
||||
and **gate on version rather than writing a fixture that supports both** — old-version tests can
|
||||
simply be deleted, since the older version is only exercised through the upgrade tier.
|
||||
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
|
||||
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
|
||||
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
|
||||
|
||||
@@ -73,6 +73,29 @@ done
|
||||
# 5) Stray exited containers (debug one-shots) — best-effort prune.
|
||||
docker container prune -f >/dev/null 2>&1 || true
|
||||
|
||||
# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
|
||||
# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
|
||||
# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
|
||||
# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
|
||||
# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
|
||||
# "failure" and looked like recipe regressions.
|
||||
#
|
||||
# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
|
||||
# `image prune -a` removes only images no container references, so anything deployed (infra +
|
||||
# warm-* canonicals) is untouched; anything else is re-pulled on demand.
|
||||
#
|
||||
# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
|
||||
# data-warm and their volumes are legitimately dangling between runs.
|
||||
DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
|
||||
used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
|
||||
if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
|
||||
echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
|
||||
freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
|
||||
echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
|
||||
else
|
||||
echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
|
||||
fi
|
||||
|
||||
if [ "$removed" -eq 0 ]; then
|
||||
echo "== orphan sweep: clean (nothing to remove) =="
|
||||
else
|
||||
|
||||
@@ -78,8 +78,38 @@ ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls
|
||||
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||||
```
|
||||
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
|
||||
note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable →
|
||||
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
|
||||
HIGH: recommend `hetzner-server-recovery`.
|
||||
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
|
||||
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
|
||||
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
|
||||
> pruned them), the filesystem filled during a run, and the harness died at startup with
|
||||
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
|
||||
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
|
||||
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
|
||||
- **!testme actually produces results** (the check that would have caught the above days earlier):
|
||||
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
|
||||
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
|
||||
it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way.
|
||||
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
|
||||
and the drone step log"*. The step log lives in drone's sqlite
|
||||
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
|
||||
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
|
||||
(distroless container, swarm secret).
|
||||
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
|
||||
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
|
||||
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
|
||||
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
|
||||
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
|
||||
> inode (it recurred afterwards — that apparent fix was coincidence).
|
||||
>
|
||||
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
|
||||
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
|
||||
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
|
||||
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
|
||||
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
|
||||
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
|
||||
> if it becomes disruptive.
|
||||
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
|
||||
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
|
||||
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
|
||||
@@ -113,7 +143,8 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
|
||||
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
|
||||
|
||||
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
|
||||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
|
||||
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
|
||||
producing results.json, disk under
|
||||
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
|
||||
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
|
||||
|
||||
|
||||
@@ -83,8 +83,29 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
||||
changed upstream, what the test currently asserts.
|
||||
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
|
||||
|
||||
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
|
||||
> ```
|
||||
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
|
||||
> ```
|
||||
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
|
||||
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
|
||||
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
|
||||
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
|
||||
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
|
||||
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
|
||||
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
|
||||
> two CVSS-9.8 RCEs" finding.
|
||||
|
||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||
|
||||
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
|
||||
> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
|
||||
> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
|
||||
> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
|
||||
> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
|
||||
> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
|
||||
> wait but **never the assertion**.
|
||||
|
||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||
Swarm). For each `STALE_TESTS` entry:
|
||||
|
||||
|
||||
@@ -39,6 +39,8 @@ keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c i
|
||||
```
|
||||
advisory-scan.py <recipe> [--from <version>] [--to <version>]
|
||||
[--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
|
||||
|
||||
advisory-scan.py <recipe> --compose-to <URL> [--compose-from <URL>] # windows derived, not typed
|
||||
```
|
||||
|
||||
| Input | Meaning |
|
||||
@@ -46,6 +48,8 @@ advisory-scan.py <recipe> [--from <version>] [--to <version>]
|
||||
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
|
||||
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
|
||||
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
|
||||
| `--compose-to URL` | **Derive every window by diffing this compose against its baseline**, instead of typing `--from/--to/--image`. Point it at a PR's `compose.yml`. |
|
||||
| `--compose-from URL` | Baseline for the above. Default: the same repo's **default branch, resolved from the API** — never assumed to be `main`. |
|
||||
| `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
|
||||
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
|
||||
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
|
||||
@@ -140,6 +144,33 @@ Two invariants govern this step, both learned from a wrong answer in production.
|
||||
> `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
|
||||
> cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".
|
||||
|
||||
### 3b. Deriving the windows from a compose diff (`--compose-to`)
|
||||
|
||||
Typing `--from/--to/--image` by hand means someone has to remember that the recipe also bumped its
|
||||
redis. That is how sidecar CVEs went uncounted for months. This mode reads the windows off the diff:
|
||||
|
||||
1. Fetch both compose files (baseline = the repo's **default branch from the API**, since several
|
||||
recipes keep a stale `main` beside a live `master`).
|
||||
2. Parse `{service: (image-repo, tag)}` — keyed by **service, not image repo**, because an upgrade
|
||||
may change the repo itself (plausible moved `plausible/analytics` →
|
||||
`ghcr.io/plausible/community-edition`; keyed by repo that reads as one image vanishing and an
|
||||
unrelated one appearing, losing the app window entirely).
|
||||
3. Every service whose tag or repo changed becomes a window. The `app` service drives `--from/--to`
|
||||
(coop-cloud convention: it is the recipe's primary image); the rest become `--image` windows.
|
||||
Unchanged images produce no window — inventing one would be a false count.
|
||||
4. The derived windows are printed to stderr before the scan, so the inputs are auditable.
|
||||
|
||||
Image names are matched against advisory sources **both ways** — an image name is often longer than
|
||||
its source repo (`clickhouse/clickhouse-server` vs `ClickHouse/ClickHouse`) and sometimes shorter
|
||||
(`redis` vs `redis/redis`).
|
||||
|
||||
Verified on plausible PR #5: from the compose URL alone it derives `v2.0.0 → v3.2.1` plus
|
||||
`clickhouse-server 23.4.2.11-alpine → 24.12-alpine`, and reports **6** — identical to the
|
||||
hand-specified args.
|
||||
|
||||
`--from/--to/--image` remain available for finer-grained checks (scanning a window that is not a
|
||||
literal compose diff, e.g. "what would the compatibility-safe target fix?").
|
||||
|
||||
### 4a. By patched version (preferred — exact)
|
||||
|
||||
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
|
||||
|
||||
+178
-7
@@ -164,6 +164,21 @@ def _vkey(v: str | None) -> tuple:
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _superseded_on_target_line(kt: tuple, cands: list[tuple]) -> bool:
|
||||
"""Does a patched version on the TARGET's own release line sit ABOVE the target?
|
||||
|
||||
Projects maintain several branches at once and backport per branch, so "some patched version is
|
||||
inside the numeric window" is not the same as "the version we land on has the fix". ClickHouse
|
||||
fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20; an upgrade landing on 23.10.4.25 crosses the
|
||||
23.9 fix numerically but is still BELOW its own line's fix, so it does NOT have it. When the
|
||||
advisory names a fix on the target's own line and the target is older than it, that is proof of
|
||||
absence and outranks any other candidate."""
|
||||
if len(kt) < 2:
|
||||
return False
|
||||
line = kt[:2]
|
||||
return any(c[:2] == line and c > kt for c in cands if len(c) >= 2)
|
||||
|
||||
|
||||
def _within(kf: tuple, kt: tuple, c: tuple) -> bool:
|
||||
"""Is patched-version `c` inside the window (kf, kt] — exclusive lower, inclusive upper?
|
||||
|
||||
@@ -645,7 +660,14 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
windows[primary] = (v_from, v_to)
|
||||
for key, wf, wt in (images or []):
|
||||
for src in gh_sources:
|
||||
if key.lower() in src.lower() and src not in windows:
|
||||
if src in windows:
|
||||
continue
|
||||
# Match BOTH ways: an image name is often longer than its source repo
|
||||
# (`clickhouse/clickhouse-server` vs source `ClickHouse/ClickHouse`) and sometimes
|
||||
# shorter (`redis` vs `redis/redis`). One-directional matching silently dropped the
|
||||
# clickhouse window when the key was derived from a compose file.
|
||||
k, name = key.lower(), src.split("/")[-1].lower()
|
||||
if k in src.lower() or name in k:
|
||||
windows[src] = (wf, wt)
|
||||
report["windows"] = {k: {"from": f, "to": t} for k, (f, t) in windows.items()}
|
||||
|
||||
@@ -665,6 +687,10 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
continue
|
||||
patched = e.get("patched") or ""
|
||||
cands = [_vkey(t) for t in re.findall(r"\d+(?:\.\d+)*", patched)]
|
||||
if kf and kt and _superseded_on_target_line(kt, cands):
|
||||
# The target's own line got the fix LATER than the target: not fixed here.
|
||||
e.setdefault("classification", "outside-window")
|
||||
continue
|
||||
if kf and kt and any(_within(kf, kt, c) for c in cands):
|
||||
got.add(cve)
|
||||
elif not patched or PLACEHOLDER_RE.search(patched):
|
||||
@@ -778,8 +804,19 @@ def scan(recipe: str, v_from: str | None, v_to: str | None, registry_dir: str,
|
||||
report["unclassified"] = sorted(unknown)
|
||||
# NEVER report 0 for something we could not determine — a 0 asserts safety. If ANY requested
|
||||
# window could not be ordered at all, the total is UNKNOWN rather than a partial number.
|
||||
report["count_known"] = not unresolved_any
|
||||
report["cve_count_fixed"] = len(fixed_set) if not unresolved_any else None
|
||||
# A scan with NO usable source has not measured anything, so it must not report a number —
|
||||
# least of all 0, which asserts safety. mumble had no cc-ci-plan/upstream/mumble.md at all and
|
||||
# still produced "0 identified", which was then published as a clean 0 in a CVE report.
|
||||
usable_sources = [
|
||||
s for s in report["sources"]
|
||||
if s["status"] == "ok" or s["status"].startswith("no-advisories-published")
|
||||
]
|
||||
no_sources = not usable_sources
|
||||
if no_sources:
|
||||
report["no_usable_sources"] = True
|
||||
report["count_known"] = not unresolved_any and not no_sources
|
||||
report["cve_count_fixed"] = (len(fixed_set)
|
||||
if (not unresolved_any and not no_sources) else None)
|
||||
report["cve_count_total_seen"] = len(report["cves"])
|
||||
# Only GENUINE failures make a count unreliable. "no-advisories-published" (404: the repo has
|
||||
# no advisory feed) and "skipped: template URL" are benign and must not degrade the verdict.
|
||||
@@ -802,10 +839,16 @@ def markdown(rep: dict) -> str:
|
||||
f"{rep.get('from') or '?'} → {rep.get('to') or '?'}"]
|
||||
if not rep.get("count_known", True):
|
||||
L.append("\n**CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count.**")
|
||||
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes numeric "
|
||||
"ordering meaningless across this jump, so no advisory could be classified. Render "
|
||||
"this recipe's cve cell as `?`, never `0`. Read the vendor's release notes for the "
|
||||
"jump and count by hand.")
|
||||
if rep.get("no_usable_sources"):
|
||||
L.append("\n⚠ This is NOT zero. **No usable source was checked at all** — the registry "
|
||||
"file `cc-ci-plan/upstream/<recipe>.md` is missing or every source failed, so "
|
||||
"nothing was measured. Render this recipe's cve cell as `?`, never `0`, and add "
|
||||
"the registry file.")
|
||||
else:
|
||||
L.append("\n⚠ This is NOT zero. A version-scheme change (e.g. semver → calver) makes "
|
||||
"numeric ordering meaningless across this jump, so no advisory could be "
|
||||
"classified. Render this recipe's cve cell as `?`, never `0`. Read the vendor's "
|
||||
"release notes for the jump and count by hand.")
|
||||
if rep["unclassified"]:
|
||||
L.append(f"\nAdvisories seen but unclassifiable ({len(rep['unclassified'])}) — includes "
|
||||
f"other images in this recipe: " + ", ".join(rep["unclassified"][:12]))
|
||||
@@ -857,6 +900,113 @@ def markdown(rep: dict) -> str:
|
||||
return "\n".join(L)
|
||||
|
||||
|
||||
def _gitea_auth(url: str) -> dict:
|
||||
"""Basic auth for the private mirror, from /srv/cc-ci/.testenv.
|
||||
|
||||
Sent as a HEADER, never embedded in the URL: in-URL credentials leak into shell history, process
|
||||
lists and error messages, and urllib mis-parses a password containing a colon."""
|
||||
host = re.sub(r"^https?://", "", url).split("/")[0]
|
||||
env = {}
|
||||
try:
|
||||
for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
|
||||
if "=" in ln and not ln.strip().startswith("#"):
|
||||
k, v = ln.strip().split("=", 1)
|
||||
env[k] = v.strip().strip("\"'")
|
||||
except OSError:
|
||||
return {}
|
||||
if host != env.get("GITEA_URL", "git.autonomic.zone"):
|
||||
return {}
|
||||
u, pw = env.get("GITEA_USERNAME"), env.get("GITEA_PASSWORD")
|
||||
if not (u and pw):
|
||||
return {}
|
||||
import base64 as _b64
|
||||
return {"Authorization": "Basic " + _b64.b64encode(f"{u}:{pw}".encode()).decode()}
|
||||
|
||||
|
||||
def _compose_images(url: str) -> dict[str, tuple[str, str]]:
|
||||
"""{service: (image-repo, tag)} for a compose file.
|
||||
|
||||
Keyed by SERVICE, not by image repo, because an upgrade may change the repo itself: plausible
|
||||
moved `plausible/analytics` -> `ghcr.io/plausible/community-edition`. Keyed by repo that reads
|
||||
as one image vanishing and an unrelated one appearing, and the app's version window is lost —
|
||||
which is exactly the upgrade most worth scanning."""
|
||||
txt = _fetch(url, _gitea_auth(url))
|
||||
out, svc = {}, None
|
||||
in_services = False
|
||||
for line in txt.splitlines():
|
||||
if re.match(r"^services:\s*$", line):
|
||||
in_services = True
|
||||
continue
|
||||
if in_services and re.match(r"^\S", line):
|
||||
in_services = False
|
||||
if not in_services:
|
||||
continue
|
||||
m = re.match(r"^ (\S+):\s*$", line)
|
||||
if m:
|
||||
svc = m.group(1)
|
||||
continue
|
||||
m = re.match(r"^\s+image:\s*[\"']?([^\"'\s]+)", line)
|
||||
if m and svc:
|
||||
ref = m.group(1).split("@", 1)[0]
|
||||
if "${" in ref or "$(" in ref:
|
||||
continue
|
||||
repo, _, tag = ref.rpartition(":")
|
||||
if repo and tag:
|
||||
out[svc] = (repo, tag)
|
||||
return out
|
||||
|
||||
|
||||
def _default_branch_compose(url: str) -> str | None:
|
||||
"""Same repo as `url`, but its DEFAULT branch — resolved from the API, never assumed.
|
||||
|
||||
Several coopcloud recipes keep a stale `main` beside the real default `master` (gitea's `main`
|
||||
is 1.24.2-rootless while `master` has 1.27.1-rootless), so guessing the branch produces a
|
||||
confidently wrong baseline."""
|
||||
m = re.match(r"(https?://[^/]+)/([^/]+)/([^/]+)/(?:raw|src)/branch/[^/]+/(.*)$", url)
|
||||
if not m:
|
||||
return None
|
||||
host, owner, repo, path = m.groups()
|
||||
try:
|
||||
meta = json.loads(_fetch(f"{host}/api/v1/repos/{owner}/{repo}", _gitea_auth(host)))
|
||||
br = meta.get("default_branch")
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
return f"{host}/{owner}/{repo}/raw/branch/{br}/{path}" if br else None
|
||||
|
||||
|
||||
def windows_from_compose(to_url: str, from_url: str | None = None) -> tuple[list, str | None]:
|
||||
"""Derive the scan's version windows by DIFFING two compose files.
|
||||
|
||||
This is the deterministic alternative to a human (or a model) deciding which `--image` args a
|
||||
given upgrade needs. Point it at a PR's compose and it reads the windows straight off the diff:
|
||||
every image whose tag changed becomes a window, every image that did not change is correctly
|
||||
left out, and nothing depends on anyone remembering that the recipe also bumped its redis.
|
||||
|
||||
Returns (windows, note) where windows is [(image-name, from, to)].
|
||||
"""
|
||||
if from_url is None:
|
||||
from_url = _default_branch_compose(to_url)
|
||||
if not from_url:
|
||||
raise SystemExit("could not resolve a baseline compose; pass --compose-from explicitly")
|
||||
new, old = _compose_images(to_url), _compose_images(from_url)
|
||||
app, others = None, []
|
||||
for svc, (repo, tag) in sorted(new.items()):
|
||||
if svc not in old:
|
||||
continue
|
||||
prev_repo, prev_tag = old[svc]
|
||||
if prev_tag == tag and prev_repo == repo:
|
||||
continue
|
||||
# The `app` service is the recipe's primary image by coop-cloud convention; its window drives
|
||||
# --from/--to so the scan's primary advisory source is judged against it. Everything else is
|
||||
# a sidecar window keyed by its image name.
|
||||
if svc == "app":
|
||||
app = (repo.split("/")[-1], prev_tag, tag)
|
||||
else:
|
||||
others.append((repo.split("/")[-1], prev_tag, tag))
|
||||
wins = ([app] if app else []) + others
|
||||
return wins, f"baseline {from_url}"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("recipe")
|
||||
@@ -868,6 +1018,13 @@ def main() -> int:
|
||||
"fix version published), fetch their full text + references and append a "
|
||||
"block for the agent to judge. Additive: it never changes the count above.")
|
||||
ap.add_argument("--registry", default=REGISTRY_DIR)
|
||||
ap.add_argument("--compose-to", default=None, metavar="URL",
|
||||
help="derive the windows by DIFFING this compose against its baseline, instead "
|
||||
"of passing --from/--to/--image by hand. Point it at a PR's compose.yml "
|
||||
"(e.g. .../raw/branch/<pr-branch>/compose.yml).")
|
||||
ap.add_argument("--compose-from", default=None, metavar="URL",
|
||||
help="baseline compose for --compose-to. Default: the same repo's DEFAULT "
|
||||
"branch, resolved from the API (never assumed to be `main`).")
|
||||
ap.add_argument("--image", action="append", default=[], metavar="NAME=FROM:TO",
|
||||
help="a sidecar image and the versions it moved between, e.g. "
|
||||
"--image redis=7.4:8.10 (repeatable). NAME matches a source repo name; "
|
||||
@@ -875,6 +1032,20 @@ def main() -> int:
|
||||
"being left unclassified.")
|
||||
a = ap.parse_args()
|
||||
images = []
|
||||
if a.compose_to:
|
||||
wins, note = windows_from_compose(a.compose_to, a.compose_from)
|
||||
if not wins:
|
||||
print(f"### Advisory scan — {a.recipe}\n\n**No image versions changed between the two "
|
||||
f"compose files, so this upgrade fixes no CVEs by definition.**\n\n_{note}_")
|
||||
return 0
|
||||
print(f"_derived from compose diff ({note}):_", file=sys.stderr)
|
||||
for n_, f_, t_ in wins:
|
||||
print(f"_ {n_}: {f_} → {t_}_", file=sys.stderr)
|
||||
# The `app` service (first entry when present) drives --from/--to; the rest are --image
|
||||
# windows. Passing every window as --image too is harmless: each is matched by name against
|
||||
# the advisory sources, and an unmatched name is simply ignored.
|
||||
a.v_from, a.v_to = a.v_from or wins[0][1], a.v_to or wins[0][2]
|
||||
images = list(wins[1:])
|
||||
for spec in a.image:
|
||||
name, _, rng = spec.partition('=')
|
||||
vf, _, vt = rng.partition(':')
|
||||
|
||||
Executable
+238
@@ -0,0 +1,238 @@
|
||||
#!/usr/bin/env python3
|
||||
"""audit-sources — are we still looking in the right place for each recipe's updates?
|
||||
|
||||
A recipe tracks an image repo and a set of registry URLs. Upstreams move: they rename the image,
|
||||
switch registry, archive the GitHub repo, or split a community edition out of the original. When that
|
||||
happens nothing errors — the old repo simply stops receiving tags, and the recipe looks "up to date"
|
||||
forever while real releases happen somewhere else.
|
||||
|
||||
plausible is the worked example. It tracked `plausible/analytics` on Docker Hub; upstream moved to
|
||||
`ghcr.io/plausible/community-edition`. The old repo still exists and still serves v2.0.0, so every
|
||||
survey said "no upgrades available" while v3 shipped elsewhere.
|
||||
|
||||
This reports the signals that catch that, per image and per registry URL:
|
||||
|
||||
* IMAGE GONE QUIET — newest tag is older than --quiet-days (default 365). The single strongest
|
||||
signal that releases moved somewhere else.
|
||||
* DEPRECATION WORDING — the registry description says deprecated / moved / no longer maintained.
|
||||
* GITHUB REPO ARCHIVED — upstream archived it.
|
||||
* GITHUB REPO RENAMED — the API redirects to a different owner/name than we ask for.
|
||||
* GITHUB REPO GONE — 404.
|
||||
|
||||
Everything is a SIGNAL, not a verdict: a genuinely stable image (mumble, custom-html) can be quiet
|
||||
for good reason. The output is for a human to judge, so each finding says what was measured.
|
||||
|
||||
audit-sources.py [recipe ...] [--ssh HOST] [--quiet-days N] [--json]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
_spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HERE, "resolve-images.py"))
|
||||
RI = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(RI)
|
||||
|
||||
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream"))
|
||||
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
|
||||
DEPRECATION_RE = re.compile(
|
||||
r"\b(deprecat|no longer maintain|unmaintained|superseded|moved to|migrated to|"
|
||||
r"has moved|discontinued|end.of.life|archived)\b", re.I)
|
||||
|
||||
|
||||
def _days_since(iso: str | None) -> int | None:
|
||||
if not iso:
|
||||
return None
|
||||
try:
|
||||
d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
return (datetime.now(timezone.utc) - d).days
|
||||
|
||||
|
||||
def hub_repo_meta(repo: str) -> dict:
|
||||
"""Docker Hub repo metadata: when it was last pushed to, and how it describes itself."""
|
||||
try:
|
||||
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}", RI._hub_auth())
|
||||
except urllib.error.HTTPError as e:
|
||||
return {"status": f"HTTP {e.code}"}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"status": f"{type(e).__name__}"}
|
||||
text = f"{d.get('description') or ''}\n{d.get('full_description') or ''}"
|
||||
m = DEPRECATION_RE.search(text)
|
||||
return {"status": "ok", "last_updated": d.get("last_updated"),
|
||||
"deprecation_hint": (m.group(0) if m else None),
|
||||
"archived": bool(d.get("is_archived") or d.get("status") == "inactive")}
|
||||
|
||||
|
||||
def github_repo_meta(owner: str, repo: str) -> dict:
|
||||
"""GitHub repo state — archived, renamed (the API answers with the CURRENT full_name), or gone."""
|
||||
hdrs = {"Accept": "application/vnd.github+json"}
|
||||
tok = RI._gh_token()
|
||||
if tok:
|
||||
hdrs["Authorization"] = f"Bearer {tok}"
|
||||
try:
|
||||
d = RI._json(f"https://api.github.com/repos/{owner}/{repo}", hdrs)
|
||||
except urllib.error.HTTPError as e:
|
||||
return {"status": f"HTTP {e.code}"}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"status": f"{type(e).__name__}"}
|
||||
asked, got = f"{owner}/{repo}".lower(), (d.get("full_name") or "").lower()
|
||||
return {"status": "ok", "archived": bool(d.get("archived")), "pushed_at": d.get("pushed_at"),
|
||||
"renamed_to": (d.get("full_name") if got and got != asked else None),
|
||||
"description": d.get("description") or ""}
|
||||
|
||||
|
||||
def newest_tag_date(registry: str, repo: str, tag: str) -> str | None:
|
||||
"""When was the repo's newest same-shape tag pushed? Docker Hub only (it dates its tags)."""
|
||||
if registry not in ("docker.io", "registry-1.docker.io"):
|
||||
return None
|
||||
try:
|
||||
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}/tags"
|
||||
f"?page_size=100&ordering=last_updated", RI._hub_auth())
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
want = RI.shape(tag)
|
||||
for row in d.get("results", []):
|
||||
if RI.shape(row.get("name") or "") == want:
|
||||
return row.get("last_updated")
|
||||
return (d.get("results") or [{}])[0].get("last_updated")
|
||||
|
||||
|
||||
def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
|
||||
out = {"recipe": recipe, "findings": [], "images": [], "sources": []}
|
||||
try:
|
||||
refs = (RI.compose_images_ssh(recipe, ssh, "~/.abra/recipes") if ssh
|
||||
else RI.compose_images(recipe, RI.RECIPE_DIR))
|
||||
except Exception as e: # noqa: BLE001
|
||||
out["findings"].append({"level": "error", "what": f"could not read compose: {e}"})
|
||||
return out
|
||||
|
||||
for ref in refs:
|
||||
if "${" in ref:
|
||||
continue
|
||||
info = RI.parse_ref(ref)
|
||||
row = {"ref": ref, "registry": info["registry"], "repo": info["repo"], "tag": info["tag"]}
|
||||
if info["registry"] in ("docker.io", "registry-1.docker.io"):
|
||||
meta = hub_repo_meta(info["repo"])
|
||||
row.update(meta)
|
||||
newest = newest_tag_date(info["registry"], info["repo"], info["tag"])
|
||||
row["newest_tag_pushed"] = newest
|
||||
age = _days_since(newest)
|
||||
row["newest_tag_age_days"] = age
|
||||
if age is not None and age > quiet_days:
|
||||
out["findings"].append({
|
||||
"level": "warn", "what": "image has gone quiet",
|
||||
"detail": f"{info['repo']}: newest {RI.shape(info['tag'])}-shaped tag pushed "
|
||||
f"{age} days ago — releases may have moved elsewhere"})
|
||||
if meta.get("deprecation_hint"):
|
||||
out["findings"].append({
|
||||
"level": "warn", "what": "registry text suggests deprecation",
|
||||
"detail": f"{info['repo']}: says {meta['deprecation_hint']!r}"})
|
||||
if meta.get("archived"):
|
||||
out["findings"].append({"level": "warn", "what": "registry repo archived/inactive",
|
||||
"detail": info["repo"]})
|
||||
out["images"].append(row)
|
||||
|
||||
urls, reg_path = ([], None)
|
||||
try:
|
||||
urls, reg_path = _registry_urls(recipe)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
if reg_path is None:
|
||||
out["findings"].append({"level": "warn", "what": "no upstream registry file",
|
||||
"detail": f"cc-ci-plan/upstream/{recipe}.md is missing — the advisory "
|
||||
f"scan has nowhere to look"})
|
||||
seen = set()
|
||||
for u in urls:
|
||||
m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u)
|
||||
if not m:
|
||||
continue
|
||||
owner, repo = m.group(1), m.group(2).removesuffix(".git")
|
||||
if (owner, repo) in seen:
|
||||
continue
|
||||
seen.add((owner, repo))
|
||||
meta = github_repo_meta(owner, repo)
|
||||
row = {"repo": f"{owner}/{repo}", **meta}
|
||||
age = _days_since(meta.get("pushed_at"))
|
||||
row["pushed_age_days"] = age
|
||||
out["sources"].append(row)
|
||||
if meta.get("status") != "ok":
|
||||
out["findings"].append({"level": "warn", "what": "registry source unreachable",
|
||||
"detail": f"{owner}/{repo}: {meta['status']}"})
|
||||
continue
|
||||
if meta.get("renamed_to"):
|
||||
out["findings"].append({"level": "alert", "what": "GitHub repo has MOVED",
|
||||
"detail": f"{owner}/{repo} now answers as {meta['renamed_to']}"})
|
||||
if meta.get("archived"):
|
||||
out["findings"].append({"level": "alert", "what": "GitHub repo is ARCHIVED",
|
||||
"detail": f"{owner}/{repo} — upstream development has stopped here"})
|
||||
if age is not None and age > quiet_days:
|
||||
out["findings"].append({"level": "warn", "what": "GitHub repo quiet",
|
||||
"detail": f"{owner}/{repo}: last push {age} days ago"})
|
||||
return out
|
||||
|
||||
|
||||
def _registry_urls(recipe: str):
|
||||
path = os.path.join(REGISTRY_DIR, f"{recipe}.md")
|
||||
if not os.path.exists(path):
|
||||
return [], None
|
||||
text = open(path).read()
|
||||
urls = []
|
||||
for u in re.findall(r"https?://[^\s)|\]]+", text):
|
||||
u = u.rstrip("`'\"*.,;:>)")
|
||||
if u and u not in urls:
|
||||
urls.append(u)
|
||||
return urls, path
|
||||
|
||||
|
||||
def all_recipes() -> list[str]:
|
||||
out = []
|
||||
for ln in open(USED_RECIPES):
|
||||
ln = ln.strip()
|
||||
if not ln or ln.startswith("#") or ln.startswith("`"):
|
||||
continue
|
||||
parts = ln.split()
|
||||
if len(parts) >= 2 and parts[1] in ("weekly", "external"):
|
||||
out.append(parts[0])
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("recipes", nargs="*")
|
||||
ap.add_argument("--ssh", default=None)
|
||||
ap.add_argument("--quiet-days", type=int, default=365)
|
||||
ap.add_argument("--json", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
recipes = a.recipes or all_recipes()
|
||||
reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes]
|
||||
if a.json:
|
||||
print(json.dumps(reports, indent=2))
|
||||
return 0
|
||||
alerts = 0
|
||||
for rep in reports:
|
||||
fs = rep["findings"]
|
||||
mark = "OK " if not fs else ("!! " if any(f["level"] == "alert" for f in fs) else " ? ")
|
||||
print(f"{mark} {rep['recipe']}")
|
||||
for f in fs:
|
||||
alerts += f["level"] == "alert"
|
||||
print(f" [{f['level']}] {f['what']}: {f.get('detail','')}")
|
||||
print(f"\n{len(reports)} recipes audited · "
|
||||
f"{sum(len(r['findings']) for r in reports)} findings · {alerts} alerts")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -12,7 +12,7 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica
|
||||
publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
|
||||
archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
|
||||
(a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
|
||||
newest first, each row suffixed "full report" or "CVE check"; the distinct
|
||||
newest first, each row suffixed "full" or "CVE check"; the distinct
|
||||
filename prefix just stops a sweep overwriting a weekly edition.
|
||||
|
||||
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
|
||||
@@ -50,7 +50,7 @@ INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
|
||||
HOST_REPORTS = "/var/lib/cc-ci-reports"
|
||||
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
|
||||
# prefix -> (page title, index label)
|
||||
KINDS = {"week": ("The Recipe Report", "Week of {d} — full report"),
|
||||
KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),
|
||||
"cve": ("The Recipe Report — CVE check", "{d} — CVE check")}
|
||||
|
||||
|
||||
|
||||
Executable
+64
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env bash
|
||||
# reconcile-upstream — sync recipe mirrors from TRUE upstream. Run this FIRST, always.
|
||||
# ----------------------------------------------------------------------------------
|
||||
# Every recipe we maintain is a MIRROR of a coopcloud recipe. Work done against a stale
|
||||
# mirror is wasted or wrong, in three ways we have actually hit:
|
||||
#
|
||||
# 1. A PR whose changes upstream ALREADY MERGED. mailu #6 (2024.06.57 + redis 8.10,
|
||||
# two internet-facing Roundcube CVEs) sat open and was reported as the fix for
|
||||
# those CVEs — while upstream had merged and released it as 3.1.3+2024.06.57. The
|
||||
# work was done; only our mirror was behind.
|
||||
# 2. A survey that reads the stale mirror and reports "no upgrades available", so a
|
||||
# recipe silently drops out of the weekly run.
|
||||
# 3. Reading the WRONG BRANCH. Several coopcloud recipes keep a stale `main` beside
|
||||
# the real default `master` — gitea's `main` is at 1.24.2-rootless while `master`
|
||||
# has 1.27.1-rootless plus the merged PRs. Reading `main` there says the recipe is
|
||||
# three releases behind and missing two CVSS-9.8 RCE fixes, which reads exactly
|
||||
# like a real finding. open-recipe-pr.sh resolves the default branch itself
|
||||
# (main OR master) — never hand-pick one.
|
||||
#
|
||||
# This is deterministic: it force-syncs each mirror's `main` to upstream's default
|
||||
# branch and closes any mirror PR whose changes are already upstream. No AI judgement.
|
||||
#
|
||||
# reconcile-upstream.sh <recipe>... # specific recipes
|
||||
# reconcile-upstream.sh --all # every recipe in used-recipes.md
|
||||
#
|
||||
# Safe to run repeatedly; a mirror already in sync is a no-op. Recipe work lives in
|
||||
# BRANCHES, never on mirror `main`, so force-syncing `main` discards nothing.
|
||||
set -o errexit -o nounset -o pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ORCH="$(dirname "$HERE")"
|
||||
SSH="${SSH:-cc-ci}"
|
||||
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
|
||||
RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
|
||||
USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
|
||||
|
||||
[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
|
||||
set -a; . "$TESTENV"; set +a
|
||||
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
|
||||
|
||||
if [ "${1:-}" = "--all" ]; then
|
||||
mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
|
||||
else
|
||||
[ "$#" -gt 0 ] || { echo "usage: reconcile-upstream.sh <recipe>... | --all" >&2; exit 2; }
|
||||
RECIPES=("$@")
|
||||
fi
|
||||
|
||||
synced=0; closed=0; failed=0
|
||||
for r in "${RECIPES[@]}"; do
|
||||
echo "── $r"
|
||||
if out="$(ssh "$SSH" "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s $r --reconcile-only" < "$RECONCILE" 2>&1)"; then
|
||||
printf '%s\n' "$out" | grep -E "Force-syncing|already in sync|closed PR|still open|✓" | sed 's/^/ /' || true
|
||||
synced=$((synced + 1))
|
||||
closed=$((closed + $(printf '%s' "$out" | grep -c "closed PR" || true)))
|
||||
else
|
||||
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
|
||||
echo " ✗ FAILED — do NOT proceed against this mirror until it reconciles"
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
echo
|
||||
echo "reconcile-upstream: ${synced} mirror(s) synced, ${closed} already-upstream PR(s) closed, ${failed} failed"
|
||||
[ "$failed" -eq 0 ]
|
||||
@@ -0,0 +1,459 @@
|
||||
#!/usr/bin/env python3
|
||||
"""resolve-images — what version is each of a recipe's images on, and what is newest?
|
||||
|
||||
An abra-independent version resolver. `abra recipe upgrade` is the normal path, but it has a hard
|
||||
failure mode: an image pinned with BOTH a tag and a digest makes it FATA and abandon the WHOLE
|
||||
recipe — even images it already parsed. immich pins two that way:
|
||||
|
||||
ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf6…
|
||||
docker.io/valkey/valkey:9@sha256:3acc…
|
||||
|
||||
so immich contributes NO version data at all and silently drops out of every survey. That is
|
||||
indistinguishable from "up to date" unless a human notices the missing row — which is exactly how it
|
||||
kept getting skipped, and why a CVE sweep reported it as unknown.
|
||||
|
||||
This reads the compose files directly and queries the registries itself, so a digest pin is just a
|
||||
digest pin. Output is JSON (default) or a table.
|
||||
|
||||
resolve-images.py <recipe> [--ssh HOST] [--recipe-dir DIR] [--table] [--only IMAGE]
|
||||
|
||||
The cc-ci host has no python3, so `--ssh cc-ci` reads the compose files from that host's checkout
|
||||
over ssh and does the resolving locally. That keeps the source of truth the SAME tree abra and CI
|
||||
use, rather than a second copy that can drift.
|
||||
|
||||
TAG SHAPES. Registries mix wildly different tag conventions in one repo, so "newest" is meaningless
|
||||
without a shape. Each tag is reduced to a signature by replacing digit runs with '#':
|
||||
|
||||
v3.1.0 -> v#.#.#
|
||||
1.27.1-rootless -> #.#.#-rootless
|
||||
8.10-alpine -> #.#-alpine
|
||||
14-vectorchord0.4.3-pgvectors0.2.0 -> #-vectorchord#.#.#-pgvectors#.#.#
|
||||
|
||||
Only tags sharing the CURRENT pin's shape are candidates. That keeps `-alpine` on `-alpine`, and
|
||||
stops a `latest`/`release`/`sha-…` tag from ever being proposed as an upgrade.
|
||||
|
||||
TWO ANSWERS, NOT ONE. It reports `newest_same_shape` AND `newest_within_major` (same leading number).
|
||||
For a plain app image they usually agree. For a compatibility-pinned sidecar they do not, and taking
|
||||
the max would be wrong: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
|
||||
versions that immich-server is built against, so jumping pg major because a newer tag exists breaks
|
||||
the deployment. The caller picks; this tool refuses to guess and shows both.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import shlex
|
||||
import subprocess
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
UA = "cc-ci-resolve-images (+https://git.autonomic.zone/recipe-maintainers/cc-ci)"
|
||||
TIMEOUT = int(os.environ.get("RESOLVE_IMAGES_TIMEOUT", "45"))
|
||||
RECIPE_DIR = os.environ.get("ABRA_RECIPE_DIR", os.path.expanduser("~/.abra/recipes"))
|
||||
MAX_TAG_PAGES = int(os.environ.get("RESOLVE_IMAGES_MAX_PAGES", "40"))
|
||||
|
||||
IMAGE_RE = re.compile(r"""^\s*image:\s*["']?([^"'\s]+)["']?\s*$""", re.M)
|
||||
|
||||
|
||||
RETRIES = int(os.environ.get("RESOLVE_IMAGES_RETRIES", "4"))
|
||||
|
||||
|
||||
def _fetch(url: str, headers: dict | None = None) -> bytes:
|
||||
"""GET with backoff on rate limits.
|
||||
|
||||
Docker Hub throttles anonymous clients hard, and a sweep re-reads the same popular repos
|
||||
(nginx, redis, postgres) for recipe after recipe. A 429 mid-sweep used to surface as
|
||||
'unresolved', which is indistinguishable from a real lookup failure — so retry, and let the
|
||||
per-repo cache below remove most of the requests entirely."""
|
||||
h = {"User-Agent": UA, "Accept-Encoding": "gzip"}
|
||||
h.update(headers or {})
|
||||
delay = 2.0
|
||||
for attempt in range(RETRIES):
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r:
|
||||
raw = r.read()
|
||||
if r.headers.get("Content-Encoding") == "gzip":
|
||||
raw = gzip.decompress(raw)
|
||||
return raw
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code in (429, 503) and attempt < RETRIES - 1:
|
||||
time.sleep(delay)
|
||||
delay *= 2
|
||||
continue
|
||||
raise
|
||||
raise RuntimeError("unreachable")
|
||||
|
||||
|
||||
_HUB_JWT: list = []
|
||||
|
||||
|
||||
def _hub_auth() -> dict:
|
||||
"""Authenticated Docker Hub calls get a far higher rate limit than anonymous ones.
|
||||
|
||||
Credentials come from /srv/cc-ci/.testenv (DOCKERHUB_USERNAME / DOCKERHUB_TOKEN), the same pair
|
||||
the CI host already uses. Absent creds are fine — the sweep just runs anonymous and slower."""
|
||||
if _HUB_JWT:
|
||||
return _HUB_JWT[0]
|
||||
env = {}
|
||||
try:
|
||||
for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
|
||||
if "=" in ln and not ln.strip().startswith("#"):
|
||||
k, v = ln.strip().split("=", 1)
|
||||
env[k] = v.strip().strip("\"'")
|
||||
except OSError:
|
||||
pass
|
||||
u = os.environ.get("DOCKERHUB_USERNAME") or env.get("DOCKERHUB_USERNAME")
|
||||
t = os.environ.get("DOCKERHUB_TOKEN") or env.get("DOCKERHUB_TOKEN")
|
||||
hdrs = {}
|
||||
if u and t:
|
||||
try:
|
||||
body = json.dumps({"username": u, "password": t}).encode()
|
||||
req = urllib.request.Request("https://hub.docker.com/v2/users/login",
|
||||
data=body, method="POST",
|
||||
headers={"Content-Type": "application/json", "User-Agent": UA})
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
tokj = json.load(r).get("token")
|
||||
if tokj:
|
||||
hdrs = {"Authorization": f"JWT {tokj}"}
|
||||
except Exception: # noqa: BLE001 — anonymous is a valid fallback
|
||||
hdrs = {}
|
||||
_HUB_JWT.append(hdrs)
|
||||
return hdrs
|
||||
|
||||
|
||||
def _json(url: str, headers: dict | None = None):
|
||||
return json.loads(_fetch(url, headers))
|
||||
|
||||
|
||||
def shape(tag: str) -> str:
|
||||
"""Signature of a tag with every digit run replaced by '#'. See module docstring."""
|
||||
return re.sub(r"\d+", "#", tag)
|
||||
|
||||
|
||||
def vkey(tag: str) -> tuple:
|
||||
"""Ordering key: every number in the tag, in order. '1.27.10' > '1.27.9'; text ignored."""
|
||||
return tuple(int(x) for x in re.findall(r"\d+", tag))
|
||||
|
||||
|
||||
def parse_ref(ref: str) -> dict:
|
||||
"""Split an image reference into registry / repo / tag / digest."""
|
||||
digest = None
|
||||
if "@" in ref:
|
||||
ref, _, digest = ref.partition("@")
|
||||
host, repo, tag = "docker.io", ref, "latest"
|
||||
# A leading component is a REGISTRY only when there is a path after it. Without the slash test,
|
||||
# a bare `postgres:15.18` looks like host "postgres:15.18" because of the tag's colon — which
|
||||
# silently sent every library image to a nonexistent registry.
|
||||
if "/" in ref:
|
||||
first = ref.split("/")[0]
|
||||
if "." in first or ":" in first or first == "localhost":
|
||||
host, _, repo = ref.partition("/")
|
||||
if ":" in repo.split("/")[-1]:
|
||||
repo, _, tag = repo.rpartition(":")
|
||||
if host == "docker.io" and "/" not in repo:
|
||||
repo = f"library/{repo}" # bare `redis` is really `library/redis`
|
||||
return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
|
||||
|
||||
|
||||
HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10"))
|
||||
|
||||
|
||||
def _hub_tag_exists(repo: str, tag: str) -> bool:
|
||||
try:
|
||||
_json(f"https://hub.docker.com/v2/repositories/{repo}/tags/{tag}", _hub_auth())
|
||||
return True
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
|
||||
|
||||
def _hub_tags(repo: str) -> list[str]:
|
||||
"""Recently-pushed tags, newest first.
|
||||
|
||||
Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical —
|
||||
but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so
|
||||
ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade.
|
||||
(ghcr offers no ordering, which is why that path needs a different strategy.)
|
||||
"""
|
||||
tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
|
||||
f"?page_size=100&ordering=last_updated")
|
||||
auth = _hub_auth()
|
||||
for _ in range(HUB_RECENT_PAGES):
|
||||
d = _json(url, auth)
|
||||
tags += [r["name"] for r in d.get("results", [])]
|
||||
url = d.get("next")
|
||||
if not url:
|
||||
break
|
||||
return tags
|
||||
|
||||
|
||||
def _oci_bearer(host: str, repo: str) -> dict:
|
||||
"""Token for an OCI registry, discovered from its own auth challenge.
|
||||
|
||||
Registries do NOT share a token endpoint. ghcr answers at /token?scope=…&service=ghcr.io, but
|
||||
lscr.io and dock.mau.dev advertise different realms, and assuming ghcr's shape made both 401 —
|
||||
which then read as "could not resolve" rather than "asked the wrong URL". The registry tells us
|
||||
where to go in its WWW-Authenticate header; use that."""
|
||||
try:
|
||||
urllib.request.urlopen(
|
||||
urllib.request.Request(f"https://{host}/v2/{repo}/tags/list?n=1",
|
||||
headers={"User-Agent": UA}), timeout=TIMEOUT)
|
||||
return {} # no auth needed
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code != 401:
|
||||
return {}
|
||||
chal = e.headers.get("WWW-Authenticate", "") or ""
|
||||
except Exception: # noqa: BLE001
|
||||
return {}
|
||||
if not chal.lower().startswith("bearer"):
|
||||
return {}
|
||||
parts = dict(re.findall(r'(\w+)="([^"]*)"', chal))
|
||||
realm = parts.get("realm")
|
||||
if not realm:
|
||||
return {}
|
||||
q = {"service": parts.get("service", host), "scope": parts.get("scope", f"repository:{repo}:pull")}
|
||||
url = realm + ("&" if "?" in realm else "?") + urllib.parse.urlencode(q)
|
||||
try:
|
||||
tok = (_json(url) or {}).get("token") or (_json(url) or {}).get("access_token")
|
||||
return {"Authorization": f"Bearer {tok}"} if tok else {}
|
||||
except Exception: # noqa: BLE001
|
||||
return {}
|
||||
|
||||
|
||||
def _oci_tags(host: str, repo: str) -> list[str]:
|
||||
"""Tags from any OCI/v2 registry, with challenge-derived auth and Link pagination.
|
||||
|
||||
ghcr paginates hard — immich-server has >40,000 tags — and a truncated listing silently hides
|
||||
the newest release line, so follow the cursor and let the caller's integrity check catch a read
|
||||
that never reached the current pin."""
|
||||
hdrs = _oci_bearer(host, repo)
|
||||
tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
|
||||
for _ in range(MAX_TAG_PAGES):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
tags += (json.load(r) or {}).get("tags") or []
|
||||
link = r.headers.get("Link", "") or ""
|
||||
m = re.search(r'<([^>]+)>;\s*rel="next"', link)
|
||||
if not m:
|
||||
break
|
||||
nxt = m.group(1)
|
||||
url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
|
||||
return tags
|
||||
|
||||
|
||||
def _gh_token() -> str | None:
|
||||
tok = os.environ.get("GITHUB_TOKEN")
|
||||
if tok:
|
||||
return tok.strip()
|
||||
try:
|
||||
return open(os.environ.get("GITHUB_TOKEN_FILE", "/srv/cc-ci/.github-token")).read().strip() or None
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def github_release_tags(owner: str, repo: str, max_pages: int = 4) -> list[str]:
|
||||
"""Release tag names for a GitHub repo, newest first.
|
||||
|
||||
FALLBACK for registries whose tag listing cannot be enumerated. ghcr has no ordering and no
|
||||
server-side filter, and immich-machine-learning carries >40,000 tags — a full read is impractical
|
||||
and a partial read silently hides the newest release line. The project's RELEASES are ordered,
|
||||
small, and authoritative: container tags track them. (The GitHub Packages API would answer this
|
||||
directly but needs a scoped token; this scan's token deliberately has none.)
|
||||
"""
|
||||
hdrs = {"Accept": "application/vnd.github+json"}
|
||||
tok = _gh_token()
|
||||
if tok:
|
||||
hdrs["Authorization"] = f"Bearer {tok}"
|
||||
out = []
|
||||
for page in range(1, max_pages + 1):
|
||||
try:
|
||||
rows = _json(f"https://api.github.com/repos/{owner}/{repo}/releases"
|
||||
f"?per_page=100&page={page}", hdrs)
|
||||
except Exception: # noqa: BLE001
|
||||
break
|
||||
if not rows:
|
||||
break
|
||||
out += [r.get("tag_name") or "" for r in rows]
|
||||
return [t for t in out if t]
|
||||
|
||||
|
||||
def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
|
||||
"""Candidate GitHub repos whose releases track this image's tags."""
|
||||
if "ghcr.io" not in registry:
|
||||
return []
|
||||
parts = repo.split("/")
|
||||
if len(parts) < 2:
|
||||
return []
|
||||
owner, name = parts[0], parts[-1]
|
||||
cands = [(owner, name)]
|
||||
# ghcr.io/immich-app/immich-machine-learning is built from immich-app/immich.
|
||||
if name.startswith(owner.split("-")[0]):
|
||||
cands.append((owner, owner.split("-")[0]))
|
||||
return cands
|
||||
|
||||
|
||||
_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
|
||||
|
||||
|
||||
def list_tags(registry: str, repo: str) -> tuple[list[str], str | None]:
|
||||
if (registry, repo) in _TAG_CACHE:
|
||||
return _TAG_CACHE[(registry, repo)]
|
||||
res = _list_tags_uncached(registry, repo)
|
||||
_TAG_CACHE[(registry, repo)] = res
|
||||
return res
|
||||
|
||||
|
||||
def _list_tags_uncached(registry: str, repo: str) -> tuple[list[str], str | None]:
|
||||
try:
|
||||
return (_hub_tags(repo) if registry in ("docker.io", "registry-1.docker.io")
|
||||
else _oci_tags(registry, repo)), None
|
||||
except urllib.error.HTTPError as e:
|
||||
return [], f"HTTP {e.code}"
|
||||
except Exception as e: # noqa: BLE001
|
||||
return [], f"{type(e).__name__}: {e}"
|
||||
|
||||
|
||||
def resolve(ref: str) -> dict:
|
||||
"""Current pin -> newest same-shape tag, and newest within the current major."""
|
||||
if "${" in ref or "$(" in ref:
|
||||
# The tag is a compose variable (ghost pins `ghost:${IMAGE_VERSION}-alpine`). Its real value
|
||||
# lives in .env, not here. Report it as skipped, never as a failed lookup.
|
||||
return {**parse_ref(ref), "ref": ref, "shape": None, "candidates": 0,
|
||||
"newest_same_shape": None, "newest_within_major": None,
|
||||
"upgrade_available": False,
|
||||
"status": "skipped: templated ref (tag comes from a compose variable)"}
|
||||
info = parse_ref(ref)
|
||||
out = {**info, "ref": ref, "shape": shape(info["tag"]), "status": "ok",
|
||||
"newest_same_shape": None, "newest_within_major": None, "candidates": 0,
|
||||
"upgrade_available": False}
|
||||
tags, err = list_tags(info["registry"], info["repo"])
|
||||
if err:
|
||||
out["status"] = f"error: {err}"
|
||||
return out
|
||||
out["tags_seen"] = len(set(tags))
|
||||
# INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
|
||||
# the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
|
||||
# of thousands of tags and a truncated read silently hides whole release lines. immich's
|
||||
# machine-learning image is pinned v3.1.0, which EXISTS, yet a short read reported v1.134.0 as
|
||||
# newest; without this check that becomes a confident, wrong answer.
|
||||
if info["tag"] not in set(tags):
|
||||
# Docker Hub: the window is recency-ordered, so the pin being outside it just means the pin
|
||||
# is old — which is fine, because anything NEWER is necessarily inside the window. Confirm
|
||||
# the pin genuinely exists (so a typo is still caught) and carry on.
|
||||
if info["registry"] in ("docker.io", "registry-1.docker.io") and _hub_tag_exists(info["repo"], info["tag"]):
|
||||
out["source"] = f"docker-hub:recent-{HUB_RECENT_PAGES * 100}"
|
||||
tags = list(tags) + [info["tag"]]
|
||||
else:
|
||||
for owner, name in _release_fallback_repos(info["registry"], info["repo"]):
|
||||
rel = github_release_tags(owner, name)
|
||||
if info["tag"] in rel:
|
||||
tags = rel
|
||||
out["source"] = f"github-releases:{owner}/{name}"
|
||||
out["tags_seen"] = len(set(rel))
|
||||
break
|
||||
else:
|
||||
out["status"] = ("error: tag listing incomplete — the current pin "
|
||||
f"{info['tag']!r} is absent from {len(set(tags))} registry tags "
|
||||
f"and from the project's GitHub releases")
|
||||
return out
|
||||
want, cur = out["shape"], vkey(info["tag"])
|
||||
same = [t for t in set(tags) if shape(t) == want and vkey(t)]
|
||||
out["candidates"] = len(same)
|
||||
if not same:
|
||||
# Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings.
|
||||
out["status"] = "no comparable tags (shape has no numeric siblings)"
|
||||
return out
|
||||
newest = max(same, key=vkey)
|
||||
out["newest_same_shape"] = newest
|
||||
if cur:
|
||||
within = [t for t in same if vkey(t)[:1] == cur[:1]]
|
||||
if within:
|
||||
out["newest_within_major"] = max(within, key=vkey)
|
||||
out["upgrade_available"] = bool(cur and vkey(newest) > cur)
|
||||
return out
|
||||
|
||||
|
||||
def compose_images_ssh(recipe: str, host: str, recipe_dir: str) -> list[str]:
|
||||
"""Same as compose_images, but the recipe tree lives on another host (cc-ci has no python3)."""
|
||||
# NB: no shell-quoting of the directory — it may legitimately start with ~ or $HOME, and
|
||||
# quoting it stops the remote shell expanding it, which yields an empty (and silent) result.
|
||||
d = f"{recipe_dir}/{shlex.quote(recipe)}".replace("~", "$HOME")
|
||||
cmd = (f'for f in {d}/compose*.yml; do case "$f" in *compose.ccci.yml) continue;; esac; '
|
||||
f'[ -f "$f" ] && {{ cat "$f"; echo; }}; done; exit 0')
|
||||
out = subprocess.run(["ssh", host, cmd], capture_output=True, text=True, timeout=120)
|
||||
if out.returncode != 0:
|
||||
raise RuntimeError(f"ssh {host}: {(out.stderr.strip() or 'no output')[:200]}")
|
||||
if not out.stdout.strip():
|
||||
raise RuntimeError(f"ssh {host}: no compose files found under {d}")
|
||||
refs = []
|
||||
for m in IMAGE_RE.finditer(out.stdout):
|
||||
if m.group(1) not in refs:
|
||||
refs.append(m.group(1))
|
||||
return refs
|
||||
|
||||
|
||||
def compose_images(recipe: str, recipe_dir: str) -> list[str]:
|
||||
"""Every `image:` ref in the recipe's own compose files (the cc-ci overlay is NOT the recipe)."""
|
||||
refs, base = [], os.path.join(recipe_dir, recipe)
|
||||
for path in sorted(glob.glob(os.path.join(base, "compose*.yml"))):
|
||||
if os.path.basename(path) == "compose.ccci.yml":
|
||||
continue
|
||||
try:
|
||||
for m in IMAGE_RE.finditer(open(path).read()):
|
||||
if m.group(1) not in refs:
|
||||
refs.append(m.group(1))
|
||||
except OSError:
|
||||
continue
|
||||
return refs
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("recipe")
|
||||
ap.add_argument("--ssh", default=None, metavar="HOST",
|
||||
help="read the recipe's compose files from HOST over ssh (e.g. --ssh cc-ci); "
|
||||
"resolving still happens locally")
|
||||
ap.add_argument("--recipe-dir", default=RECIPE_DIR)
|
||||
ap.add_argument("--table", action="store_true", help="human-readable table instead of JSON")
|
||||
ap.add_argument("--only", default=None, help="resolve just the images whose ref contains this")
|
||||
a = ap.parse_args()
|
||||
|
||||
rdir = a.recipe_dir if a.recipe_dir != RECIPE_DIR or not a.ssh else "~/.abra/recipes"
|
||||
refs = (compose_images_ssh(a.recipe, a.ssh, rdir) if a.ssh
|
||||
else compose_images(a.recipe, a.recipe_dir))
|
||||
if a.only:
|
||||
refs = [r for r in refs if a.only in r]
|
||||
results = [resolve(r) for r in refs]
|
||||
report = {
|
||||
"recipe": a.recipe,
|
||||
"images": results,
|
||||
"upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
|
||||
"unresolved": [r["ref"] for r in results if r["status"].startswith("error")],
|
||||
# The whole point: distinguish "checked, current" from "could not check".
|
||||
"all_resolved": not any(r["status"].startswith("error") for r in results),
|
||||
}
|
||||
if not a.table:
|
||||
print(json.dumps(report, indent=2))
|
||||
return 0
|
||||
print(f"{a.recipe} — {len(results)} images")
|
||||
for r in results:
|
||||
flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current")
|
||||
print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else ""))
|
||||
print(f" shape={r['shape']} candidates={r['candidates']}"
|
||||
f" newest_same_shape={r['newest_same_shape']}"
|
||||
f" newest_within_major={r['newest_within_major']}")
|
||||
if r["status"] != "ok":
|
||||
print(f" status: {r['status']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -503,6 +503,114 @@ class TestReleaseNoteResolution(unittest.TestCase):
|
||||
self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {})
|
||||
|
||||
|
||||
class TestReleaseLineSemantics(unittest.TestCase):
|
||||
"""A fix inside the numeric window is not a fix on the branch you actually land on."""
|
||||
|
||||
def test_fix_later_on_the_targets_own_line_is_not_counted(self):
|
||||
# ClickHouse fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20. Landing on 23.10.4.25 crosses
|
||||
# the 23.9 fix numerically but is BELOW its own line's fix, so it does not have it.
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
||||
[adv("CVE-2023-48704", patched="v23.10.5.20; v23.9.6.20; v23.8.8.20")])],
|
||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], [])
|
||||
|
||||
def test_fix_earlier_on_the_targets_own_line_is_counted(self):
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
||||
[adv("CVE-2023-47118", patched="v23.10.2.13; v23.8.6.16")])],
|
||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-47118"])
|
||||
|
||||
def test_fix_exactly_at_the_target_is_counted(self):
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse",
|
||||
[adv("CVE-2023-48298", patched="v23.10.4.25; v23.9.5.29")])],
|
||||
v_from="23.4.2.11", v_to="23.10.4.25",
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-48298"])
|
||||
|
||||
def test_no_fix_on_the_target_line_falls_back_to_the_window(self):
|
||||
# redis fixes 7.4.6/8.0.4/8.2.2 with no 8.10.x entry; landing on 8.10 still has them,
|
||||
# because nothing on the 8.10 line is named as a LATER fix.
|
||||
rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2")])],
|
||||
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
|
||||
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"])
|
||||
|
||||
|
||||
class TestComposeDerivedWindows(unittest.TestCase):
|
||||
"""Windows read off a compose diff, so nobody has to remember which --image args an upgrade needs."""
|
||||
|
||||
OLD = """
|
||||
services:
|
||||
app:
|
||||
image: "plausible/analytics:v2.0.0"
|
||||
db:
|
||||
image: pgautoupgrade/pgautoupgrade:18-alpine
|
||||
plausible_events_db:
|
||||
image: clickhouse/clickhouse-server:23.4.2.11-alpine
|
||||
volumes:
|
||||
data:
|
||||
"""
|
||||
NEW = """
|
||||
services:
|
||||
app:
|
||||
image: "ghcr.io/plausible/community-edition:v3.2.1"
|
||||
db:
|
||||
image: pgautoupgrade/pgautoupgrade:18-alpine
|
||||
plausible_events_db:
|
||||
image: clickhouse/clickhouse-server:24.12-alpine
|
||||
volumes:
|
||||
data:
|
||||
"""
|
||||
|
||||
def _windows(self, old=None, new=None):
|
||||
pages = {"to": new if new is not None else self.NEW,
|
||||
"from": old if old is not None else self.OLD}
|
||||
with unittest.mock.patch.object(A, "_fetch", lambda u, h=None: pages["to" if "to" in u else "from"]), \
|
||||
unittest.mock.patch.object(A, "_gitea_auth", lambda u: {}):
|
||||
return A.windows_from_compose("http://x/to", "http://x/from")[0]
|
||||
|
||||
def test_app_service_leads_and_sidecars_follow(self):
|
||||
w = self._windows()
|
||||
self.assertEqual(w[0], ("community-edition", "v2.0.0", "v3.2.1"))
|
||||
self.assertIn(("clickhouse-server", "23.4.2.11-alpine", "24.12-alpine"), w)
|
||||
|
||||
def test_unchanged_images_are_not_windows(self):
|
||||
# pgautoupgrade is identical in both; inventing a window for it would be a false count.
|
||||
self.assertNotIn("pgautoupgrade", [n for n, _, _ in self._windows()])
|
||||
|
||||
def test_a_changed_image_REPO_is_still_the_same_service(self):
|
||||
# plausible/analytics -> ghcr.io/plausible/community-edition. Keyed by image repo this reads
|
||||
# as one image vanishing and another appearing, and the app window is lost entirely.
|
||||
w = self._windows()
|
||||
self.assertTrue(any(n == "community-edition" and f == "v2.0.0" for n, f, _ in w))
|
||||
|
||||
def test_no_change_yields_no_windows(self):
|
||||
self.assertEqual(self._windows(old=self.NEW, new=self.NEW), [])
|
||||
|
||||
def test_templated_tags_are_skipped(self):
|
||||
new = self.NEW.replace('ghcr.io/plausible/community-edition:v3.2.1', 'ghost:${IMAGE_VERSION}')
|
||||
self.assertNotIn("ghost", [n for n, _, _ in self._windows(new=new)])
|
||||
|
||||
|
||||
class TestImageNameMatching(unittest.TestCase):
|
||||
"""An image name and its advisory source rarely spell each other exactly."""
|
||||
|
||||
def test_matches_when_the_image_name_is_LONGER_than_the_source(self):
|
||||
# clickhouse/clickhouse-server vs source ClickHouse/ClickHouse — one-directional matching
|
||||
# dropped this window silently when the key came from a compose file.
|
||||
rep = run_scan([gh("ClickHouse/ClickHouse", [adv("CVE-1", patched="23.10.2.13")])],
|
||||
images=[("clickhouse-server", "23.4.2.11", "24.12")],
|
||||
urls=["https://github.com/ClickHouse/ClickHouse"])
|
||||
self.assertIn("github-advisories:ClickHouse/ClickHouse", rep["windows"])
|
||||
self.assertEqual(rep["cve_count_fixed"], 1)
|
||||
|
||||
def test_matches_when_the_image_name_is_SHORTER_than_the_source(self):
|
||||
rep = run_scan([gh("redis/redis", [adv("CVE-2", patched="7.4.1")])],
|
||||
images=[("redis", "7.4", "8.10")], urls=["https://github.com/redis/redis"])
|
||||
self.assertEqual(rep["cve_count_fixed"], 1)
|
||||
|
||||
|
||||
class TestAdjudicationEvidenceAssembly(unittest.TestCase):
|
||||
"""Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""
|
||||
|
||||
@@ -624,16 +732,22 @@ class TestHistoricReportNumbers(unittest.TestCase):
|
||||
for cve in delta:
|
||||
self.assertIn("redis", both["cves"][cve]["sources"][0])
|
||||
|
||||
def test_mailu_scan_finds_zero_and_says_so_knowingly(self):
|
||||
# Published as 2 via the UNION with release-note reading; the scan's own contribution is 0,
|
||||
# and 0 here must mean "checked, none", not "could not check".
|
||||
def test_mailu_finds_the_roundcube_pair_without_an_agent(self):
|
||||
# Published as 2 on 2026-08-07, but only because an agent read the release notes; the scan
|
||||
# itself contributed 0. It now reaches 2 deterministically: the CVEs appear only on
|
||||
# github.com/Mailu/Mailu/releases, and release 2024.06.56 (inside the window) names them.
|
||||
rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")])
|
||||
self.assertEqual(rep["cve_count_fixed"], 0)
|
||||
self.assertTrue(rep["count_known"])
|
||||
self.assertEqual(rep["cve_count_fixed"], 2)
|
||||
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-54432", "CVE-2026-54433"})
|
||||
# The redis bump fixes nothing new — every advisory it crosses was fixed at or before 8.6.3.
|
||||
self.assertEqual(rep["cve_count_indeterminate"], 0)
|
||||
|
||||
def test_keycloak_26_7_0_to_26_7_1_is_7(self):
|
||||
def test_keycloak_26_7_0_to_26_7_1_is_12(self):
|
||||
# Was 7 while only the GHSA feed was consulted. keycloak lists five more CVEs in the 26.7.1
|
||||
# release notes' fixed-issues section that it never filed as advisories — the gitea pattern.
|
||||
rep = self._count("keycloak", "26.7.0", "26.7.1")
|
||||
self.assertEqual(rep["cve_count_fixed"], 7)
|
||||
self.assertEqual(rep["cve_count_fixed"], 12)
|
||||
self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 5)
|
||||
|
||||
|
||||
def _main():
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# Upstream sources — mumble
|
||||
|
||||
| service | image | source repo | releases / changelog |
|
||||
|---------|-------|-------------|----------------------|
|
||||
| app | mumblevoip/mumble-server | https://github.com/mumble-voip/mumble | https://github.com/mumble-voip/mumble/releases |
|
||||
| web | rankenstein/mumble-web | https://github.com/rankenstein/mumble-web | https://github.com/rankenstein/mumble-web/releases |
|
||||
|
||||
## Standing notes
|
||||
- This file was **missing entirely** until 2026-08-11. Without it the advisory scan had no source to
|
||||
query, and still printed "0 identified by the deterministic scan" — which was then published as a
|
||||
clean `0` in the 2026-08-11 CVE check. The scan now refuses to emit a count when it has no usable
|
||||
source (it reports UNKNOWN), and `audit-sources.py` flags a missing registry file directly.
|
||||
- `mumblevoip/mumble-server` tracks the upstream server releases and DOES publish GitHub security
|
||||
advisories, so it is the recipe's primary CVE source.
|
||||
- `rankenstein/mumble-web` is a **fork** of the original `Johni0702/mumble-web`, which has been
|
||||
dormant since 2023-05. The fork itself last pushed 2023-07 and its Docker tag `0.5` was last built
|
||||
well over five years ago. Neither is archived, but treat the web client as effectively unmaintained:
|
||||
if a CVE lands there, expect no upstream fix and plan a replacement rather than an upgrade.
|
||||
- The server image tag is `v<version>-<build>` (e.g. `v1.6.870-4`); the trailing number is the image
|
||||
build, not an app version, and moves independently of upstream releases — `abra recipe upgrade`
|
||||
reports "no new versions" for it, so use `resolve-images.py` to see those bumps.
|
||||
Reference in New Issue
Block a user