Compare commits

...
Author SHA1 Message Date
autonomic-bot ab5d11ea2a Merge pull request 'advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?'' (#11) from feat/nvd-cpe-source into main 2026-08-11 22:17:13 +00:00
autonomic-bot 74117c2260 cve-check: record the remedy for a blind recipe, not just the symptom
The skill said to render a sourceless recipe as '?'. It now says how to stop it
being sourceless: declare an NVD CPE in the registry. That is what took the fleet
from two blind recipes to zero, and it is the first thing to try when the sweep
flags another.
2026-08-11 22:16:52 +00:00
autonomic-bot 985dc06e47 advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?'
Two recipes could not see CVEs at all. mattermost-lts has an empty GitHub advisory
feed and renders its security bulletins client-side, so a text sweep finds nothing;
mumble publishes nothing anywhere the registry points. Both returned '?' - nothing
measured - which is honest but useless.

NVD is CPE-indexed and carries structured version ranges, so it answers where the
vendor does not. Declared per recipe as 'nvd-cpe: <image> = <cpe:2.3:...>'.

  mattermost-lts 10.5.0  -> 10.12.4   165 CVEs
  mattermost-lts 10.11.22 -> 10.12.4    0 CVEs  (measured, not unknown)
  mumble         1.3.0   -> 1.6.870      2 CVEs

Both NVD range forms are used: versionEndExcluding is a patched version;
versionEndIncluding means the fix version is unpublished but the upgrade delivers
it whenever it crosses X.

That 0 for the actual mattermost upgrade is the interesting one, and it needed a
new rule to be correct: a fix on the line you upgrade FROM was already yours.
mattermost patches every maintained line at once, so 10.11.22 -> 10.12.4 crosses
10.12.1 while 10.11.22 already had the 10.11.4 backport. Without the rule the scan
claimed 12 CVEs the upgrade did not deliver.

The rule is skipped for placeholders: '7.4.X' parses to a bare 7.4 and would read
as 'already fixed at 7.4', which silently dropped redis CVE-2024-46981 and took
discourse 140 -> 139 before I caught it.

79 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 / plausible 6 unchanged.
Fleet sweep: 0 recipes with no usable CVE source, down from 2.
2026-08-11 22:16:37 +00:00
autonomic-bot 4b9978ac02 Merge pull request 'audit-sources --security-sources: find the recipes that cannot see CVEs at all' (#10) from feat/audit-security-sources into main 2026-08-11 20:02:24 +00:00
autonomic-bot 46ace30b4d audit-sources --security-sources: find the recipes that cannot see CVEs at all
Follow-up to the nginx blind spot. Sweeping all 22 recipes for sources whose CVEs
are USABLE (structured advisory feed, or a changelog attributable to releases)
rather than merely visible.

Before the changelog-attribution fix: 20 unusable sources. After: 5, and all five
are redundant - the same project also publishes an advisory feed (redis, gitea,
minio, clickhouse), so nothing is actually lost.

One real find, same shape as nginx: ONLYOFFICE/DocumentServer publishes NO GitHub
advisories, and the registry pointed its CHANGELOG.md at the GitHub *blob* page -
636KB of markup in which the release headings do not survive HTML-stripping, so 24
CVEs were visible and NONE attributable. The raw URL attributes all 24. Rather than
fix one registry line, advisory-scan now normalises github.com/../blob/.. to
raw.githubusercontent.com, which fixes every entry present and future.
lasuite-drive bumps documentserver, so this was live.

Genuinely blind after all that: mattermost-lts and mumble - no advisory feed, no
attributable changelog, no CVE data anywhere the registry points. mattermost is the
notable one: its bulletins are client-side rendered, so a regex sweep sees nothing.
Their scans can report 0 while nothing was measured, so /cve-check now renders those
recipes as ? and says why.

The audit output distinguishes a blind RECIPE from an unparseable PAGE, because
conflating them made 5 harmless redundancies look like 5 gaps.
2026-08-11 20:02:06 +00:00
autonomic-bot dab3edf3c2 Merge pull request 'advisory-scan: attribute vendor-changelog CVEs to the release that fixed them' (#9) from feat/changelog-version-attribution into main 2026-08-11 19:50:00 +00:00
autonomic-bot db37f1618b advisory-scan: attribute vendor-changelog CVEs to the release that fixed them
nginx publishes NO GitHub security advisories. Every nginx CVE we can see comes
from nginx.org/en/CHANGES, and the scan scraped ids out of it without attributing
them to a release - so they had no patched version, could never be classified, and
every nginx bump in the fleet reported 0 CVEs. nginx is a sidecar in most recipes,
so this was a fleet-wide blind spot.

Measured on the two PRs that prompted the question:
  lasuite-docs#7  nginx 1.31.1 -> 1.31.3   0 -> 6 CVEs
  lasuite-drive#6 nginx 1.31.2 -> 1.31.3   0 -> 3 CVEs
matching a hand count of the changelog exactly (three fixed in 1.31.2, three in
1.31.3; the narrower window correctly counts only the latter).

How: when a vendor page is organised by release, each CVE is attributed to the
nearest preceding release heading ('Changes with nginx 1.31.3', '## v1.31.3'),
and that becomes its fixed-in version. The CVE is tied to a window by the image
name appearing in the page URL (window 'nginx' <-> nginx.org/...). A changelog
lists the project's whole history, so only releases the window actually crosses
count - asserted by a test that the 2013 entries stay out.

76 tests. discourse 140 / gitea 2 / mailu 2 / keycloak 12 unchanged.
2026-08-11 19:49:43 +00:00
autonomic-bot 4bad1ea6db Merge pull request 'advisory-scan: derive the scan windows from a compose diff (--compose-to)' (#8) from feat/advisory-scan-compose into main 2026-08-11 19:24:46 +00:00
autonomic-bot ef58e33102 advisory-scan: derive windows from a compose diff (--compose-to)
Typing --from/--to/--image by hand means someone has to remember the recipe also
bumped its redis. That is how sidecar CVEs went uncounted for months. Point this
at a PR's compose.yml and it reads the windows off the diff instead.

  advisory-scan.py plausible --compose-to <.../branch/<pr>/compose.yml>
    -> community-edition: v2.0.0 -> v3.2.1
    -> clickhouse-server: 23.4.2.11-alpine -> 24.12-alpine
    -> 6 CVEs, identical to the hand-specified args

Details that mattered:

- keyed by SERVICE, not image repo. plausible moved plausible/analytics ->
  ghcr.io/plausible/community-edition; keyed by repo that reads as one image
  vanishing and an unrelated one appearing, and the app window - the one carrying
  the critical - is lost entirely.
- the baseline is the repo's DEFAULT BRANCH resolved from the API, never assumed
  to be main, because several recipes keep a stale main beside a live master.
- image names are matched against advisory sources BOTH ways: an image name is
  often longer than its source repo (clickhouse/clickhouse-server vs
  ClickHouse/ClickHouse) and sometimes shorter (redis vs redis/redis). One
  direction silently dropped the clickhouse window.
- credentials go in an Authorization header, never the URL: in-URL creds leak
  into shell history and process lists, and urllib mis-parses a password
  containing a colon.

--from/--to/--image remain for finer-grained checks (scanning a window that is
not a literal compose diff). 71 tests; discourse 140 / gitea 2 / mailu 2
unchanged.
2026-08-11 19:24:24 +00:00
autonomic-bot c352ea9058 Merge pull request 'add /cc-ci-cleanup — reconcile, close dead PRs, report what blocks the rest' (#7) from skill/cc-ci-cleanup into main 2026-08-11 19:17:19 +00:00
autonomic-bot fae2fbe21b add /cc-ci-cleanup — reconcile, close dead PRs, report what actually blocks the rest
25 open recipe PRs had accumulated, and the list had stopped being readable: CI
sweep artifacts that were never meant to merge sat next to genuine CVE fixes, and
three PRs the operator had been told were outstanding were in fact already merged
upstream (discourse #6 with 140 CVEs, keycloak #6 with 12, n8n #5) — visible only
once the mirrors were reconciled.

The skill: reconcile every mirror from true upstream FIRST (that step alone closed
those three), survey every open PR deterministically, close the ones that cannot
merge or were never meant to, and report prioritised action items — CVE-carrying
first — for the ones that should land. It never merges a recipe PR.

pr-survey.py gathers the facts and decides nothing: behind_main, mergeable,
diff_files, which images the PR adds vs which are already pinned in main, the
newest !testme verdict, branch kind, age.

One correctness detail worth the extra state: a FAILED diff fetch is reported as
unknown, never as an empty diff. gitea #4 reads that way (force-pushed branch)
while being a verified green fix, and 'empty diff' is a close signal — so the
tool says DIFF-UNREADABLE(do not close on this) instead.
2026-08-11 19:17:00 +00:00
autonomic-bot 6ebc35bc18 Merge pull request 'AGENTS.md: ship infra work as PRs, self-merge, operator reviews retrospectively' (#6) from policy/pr-then-merge into main 2026-08-11 19:08:55 +00:00
autonomic-bot 96c536f543 AGENTS.md: ship infra work as PRs, self-merge, operator reviews retrospectively
Operator policy (2026-08-11). For cc-ci-orchestrator and cc-ci: branch, open a PR
whose description is written to be read AFTER the fact, merge it yourself once
verified, and let the operator review retrospectively. The PR is not a gate — it
is how the work stays legible — so a description that says 'fix scanner' has
failed at its only job.

Explicitly does NOT extend to recipe repos: those are created and verified but
never agent-merged, because they change what deploys on other people's
infrastructure.

Also records what to do when work has already landed on main without a PR: pin a
branch at the pre-work commit and PR against that, rather than rewriting
published history.
2026-08-11 19:08:39 +00:00
autonomic-bot 94ce5c4de2 cc-ci-status: correct the ENOSPC note — inode recreation was coincidence
I recorded that recreating the runs directory with a fresh inode preceded
recovery. It recurred afterwards (build 1252), so that was not the fix.

The real signal is that it is INTERMITTENT and tracks concurrent activity: every
failure landed while a second run or manual deploy was in flight, and every build
on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a
failing build. Practical guidance is therefore to wait for the host to go quiet
and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is
the obvious knob to test if it becomes disruptive.

Root cause still not established, and the note now says so plainly rather than
presenting a coincidence as understood.
2026-08-11 18:54:58 +00:00
autonomic-bot bb7ebb4a27 reconcile-upstream.sh: one deterministic entry point, mandated before PR work
Working against a stale mirror has cost us three different ways:

- mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while
  upstream had already merged AND released it (3.1.3+2024.06.57). The work was
  done; only our mirror was behind. Reconciling closed the PR automatically.
- a stale mirror makes a survey report 'no upgrades available', so the recipe
  silently drops out of the weekly run.
- reading the wrong branch: several coopcloud recipes keep a stale 'main' beside
  the real default 'master'. gitea's main is 1.24.2-rootless while master has
  1.27.1-rootless and the merged PRs, so reading main manufactures a false
  'three releases behind, missing two CVSS-9.8 RCEs' finding.

The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and
already resolves the default branch itself. What was missing was a single obvious
entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or
--all, and is idempotent — recipe work lives in branches, never on mirror main, so
force-syncing main discards nothing.

/ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now
require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now
point at the shared script.
2026-08-11 18:38:09 +00:00
autonomic-bot ecf126d98d cc-ci-status: record the ENOSPC-with-free-disk failure and what recovered it
Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.

Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.
2026-08-11 17:39:56 +00:00
autonomic-bot ab88e59c21 cc-ci: prune unused images in the sweep; catch a starving host before CI dies
The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.

Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.

Two changes so it does not recur:

- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
  unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
  layer cache so runs stay fast. 'docker image prune -a' spares anything a container
  references, so infra and warm-* canonicals are safe. Volumes are still NOT
  pruned — warm-* canonical volumes are data-warm and legitimately dangling.

- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
  steady-state measure: the host was at 73% when runs started failing. It also now
  checks that recent builds actually produced results.json — an empty run dir is
  the fingerprint of a host problem masquerading as a recipe failure — and records
  how to read a drone step log out of its sqlite when the API token is unreachable.
2026-08-11 15:41:40 +00:00
autonomic-bot e89da2d842 audit-sources: check we are still looking where releases actually happen
A recipe tracks an image repo and a set of registry URLs. When upstream moves,
nothing errors — the old repo just stops receiving tags and the recipe looks
'up to date' forever. plausible is the case: it tracked plausible/analytics on
Docker Hub while upstream moved to ghcr.io/plausible/community-edition. Every
survey said 'no upgrades available' while v3 shipped elsewhere.

audit-sources.py reports the signals that catch it, per image and per registry
URL: image gone quiet (newest tag older than --quiet-days), deprecation wording
in the registry description, and GitHub repos that are archived, renamed or
gone. Signals, not verdicts — a stable image can be quiet for good reason — so
each finding says what was measured.

First run over 22 recipes, 11 findings, 4 alerts. It independently re-derived
the plausible case (analytics quiet 1126 days), and found:
  - drone: harness/drone now answers as harness/harness (the image is fine)
  - lasuite-docs, lasuite-drive: minio/minio is ARCHIVED on GitHub
  - lasuite-docs: docspecio/api is ARCHIVED
  - matrix-synapse: halfshot/matrix-appservice-discord image quiet 2078 days
  - mumble: NO cc-ci-plan/upstream/mumble.md at all

That last one exposed a scanner bug. With no registry file there is no source to
query, yet the scan still printed '0 identified by the deterministic scan' — and
that 0 was published as a clean count in the 2026-08-11 CVE check. A scan with no
usable source has measured nothing and must not report a number, least of all 0.
It now returns UNKNOWN and says the registry file is missing.

upstream/mumble.md added; mumble now scans 6 sources for a genuine 0.
2026-08-11 15:02:06 +00:00
autonomic-bot 6c91373357 skills: point every test-editing path at tests/STYLE.md
/recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author
test changes, and all three had only 'never weaken a test' as guidance. That did
not stop the plausible failure: the fixture INSERTed rows into the app's database,
which was correct for v2 and silently wrong for v3, where a site must belong to a
team. Events were acked 202 and discarded; the recipe sat RED for six weeks.

The rule that would have prevented it — set state up through the app's own
interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and
each of the three paths is told to read it before editing a test.
2026-08-11 14:41:10 +00:00
autonomic-bot 1db85a7e77 resolve-images: abra-independent version resolver; fix release-line over-count
immich pins two images with BOTH a tag and a digest, which makes abra FATA and
abandon the WHOLE recipe. It therefore contributed no version data at all and
silently dropped out of every survey — indistinguishable from 'up to date'. The
standing answer was prose in three skills telling an agent to check registries by
hand. This replaces it with a tool.

resolve-images.py reads the compose files and queries registries itself:
  - Docker Hub, ghcr, and any OCI registry via its own auth challenge (lscr.io
    and dock.mau.dev advertise different realms; assuming ghcr's shape 401'd).
  - tag SHAPES (digits -> '#') so -alpine stays on -alpine and 'latest' is never
    proposed as an upgrade.
  - reports newest_within_major AND newest_same_shape, and refuses to choose:
    immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
    build immich-server expects, so taking the newest breaks the deploy.
  - integrity check: if the CURRENT pin is absent from the listing, the listing
    was truncated and any 'newest' is a guess. ghcr caps out past 40k tags, so
    that falls back to the project's GitHub releases.
  - per-repo cache + backoff + Docker Hub auth: a fleet sweep re-reads nginx,
    redis and postgres many times and was getting 429s reported as 'unresolved'.

21/21 recipes now resolve. It found upgrades abra missed entirely in five:
mumble (abra said 'no new versions'; four patches behind), plausible's
clickhouse, lasuite-drive's collabora, gitea's mariadb, immich's postgres.
plausible's carried four CVEs, three high.

Also fixes a real over-count found while validating that: a fix inside the
numeric window is not a fix on the branch you land on. ClickHouse patched
CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20 — landing on 23.10.4.25 crosses the
23.9 fix but sits below its own line's, so it does NOT have it. A fix named on
the target's own line and above the target is now proof of absence.

70 tests (64 offline + 6 live). keycloak's live expectation moves 7 -> 12 and
mailu's 0 -> 2: both are the release-note source finding real fixes that were
never filed as advisories.
2026-08-11 04:56:39 +00:00
autonomic-bot 8df32edfcf cve-check: reconcile is mandatory; do not assume the default branch is main
Skipping the reconcile to keep the sweep 'read-only' was wrong. It researches a
stale checkout, and on the first real run left two recipes with no survey output
at all — indistinguishable from 'no upgrades available' unless you look. The
reconcile is safe precisely because recipe work lives in branches, not on main.

Also documents the trap that produced a false finding in that run: several
coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's
main is at 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs,
so reading main reports a recipe three releases behind and missing two CVSS-9.8
RCE fixes. Resolve default_branch from the API before reading any file.

And: no output is not 'no upgrade'. It is a third outcome, and only becomes '?'
after the direct registry check has also failed.
2026-08-11 04:32:49 +00:00
autonomic-bot 18caf047bf advisory-scan: two more cases decided in pass 1, found by the first real /cve-check
1. Release-note resolution now covers vendor pages on the same repo. It required
   a github-advisories: source, so mailu's Roundcube CVEs — announced only on
   github.com/Mailu/Mailu/releases — went to pass 2 even though the answer was
   sitting in the release notes. mailu now reports 2 deterministically, matching
   what previously took an agent reading the notes.

2. 'All known fix versions predate the version we were on' is now a DECISION,
   not an unknown. mailu's redis 8.8.0 -> 8.10.0 crosses 12 advisories all fixed
   by 8.6.3 or earlier; reporting them as 'could not judge' overstated the
   uncertainty. Recorded as outside-window with the naming tags as evidence.
   A fix landing ABOVE the window still stays indeterminate on purpose: that is
   an open vulnerability and must stay visible.

60 offline tests (was 58). discourse 140 / gitea 2 unchanged.
2026-08-11 04:24:15 +00:00
autonomic-bot b0bdce2c15 add /cve-check and /cve-check-and-upgrade
/cve-check answers 'what are we exposed to that an upgrade would fix?' without
running an upgrade: per-recipe, resolve the available window for EVERY image
(sidecars included), run the advisory scan over it, adjudicate whatever pass 1
could not decide, publish a report. Read-only — no PRs, no CI, no merges.

/cve-check-and-upgrade does that sweep, then runs /recipe-upgrade only on the
recipes whose upgrade actually closes a CVE, worst severity first, and reports
on both. --min-severity high for just the urgent ones; --dry-run prints the
queue and stops. Never merges.

Deliberate choices, each written into the skills:
- externals are SWEPT but never upgraded here — a security sweep that skipped
  deployed software would misreport exposure, but we don't maintain them.
- an unknown count never justifies an upgrade AND is never treated as clean;
  it goes to the Addendum.
- no upgrade available means 0 CVEs, not '?'.
- subagents are told which CVEs justify their upgrade, so the PR says why it
  exists — a PR naming the RCE it closes gets reviewed sooner.

recipe-report.py grows a page kind: 'cve' files as cve-DATE.html so a sweep
can't overwrite a weekly edition, while BOTH appear in the same archive index,
suffixed 'full report' / 'CVE check'.

/help and /cc-ci-status updated to route to them.
2026-08-11 04:04:59 +00:00
autonomic-bot 44cb9b6704 advisory-scan: tests, audit, and two real undercounts they found
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions
against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives
every count with a SEPARATE semver implementation and its own release fetch and
diffs against the scanner. Both found real defects:

1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE
   patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the
   norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also
   keeps the upper bound conservative (18.5 stays out of a window ending at 18).

2. Advisories with no knowable fix version were silently counted as 'not fixed'.
   Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an
   open-ended range — six of them high severity. They are now INDETERMINATE:
   not counted, not dismissed, and surfaced in the output.

   All twelve turned out to be genuinely fixed: redis names each in the release
   notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10,
   7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from
   release notes, with the naming tags recorded as the citation. discourse's
   redis contribution goes 5 -> 17, and its total 128 -> 140.

Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot
settle: it hands over each open case's full evidence, plus every verdict pass 1
reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that
evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both
gitea CVSS-9.8 RCEs — now reach it instead of being dropped.

Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is
tested there is deterministic: which cases it selects, and that truncation is
announced rather than silent.

SPEC.md rewritten around the two passes.
2026-08-11 01:29:11 +00:00
autonomic-bot 46c4fff1a6 advisory-scan: --image takes NAME=FROM:TO
Restores the single-value form (operator preference) under the --image name.
Repeat the flag per image, all in one call. Malformed values warn on stderr and
are skipped rather than aborting the scan, since it is an additive pre-step.

Counts unchanged: discourse 128 with redis / 123 without, gitea 2.
2026-08-11 00:52:40 +00:00
autonomic-bot 65bf3c095b advisory-scan: --image NAME FROM TO replaces --window KEY=FROM:TO
The image name was packed into the value, so the flag needed a hand-rolled
KEY=FROM:TO parser with its own malformed-input branch, and 'window' named the
wrong thing — the tool has two kinds of window (version ranges and, on the date
fallback, real date windows) and the flag meant only the first.

Now each part is its own argument: --image redis 7.4 8.10, repeatable, all in
one call. argparse enforces the arity, so the string parsing and its error path
are deleted. 'windows' survives internally as the computed-range concept.

Counts unchanged: discourse 128 with redis / 123 without, gitea 2.
2026-08-11 00:51:12 +00:00
autonomic-bot 8d7320f32e recipe-report: scheme changes and sidecars no longer mean '?'
The skill still told the reporter to publish '?' whenever the scan hit a
version-scheme change. The scan now resolves those by advisory publish date, so
that instruction would have re-introduced a '?' for a count it can determine.

Also documents that counts are a union across per-image windows, and that a
sidecar-sourced critical must name its image in the bulletin.
2026-08-10 22:06:38 +00:00
autonomic-bot b5f8543a9b advisory-scan: count sidecar CVEs via per-image windows
A recipe upgrades several images, each through its own version range. The scan
previously classified only the app repo, so sidecar bumps contributed nothing — the
alternative to the earlier bug where sidecars were judged by the APP's window and
produced a false 133.

Now: --window KEY=FROM:TO (repeatable) gives any other source its own range; each
window is classified independently (one may use patched-version ranges while another
falls back to advisory dates) and the count is the UNION. An image with no window is
still not counted — the scan will not guess a range it was not given. If ANY requested
window cannot be ordered, the total is UNKNOWN rather than a partial number.

/recipe-upgrade now instructs passing a --window per bumped sidecar.

Verified on discourse app 3.5.3->2026.7.1 + redis 7.4->8.10: 128 = 123 (app, by
publish date) + 5 (redis, by version range). The redis five are genuine for that bump
(patched 7.4.1 / 7.4.6 / 8.2.3) and include CVE-2025-49844, CRITICAL — previously
invisible. Regressions clean: gitea still 2, discourse without the sidecar window
still 123.
2026-08-10 22:01:28 +00:00
autonomic-bot 78ae2be8ae docs: spec for the advisory scanner's CVE detection
Step-by-step specification of cc-ci-plan/advisory-scan.py: inputs, the three source
classes and why each is ranked where it is, the union, both classification paths
(patched-version ranges, and the advisory-publish-date fallback for version-scheme
changes), the output contract, and how /recipe-report must read it.

Each rule records the production wrong answer that motivated it — the false 133 from
cross-image counting, the n8n misclassification from reading only vulnerabilities[0],
the '?' sprawl from url punctuation and benign-404s, and the 'never emit 0 for an
undetermined count' rule. Claims cross-checked against the implementation.
Keep this file in the same commit as any behaviour change.
2026-08-10 21:55:37 +00:00
autonomic-bot 98a624a13a advisory-scan: paginate, and count by advisory DATE when versions can't be ordered
Answers 'how can the weekly run produce counts like the hand count?' — by doing
exactly what the hand count did, deterministically. Two changes:

1. PAGINATION. The scanner requested per_page=100 and stopped. This endpoint caps at
   100 AND ignores ?page= (it re-returns the same rows — which is how a manual count
   first produced exact triplicates and a bogus 300). Busy projects were silently
   truncated: discourse has 286 advisories, so a single page could not see the window
   at all. Now follows the Link rel=next cursor to exhaustion.

2. DATE-BASED FALLBACK. Version strings cannot be ordered across a scheme change
   (discourse semver 3.5.3 -> calver 2026.7.1), which is why the scan first reported a
   false 133, then correctly refused. Release DATES always order. When the version path
   refuses, the scan now resolves both versions to their git tag dates on the primary
   repo and counts advisories PUBLISHED in that window, labelling the method in the
   output. The version path is still preferred when usable — it is exact rather than
   temporal.

Verified: discourse 3.5.3 -> 2026.7.1 now reports 123, matching the hand count
(1 critical, 16 high, 91 medium, 16 low; window 2025-12-30 -> 2026-07-31); gitea
1.27.0 -> 1.27.1 still reports 2 via the version path.
2026-08-10 21:02:39 +00:00
autonomic-bot fc36d0e10f advisory-scan: report UNKNOWN, never 0, when a count could not be determined
Operator: 'the scanner should not say 0 when it was not able to scan.' Correct — the
previous patch still led with '0 identified' and relegated the caveat to a footnote,
so the headline number was wrong even though the prose was right. A 0 in a security
column is an assertion of safety; it must never be emitted for an undetermined result.

Now: cve_count_fixed is null (not 0) in JSON, a count_known flag distinguishes
'counted zero' from 'could not count', and the markdown headline reads
'CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count' with an
explicit 'This is NOT zero' and instructions to render '?'.

Verified: discourse 3.5.3 -> 2026.7.1 (semver->calver) now reports UNKNOWN; gitea
1.27.0 -> 1.27.1 still reports 2.
2026-08-10 20:52:51 +00:00
autonomic-bot 1daf0fa616 advisory-scan: stop cross-image and cross-scheme miscounting (discourse's false 133)
Operator disbelieved discourse's '133 CVEs fixed' — correctly. Two defects made it
confidently wrong:

1. ONE WINDOW APPLIED TO EVERY IMAGE. The scan queries all source repos in the
   recipe's registry (app + redis/postgres/nginx sidecars) but judged them all with
   the APP's version window. 34 of the 133 were redis advisories, including
   CVE-2021-21309 — patched in redis 6.0.11 back in 2021 — scored as 'fixed by this
   upgrade' purely because 6.0.11 sits numerically inside discourse's 3.5.3 ->
   2026.7.1 range. Only the PRIMARY app repo is now classified; other sources are
   reported as unclassified so they stay visible without inflating the count.

2. VERSION-SCHEME CHANGES BREAK ORDERING. discourse moved semver -> calver
   (3.5.3 -> 2026.7.1), so 2025.12.2 compares 'newer' than 3.5.3 while shipping
   earlier. Numeric comparison cannot order that. The scan now detects a leading-
   component jump >= 100, refuses to classify, and says so in the block: the count
   is '0 by refusal, not by evidence — read the vendor's release notes'.

Refusing to answer beats answering wrongly: a fabricated 133 in a public security
report is worse than an explicit 'cannot determine'.

Verified after the fix: discourse 133 -> 0 (with the refusal caveat), gitea still
exactly 2 (both criticals, patched 1.27.1), keycloak 7 all genuinely from
keycloak/keycloak patched in 26.7.1, plausible 1. No other count changed.
2026-08-10 20:44:27 +00:00
autonomic-bot 318d09bdab advisory-scan: eliminate spurious '?' — url punctuation, stale URL, and ? semantics
The 2026-08-07 regeneration rendered '?' for 5 of 21 recipes. '?' is meant to be a
rare 'we tried and could not tell'; at that rate it is indistinguishable from noise
and hides the real unknowns. Three causes, none of them genuine uncertainty:

1. URL EXTRACTION BUG (mine). The registry is markdown, so urls appear inside
   `backticks` and 'quotes'. The extractor captured the trailing punctuation, so
   it fetched https://docs.n8n.io/release-notes/` and https://git.autonomic.zone'`
   — both 404 on the malformed url, both 200 when clean. Trailing markdown
   punctuation is now stripped. Fixed immich + n8n.
2. STALE REGISTRY URL. mattermost-lts pointed at
   docs.mattermost.com/about/mattermost-changelog.html, which 404s; the page moved
   to /deploy/. Corrected (same class as the pgautoupgrade fix).
3. WRONG SEMANTICS FOR 'NO UPGRADE'. lasuite-docs and custom-html-tiny were
   up-to-date this run, so no scan block existed and the report fell back to '?'.
   But a recipe with no upgrade has nothing an upgrade could have fixed — that is
   0, not unknown. The report skill now says so explicitly, restricts '?' to scans
   that RAN and reported genuinely failed sources, states that benign notes
   (no-advisories-published / template URL) never trigger '?', and instructs that
   many '?' is itself a bug to raise in the Addendum.

Result across all 16 scanned recipes of that run: 0 failed sources (was 5).
Counts also improved with the classifier fix: discourse 130->133, keycloak ->7.
2026-08-10 20:23:10 +00:00
autonomic-bot 3e59924450 advisory-scan: fix version classification (multi-line patches + range expressions)
Exposed by asking whether the scan catches the n8n CVEs (CVE-2026-42231/42232). It
did not — the advisories were fetched correctly but both misclassified as
out-of-window. Two bugs:

1. Only vulnerabilities[0] was read. An advisory carries ONE ENTRY PER PATCHED
   RELEASE LINE: n8n patches three (1.123.32, 2.17.4, 2.18.1), so whichever line
   the deployment is actually on was silently dropped. gitea passed only because it
   patches a single line. Now all entries are kept.
2. patched_versions is a RANGE EXPRESSION ('>= 2.18.1'), not a bare version. Naive
   parsing produced (18,1) instead of (2,18,1), so no comparison could ever match.
   Version tokens are now extracted with a regex and the advisory counts as
   fixed-by-this-upgrade if ANY patched line falls in (from, to].

Verified: n8n 2.17.0 -> 2.18.1 now reports 12 CVEs including both criticals
(CVE-2026-42231 GHSA-q5f4-99jv-pgg5, CVE-2026-42232); gitea 1.27.0 -> 1.27.1 still
reports exactly 2. Note our deployed n8n (2.27.2+) is already past all three patched
lines, so these were never outstanding for us — the bug was in detection, not
exposure.
2026-08-10 18:50:07 +00:00
autonomic-bot 3307bdb0fe advisory-scan: separate benign source absence from real failures; fix pgautoupgrade URLs
Two refinements found by running the scan across all 14 recipes of the 2026-08-07 run:

1. A repo with no advisory feed returns HTTP 404 on /security-advisories (e.g. the
   pgautoupgrade sidecar image). That is a BENIGN ABSENCE, not a failed check.
   Likewise registry entries that are TEMPLATE urls for humans
   (…/changelog/v<VERSION>/, …/<vX.Y.Z>/…) are documentation, not fetchable.
   Counting either as a failure pushed most recipes to '?', which would make the
   unknown-vs-clean distinction meaningless again — the exact signal the ? exists to
   preserve. Both are now recorded in sources_benign; only genuine errors (rate
   limit, network, 5xx, wrong URL) land in sources_failed.

2. upstream/*.md pointed at github.com/pgautoupgrade/pgautoupgrade, which 404s —
   the repo is pgautoupgrade/docker-pgautoupgrade. Corrected in n8n, lasuite-docs,
   lasuite-drive, lasuite-meet. A 404ing registry URL means we were not scanning a
   source we believed we were.

Effect on the 2026-08-07 data: recipes with genuine failed sources 5 -> 3 (the
remainder are really unreachable vendor pages). CVE counts unchanged where they
were already sound: discourse 130, gitea 2, plausible 1.
2026-08-10 18:45:45 +00:00
autonomic-bot cf26ef863a advisory-scan: read-only GitHub token from env or file (rate limit only)
Anonymous GitHub API is 60 req/hr — a full weekly sweep across ~20 recipes exhausts
it and the scan then reports sources as failed (visible, but degraded coverage). A
token lifts it to 5000/hr.

_github_token(): GITHUB_TOKEN env wins, else GITHUB_TOKEN_FILE (default
/srv/cc-ci/.github-token, 0600, gitignored). Reading PUBLIC advisories needs NO
scopes — a classic PAT with nothing ticked, or fine-grained limited to 'Public
repositories: read'. The tool only ever GETs advisories; do not grant write scopes.
A missing token is not an error: the scan runs anonymously and surfaces failures.

Also gitignores .github-token and .hcloud-token.
2026-08-10 18:38:45 +00:00
autonomic-bot 5775fe23f8 security: deterministic advisory scan as an ADDITIVE pre-step
Why: gitea 1.27.1 fixed CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8). The
2026-08-03 report printed gitea's CVE count as '1', the 2026-08-07 report as
'none'. Cause chain: the upgrade subagent read the GitHub release notes, which
name NEITHER cve (they are announced only in the vendor blog's security section),
so it recorded one unrelated minor item; the report then derived security content
from those notes plus model knowledge, and the model's training predates the CVEs.
Nothing in the pipeline ever queried an advisory source.

cc-ci-plan/advisory-scan.py — deterministic, per recipe, per upgrade window:
  1. GitHub Security Advisories API for every source repo in the upstream registry.
     PRIMARY: CVE + GHSA + severity + vulnerable/patched ranges, so 'fixed by THIS
     upgrade' is computed. Needs no new per-recipe config (134 registry URLs are
     already github.com).
  2. Vendor release/security pages — every registry URL, fetched + regex-scanned.
     This is the source that actually had the gitea CVEs.
  3. OSV where a package mapping exists — supplementary.
Each source reports its own status so 'checked, none found' is never confused with
'not checked'. Source selection was measured, not assumed: for these two CVEs OSV
404'd and NVD's API had them by neither CPE, id, nor keyword — advisory DBs lag the
vendor, hence 1+2 lead.

Wiring is strictly ADDITIVE:
- /recipe-upgrade gains step 2a: run the scan, paste the block into the per-recipe
  log, and report the UNION of it and the existing release-note reading. The scan
  may never lower a count established by reading.
- /recipe-report treats the block as a FURTHER source, prefers its advisory ids /
  severities / fixed-in versions for citation, and must render '?' (not 'none')
  when a scan is absent or has failed sources — the false-clean 'none' is exactly
  what happened on 2026-08-07.
- upstream/gitea.md records blog.gitea.com as the security-announcement URL.

Verified on the real regression: 1.27.0 -> 1.27.1 now yields exactly the 2 missed
criticals with their GHSA ids; the wider 1.26.2 -> 1.27.1 window yields 62.
2026-08-10 18:25:58 +00:00
autonomic-bot 02dbd71b49 recipe-upgrade: stop upgrade-PR branches drifting behind upstream
The extend path grafts HEAD^{tree} WHOLESALE onto the existing upgrade-* branch
(commit-tree -p <branch tip>). Reconcile force-syncs the MIRROR's main to upstream
but never brought the branch — or the local checkout — forward, so each week the
PR base drifted further back and upstream changes made since the branch was cut
were silently absent from the pushed tree. CI then verified a tree that would
never deploy.

Found on gitea PR #5 (2026-08-10): base 0ab323d predated upstream's
'BREAKING CHANGE: remove forgejo' (37ebd22), so the 1.27.1 bump fixing
CVE-2026-60004 + CVE-2026-59774 was !testme-GREEN against a forgejo-bearing tree.

Two changes:
1. Before pushing, if the local work does not contain the freshly-synced upstream
   main, merge upstream in — and FAIL LOUDLY (exit 1, naming the checkout) if that
   cannot auto-merge, rather than pushing a tree that omits upstream changes.
2. The extend commit now also parents on upstream main when the branch predated it,
   so the recorded history matches the pushed tree. Without it the merge-base stays
   stale and a later merge can REVERT upstream's changes. Still no force-push.

Verified against the real gitea drift: detection fires, merge is clean, resulting
tree keeps forgejo removed AND the 1.27.1 pin, history contains upstream.
2026-08-10 16:24:37 +00:00
autonomic-bot 8f85a238cc journal: 2026-08-07 run finished; supervisor-gate, report-pin, and subagent-model bugs fixed 2026-08-10 15:57:25 +00:00
autonomic-bot 80008da80d opencode: drop the misplaced project config
It lived in the orchestrator repo (the parent session's project) and so never
governed the task-tool subagents, which resolve their parent session's directory —
for launcher-started runs that is /srv/cc-ci-orch/cc-ci. The real config now lives
there (cc-ci repo, operator-approved). Extension was never the problem: .jsonc
parsed and resolved fine; the LOCATION was wrong.
2026-08-10 15:56:55 +00:00
autonomic-bot 242a6d9659 report: clear+re-pin the session id at launch (fixes un-watchdogged report runs)
Regression from the 2026-08-04 session-pinning work: lu._session_id() prefers the
pin file, but launch-report.py never cleared or re-established it. A surviving pin
from a PREVIOUS report run points at a session whose last message already carries
RECIPE REPORT COMPLETE, so the shared watchdog evaluates _completed()=True and
exits within one poll ('run completed — exiting'), leaving the CURRENT run
unwatched. Observed live: the 2026-08-07 finish-run's report step was watched by a
watchdog that quit after 3 minutes against an Aug-4 pin, then the report session
ended early with nothing to resume it.

start() now archives stale titles, clears the pin, snapshots existing ids, and
re-pins the new session after launch — the same contract launch-upgrader.start()
already follows. Scopes the shared helpers via UPGRADER_SESSION=<report session>.
2026-08-10 15:41:48 +00:00
autonomic-bot d441c6caaf supervisor: fix 3-day progress-gate deadlock (2026-08-07 run)
Root cause chain, all confirmed on the live system:
1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's
   kickoff PROMPT is an argv element, and the supervisor's prompt text contains the
   literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a
   live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>).
2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its
   process alive and SPINNING while emitting nothing (verified: 3 days, zero session
   output, still burning CPU). Progress now REQUIRES the session tree to have advanced
   within STALL_MIN; a live-but-idle proc is explicitly logged as stalled.
Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly
run sat unfinished and unreported.

Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may
resume when the wall lifts and their context is the run's state. New _billing_blocked()
detects the wall from the log tail and the gate surfaces
'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'.

Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while
still matching the real finisher; gate now reasons 'session advanced Nm ago'.
2026-08-10 15:17:02 +00:00
autonomic-bot d101147b93 upstream(plausible): note v3.2.1 ships with clickhouse 24.12-alpine 2026-08-07 05:14:27 +00:00
autonomic-bot fb1dc7af9c upstream(mattermost-lts): 2026-08-07 re-check (11.10.0 pre-release; ESR=11.7.8) 2026-08-07 04:47:40 +00:00
autonomic-bot 0347511a84 upstream(n8n): release-notes sources 2026-08-07 04:44:40 +00:00
autonomic-bot f8888b2082 upstream(lasuite-drive): fix collabora release-notes URL + note minio AIStor move 2026-08-07 04:21:19 +00:00
autonomic-bot a56734de0b upstream(immich): 2026-08-07 — upstream main at v3.1.0; runner clone-token stale 2026-08-07 03:51:02 +00:00
autonomic-bot 5424954b3f upstream(gitea): note 1.27.1 patch release 2026-08-07 03:15:22 +00:00
autonomic-bot 51b067770c upstream(discourse): re-confirm pg18 newest (2026-08-07) 2026-08-07 02:53:05 +00:00
autonomic-bot 34d62fa049 journal: three pending weekly-run PRs unblocked (keycloak/mailu re-verified, discourse basefloor fix merged) 2026-08-04 17:58:02 +00:00
autonomic-bot 995bcf82d7 journal: 2026-08-04 weekly-run completion + bridge/wordpress/watchdog/naming wrap-up 2026-08-04 17:04:08 +00:00
autonomic-bot 04a04e51ac session naming: archive- prefix convention + cc-ci-report unique-name invariant
- _archive_stale_titles default label is now 'archive-<title> —' (operator
  convention: all archived sessions start with archive- so they sort/filter
  together in the web UI).
- launch-report.py start() archives older cc-ci-report sessions before launch,
  same invariant as upgrader/supervisor (reuses the launch-upgrader helper).
- 33 sessions restyled/archived live; canonical names now unique:
  cc-ci-upgrader (idle finisher), cc-ci-report (generating), cc-ci-supervisor
  (none — only exists during a rescue).
2026-08-04 16:58:30 +00:00
autonomic-bot f750622e3d supervisor: unique cc-ci-supervisor web-UI name (same invariant as the upgrader)
_archive_stale_titles() generalized to (title, label); launch-supervisor's
spawn_supervisor() archives every older 'cc-ci-supervisor' session before
launching, so exactly one session carries the name. 11 historical supervisor
sessions archive-renamed live ('supervisor archive — <date> <time>').
2026-08-04 16:49:03 +00:00
autonomic-bot 0b6cc632d4 launch-upgrader: fix watchdog wrong-session resume + unique web-UI name invariant
Bug (2026-08-04 16:00): _session_id() sorted candidates on (s.time.created) which the
/session API rows DON'T carry — every key was 0, 'newest' degraded to server list
order, and the watchdog resumed the old giant unresumable session, kill_session()ing
the healthy fresh run mid-work.

Fixes:
- Pin the managed session id at launch/resume to LOG_DIR/.{SESSION}-session-id;
  _session_id() prefers the pin, validated via direct GET /session/<id> (the LIST is
  paginated ~100 rows, membership scans lie). Title lookup is only the fallback and
  now sorts on authoritative sqlite time_created.
- _archive_stale_titles() at start: every older top-level session titled
  cc-ci-upgrader is renamed 'upgrader archive — weekly <date>', so EXACTLY ONE
  session ever carries the canonical name in the opencode web UI (easy to find;
  finished runs stay browsable under archive names). 11 historical sessions
  renamed live today; the in-flight finisher pinned.
Verified live: _session_id() returns the pinned running session; tree-idle 0.0min
while subagents active. Full synthetic-stall watchdog confirmation queued post-run
(task #13).
2026-08-04 16:43:26 +00:00
autonomic-bot e8d7d09445 upstream(n8n): add 2.32.x / 2.33.x release-notes coverage
Covers the 2.32.4 -> 2.33.3 range for the 2026-08-03 /upgrade-all run:
2.33.0 minor features (admin-managed instance creds, workflow review
requests + publish/unpublish API, API deprecation of activate/deactivate
endpoints, optional N8N_SCHEDULER_MAX_ATTEMPTS env) and the 2.33.1-2.33.3
patches; notes 2.34.0 exists but is not this run's target.
2026-08-04 16:27:43 +00:00
autonomic-bot b2063b8235 upstream(mattermost-lts): 2026-08-04 re-check — 11.9.0 exists, operator-directed bump 2026-08-04 16:07:53 +00:00
autonomic-bot 0d37a891f7 opencode: subagents (agent.general) on deepseek-v4-pro via ZEN
Operator decision: main driving sessions stay glm-5.2; every task-tool subagent
runs opencode/deepseek-v4-pro (~3-5x cheaper, near-free cache hits) — the weekly
upgrade run burns most of its budget in subagents. Report generator stays glm-5.2
(launch-report.py default). Project-scoped config.
2026-08-04 15:54:55 +00:00
autonomic-bot cb20bea7cd recovery: give the incident tooling a permanent home (scripts/recovery/)
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
  power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
  name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
  session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
  screenshot / key / type subcommands; encodes the reset-invalidates-session and
  single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
  incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
  instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
2026-08-04 01:57:34 +00:00
autonomic-bot be7f8bc850 cctest: unify merge policy wording — recipe PRs are never agent-merged, both sides
Operator decision: no policy difference between cc-ci and recipe-maintainer. On
inspection ARM already agrees (recipe-upgrade-cron-all: 'PRs are reviewed and merged
manually by a human afterwards... never merges anything'; 'no human review in the
middle' = skip the mid-run plan confirmation only). Wrappers previously framed this
as a cc-ci override over ARM auto-merge flows — wrong reading; now stated as ONE
unified rule. /help conventions updated to match.
2026-08-04 01:40:36 +00:00
autonomic-bot 15e4e75681 cctest: consolidate onto the existing references/recipe-maintainer submodule
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
  ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
  (same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
  refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
  references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
  orchestrator-update, upgrade-run notes).
2026-08-04 01:37:32 +00:00
autonomic-bot 49854472b8 skills: vendor autonomic-recipe-maintainer + expose all its skills as /cctest-*
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
  (.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
  own description tagged [recipe-maintainer/cctest]; body points at the canonical
  submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
  (auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
  from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.

cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
2026-08-04 01:36:11 +00:00
autonomic-bot b462f1f7f1 skills: add /help — operator orientation over the skill roster
Enumerates the live skills directory at runtime (so new skills self-include), merges
with curated grouped descriptions (status / weekly maintenance / tests / cc-ci hosts /
enrollment / recovery), a what-do-you-want-to-do situation table, and the standing
conventions (PR-visibility+direct-merge, test-before-switch, never-weaken,
single-writer, swarm serialization). Read-only.
2026-08-04 01:22:18 +00:00
autonomic-bot 02cc2c29e2 upstream(lasuite-meet): release-notes sources 2026-08-04 01:21:43 +00:00
autonomic-bot 388e7f38c9 upstream(lasuite-docs): record v5.4.0 Bearer-auth removal + redis sidecar note 2026-08-04 00:13:50 +00:00
autonomic-bot 5ade783a50 skills: add /cc-ci-status — comprehensive read-only system status check
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
2026-08-03 23:12:49 +00:00
autonomic-bot 91179f872c skills: add /recipe-enroll — end-to-end enrollment of a new maintained recipe
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
2026-08-03 22:41:16 +00:00
autonomic-bot 6b3a3b1934 upstream(discourse): 2026.1→2026.7 ESR jump notes 2026-08-03 22:35:36 +00:00
autonomic-bot 74a57d37b3 recipes: enroll wordpress as weekly-maintained
used-recipes.md: wordpress weekly row. upstream/wordpress.md: registry entry
(wordpress official image + mariadb; install-wizard/XML-RPC test notes).
Mirror recipe-maintainers/wordpress created + synced (adcd0e9f). Test suite +
bridge enrollment: cc-ci PR #14 (verify + bridge deploy deferred until the
in-flight /upgrade-all completes).
2026-08-03 21:06:20 +00:00
autonomic-bot bab6481171 Merge pull request 'flake: bump nixpkgs (nixos-26.05, 2026-08-03)' (#1) from chore/orchestrator-flake-update-20260803 into main 2026-08-03 20:53:20 +00:00
109 changed files with 6825 additions and 106 deletions
+97
View File
@@ -0,0 +1,97 @@
---
name: cc-ci-cleanup
description: Tidy the fleet's open recipe PRs. Reconciles every mirror from TRUE upstream first (which alone closes PRs upstream already merged), then surveys every open PR deterministically, CLOSES the ones that can no longer be merged or were never meant to be (CI sweep artifacts, obsolete bumps, superseded duplicates) with a reason, and reports prioritised action items for the ones that SHOULD merge — what specifically is blocking each. NEVER merges a recipe PR. Invoke as /cc-ci-cleanup [recipe ...] [--dry-run].
---
# cc-ci-cleanup
Open recipe PRs accumulate and rot. Some were never meant to merge (CI sweep artifacts), some were
overtaken (upstream merged the same change, or a newer PR supersedes them), and some genuinely should
land but are quietly blocked. Left alone the list becomes noise, and a real CVE fix hides in it.
This skill separates those three, acts on the first two, and hands you a short list for the third.
**Boundaries.** It **CLOSES** irrelevant PRs and **NEVER MERGES** any recipe PR — those change what
deploys on other people's infrastructure, so a human merges them (see AGENTS.md). Closing is the only
write it performs, always with a comment saying why.
## Arguments
- `<recipe> …` — limit to these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
- `--dry-run` — classify and report, close nothing.
## Procedure
### 1. Reconcile every mirror from TRUE upstream — MANDATORY, FIRST
```
cc-ci-plan/reconcile-upstream.sh --all # or: reconcile-upstream.sh <recipe>...
```
**Do not skip this and do not reorder it.** Every signal in step 2 is measured against the mirror's
`main`; against a stale mirror they are all wrong. This step also does a chunk of the cleanup by
itself — it closes any PR whose changes upstream has already merged.
> On the first real run (2026-08-11) this alone closed **three** PRs that looked pending and were
> already merged upstream: discourse #6 (carrying **140 CVEs**), keycloak #6 (**12 CVEs**), n8n #5.
> All three had been reported to the operator as outstanding work. mailu #6 went the same way earlier
> the same day. Reconciling is not hygiene, it is how you avoid recommending work that is already done.
### 2. Survey every open PR (deterministic — no judgement yet)
```
python3 cc-ci-plan/pr-survey.py [recipe ...] # add --json for the raw facts
```
Per PR it measures: `behind_main`, `ahead`, `mergeable`, `diff_files`, the images it **adds**, which
of those are **already in main**, `obsolete`, the newest `!testme` verdict + build, `branch_kind`,
and age/idle days. It decides nothing — that is this skill's job.
### 3. Classify
**CLOSE — cannot merge, or was never meant to.** Each needs a *positive* reason, not an absence:
| signal | why it is closeable |
|---|---|
| `branch_kind: ci-artifact` (`ci/*`) | regall/cfold sweeps and `!testme` probes — harness artifacts, never intended to merge |
| `obsolete: true` | every image it adds is **already pinned in main** — it has nothing left to contribute |
| superseded | a newer PR on the same recipe makes the same bump (name both numbers in the comment) |
| `diff_files: 0` | genuinely empty diff — nothing to merge |
**NEVER close on:**
- `DIFF-UNREADABLE` — the diff could not be fetched, which is NOT an empty diff. gitea #4 reads that
way (force-pushed branch) while being a verified, green, needed fix.
- any field that came back `null`/unknown.
- a PR that carries a **CVE fix** and is the only thing carrying it, even if it looks stale — report it
instead. Losing a security fix to tidiness is far worse than a long PR list.
- `--dry-run`.
**NEEDS WORK — should merge, something blocks it.** Give the *specific* next action:
| signal | action item |
|---|---|
| `mergeable: false` | conflicts — rebase the branch on `main` and re-run `!testme` |
| `behind_main > 0` | out of date — rebase, then re-verify (a green from before main moved proves nothing) |
| `ci: failed` | diagnose via `/ci-test-review`; classify recipe-bug vs stale test |
| `ci: never-run` | run `!testme` |
| blocked on the operator | say exactly what is needed (a secret, an upstream release, a decision) |
**READY — green, current, no conflicts.** Action item is simply: review and merge.
### 4. Close the CLOSE set (skip entirely under `--dry-run`)
Comment first, then close. The comment must say **which signal** made it closeable and **what to do
if that is wrong** ("reopen if …"), so a wrong call is cheap to undo. Never close silently.
### 5. Report
Order by what deserves attention, not by recipe name:
1. **CVE-carrying PRs that should merge** — most severe first, with the CVE ids.
2. Other **READY** PRs (green + current).
3. **NEEDS WORK**, each with its one specific action.
4. **CLOSED this run**, with the reason for each.
5. Anything **deliberately left alone** despite looking stale, and why.
End with a one-line summary: `N open → C closed, R ready to merge, W need work`.
## Guardrails
- **Never merge a recipe PR.** Create/verify/close only; the operator merges.
- **Reconcile first, always.** Judging a PR against a stale mirror is how you close good work or
recommend work that is already done.
- **Close only on a positive signal**, never on "looks old". Age alone is not a reason — several
60-day-old PRs here are green and mergeable.
- **Never close a lone CVE fix.** Report it, however stale.
- Every close gets a comment with its reason and a reopen hint.
+14
View File
@@ -0,0 +1,14 @@
---
name: cc-ci-status
description: Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.
---
# cc-ci-status (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cc-ci-status/SKILL.md`**
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
thin pointer for Claude Code compatibility; opencode loads the canonical
definition from `.opencode/skills/` directly.
@@ -0,0 +1,13 @@
---
name: cctest-init-instance
description: "[recipe-maintainer/cctest] Deploy all maintained recipes to the active test instance from scratch (Wraps the autonomic-recipe-maintainer skill /init-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-init-instance.)"
---
# cctest-init-instance (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-init-instance/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/init-instance/SKILL.md`.
+13
View File
@@ -0,0 +1,13 @@
---
name: cctest-intro
description: "[recipe-maintainer/cctest] Explain what this project is and how to get started (Wraps the autonomic-recipe-maintainer skill /intro; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-intro.)"
---
# cctest-intro (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-intro/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/intro/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-new-recipe-guide
description: "[recipe-maintainer/cctest] Guide for developing a new Co-op Cloud recipe from scratch (Wraps the autonomic-recipe-maintainer skill /new-recipe-guide; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-new-recipe-guide.)"
---
# cctest-new-recipe-guide (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-new-recipe-guide/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/new-recipe-guide/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-opencode-sync
description: "[recipe-maintainer/cctest] Ensure every Claude skill has a corresponding OpenCode skill alias (Wraps the autonomic-recipe-maintainer skill /opencode-sync; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-opencode-sync.)"
---
# cctest-opencode-sync (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-opencode-sync/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/opencode-sync/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-check
description: "[recipe-maintainer/cctest] Fetch a Co-op Cloud recipe and check for available upgrades (Wraps the autonomic-recipe-maintainer skill /recipe-check; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-check.)"
---
# cctest-recipe-check (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-check/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-check/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-create-pr
description: "[recipe-maintainer/cctest] Push local recipe commits to git.autonomic.zone and open a PR against an upstream-synced main branch (Wraps the autonomic-recipe-maintainer skill /recipe-create-pr; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-create-pr.)"
---
# cctest-recipe-create-pr (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-create-pr/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-deploy
description: "[recipe-maintainer/cctest] Deploy the local recipe checkout to the test instance (Wraps the autonomic-recipe-maintainer skill /recipe-deploy; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-deploy.)"
---
# cctest-recipe-deploy (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-deploy/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-deploy/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-guidelines
description: "[recipe-maintainer/cctest] Guidelines for all recipe operations including local change preservation, version format, and secrets (Wraps the autonomic-recipe-maintainer skill /recipe-guidelines; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-guidelines.)"
---
# cctest-recipe-guidelines (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-guidelines/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-guidelines/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-init
description: "[recipe-maintainer/cctest] Create a new test instance and recipe-info for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-init; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-init.)"
---
# cctest-recipe-init (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-init/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-init/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-logging
description: "[recipe-maintainer/cctest] Logging instructions for maintaining detailed operation logs in the logs directory (Wraps the autonomic-recipe-maintainer skill /recipe-logging; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-logging.)"
---
# cctest-recipe-logging (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-logging/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-logging/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-new-tag
description: "[recipe-maintainer/cctest] Bump the recipe version and create an annotated git tag (Wraps the autonomic-recipe-maintainer skill /recipe-new-tag; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-new-tag.)"
---
# cctest-recipe-new-tag (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-new-tag/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-new-tag/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-overview
description: "[recipe-maintainer/cctest] Check all maintained recipes and recommend what to upgrade (Wraps the autonomic-recipe-maintainer skill /recipe-overview; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-overview.)"
---
# cctest-recipe-overview (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-overview/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-overview/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-review
description: "[recipe-maintainer/cctest] Review a recipe for Co-op Cloud best practices (Wraps the autonomic-recipe-maintainer skill /recipe-review; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-review.)"
---
# cctest-recipe-review (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-review/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-review/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-test-all
description: "[recipe-maintainer/cctest] Run tests for all maintained recipes, deploying each one at a time (Wraps the autonomic-recipe-maintainer skill /recipe-test-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-all.)"
---
# cctest-recipe-test-all (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-test-all/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-test-all/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-test-backup
description: "[recipe-maintainer/cctest] Test backing up and restoring a recipe's test instance (Wraps the autonomic-recipe-maintainer skill /recipe-test-backup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-backup.)"
---
# cctest-recipe-test-backup (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-test-backup/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-test-backup/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-test-new
description: "[recipe-maintainer/cctest] Test a recipe's first-time initialization from scratch (Wraps the autonomic-recipe-maintainer skill /recipe-test-new; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-new.)"
---
# cctest-recipe-test-new (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-test-new/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-test-new/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-test-update
description: "[recipe-maintainer/cctest] Test upgrading a recipe's test instance using abra app deploy (Wraps the autonomic-recipe-maintainer skill /recipe-test-update; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-update.)"
---
# cctest-recipe-test-update (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-test-update/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-test-update/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-test
description: "[recipe-maintainer/cctest] Run all tests for a Co-op Cloud recipe (Wraps the autonomic-recipe-maintainer skill /recipe-test; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test.)"
---
# cctest-recipe-test (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-test/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-test/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-upgrade-apply
description: "[recipe-maintainer/cctest] Execute a planned recipe upgrade — apply changes, deploy, test, commit/tag (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-apply; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-apply.)"
---
# cctest-recipe-upgrade-apply (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-upgrade-apply/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-apply/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-upgrade-cron-all
description: "[recipe-maintainer/cctest] Autonomous weekly upgrade run — overview all recipes, upgrade each end-to-end (sequentially by default, parallel with --parallel), open PRs (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-cron-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-cron-all.)"
---
# cctest-recipe-upgrade-cron-all (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-upgrade-cron-all/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-cron-all/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-upgrade-full
description: "[recipe-maintainer/cctest] Plan and apply a recipe upgrade end-to-end, no human review in the middle (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-full; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-full.)"
---
# cctest-recipe-upgrade-full (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-upgrade-full/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-full/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-upgrade-plan
description: "[recipe-maintainer/cctest] Create a detailed upgrade plan for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-plan; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-plan.)"
---
# cctest-recipe-upgrade-plan (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-upgrade-plan/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-upgrade-plan/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-recipe-upstream
description: "[recipe-maintainer/cctest] From a git.autonomic.zone review-PR URL, fetch the branch + tag locally and emit the commands to open the upstream PR on git.coopcloud.tech (Wraps the autonomic-recipe-maintainer skill /recipe-upstream; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upstream.)"
---
# cctest-recipe-upstream (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-recipe-upstream/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/recipe-upstream/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-setup-sandbox
description: "[recipe-maintainer/cctest] Guide for setting up a sandboxed environment to run the agent with recipe-maintainer (Wraps the autonomic-recipe-maintainer skill /setup-sandbox; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-setup-sandbox.)"
---
# cctest-setup-sandbox (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-setup-sandbox/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/setup-sandbox/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-switch-default-instance
description: "[recipe-maintainer/cctest] Switch the default test instance (b1cc or t1cc) for all recipe operations (Wraps the autonomic-recipe-maintainer skill /switch-default-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-switch-default-instance.)"
---
# cctest-switch-default-instance (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-switch-default-instance/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/switch-default-instance/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-sync-secrets
description: "[recipe-maintainer/cctest] Sync Docker secrets from the test server into recipe-info/testsecrets/ (Wraps the autonomic-recipe-maintainer skill /sync-secrets; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-sync-secrets.)"
---
# cctest-sync-secrets (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-sync-secrets/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/sync-secrets/SKILL.md`.
+13
View File
@@ -0,0 +1,13 @@
---
name: cctest-t1cc-start
description: "[recipe-maintainer/cctest] Provision the t1cc DigitalOcean test server and deploy Traefik (Wraps the autonomic-recipe-maintainer skill /t1cc-start; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-start.)"
---
# cctest-t1cc-start (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-t1cc-start/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/t1cc-start/SKILL.md`.
+13
View File
@@ -0,0 +1,13 @@
---
name: cctest-t1cc-stop
description: "[recipe-maintainer/cctest] Destroy the t1cc DigitalOcean test server via terraform (Wraps the autonomic-recipe-maintainer skill /t1cc-stop; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-stop.)"
---
# cctest-t1cc-stop (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-t1cc-stop/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/t1cc-stop/SKILL.md`.
@@ -0,0 +1,13 @@
---
name: cctest-test-context-reset
description: "[recipe-maintainer/cctest] Undeploy all apps from the test server except traefik (Wraps the autonomic-recipe-maintainer skill /test-context-reset; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-context-reset.)"
---
# cctest-test-context-reset (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-test-context-reset/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/test-context-reset/SKILL.md`.
+13
View File
@@ -0,0 +1,13 @@
---
name: cctest-test-setup
description: "[recipe-maintainer/cctest] Verify the test environment is configured correctly (Wraps the autonomic-recipe-maintainer skill /test-setup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-setup.)"
---
# cctest-test-setup (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/cctest-test-setup/SKILL.md`**
Read that file. It in turn wraps the ARM submodule skill
`references/recipe-maintainer/.opencode/skills/test-setup/SKILL.md`.
+17
View File
@@ -79,12 +79,29 @@ For each real (non-flaky) finding, write the actual fix and open a PR. **Never m
it handles the mirror to `git.autonomic.zone/recipe-maintainers/<recipe>` (upstream
`git.coopcloud.tech`). Keep the change **bounded** to the diagnosed root cause; don't rewrite the
recipe.
- **Before editing any test, read `tests/STYLE.md` in the cc-ci repo.** It encodes the rules a test
change must satisfy — set state up through the app's interface rather than its database, gate on
version instead of branching, correct the fixture/wait but NEVER the assertion, and diagnose from
the app's own telemetry before concluding a test is stale.
- **CI-server-side fix → cc-ci PR.** Branch the cc-ci product repo
(`recipe-maintainers/cc-ci`), apply the fix, and open the PR via the Gitea API (use the
`GITEA_*` creds from `/srv/cc-ci/.testenv`). **Single-writer discipline:** work on a dedicated
branch in a SEPARATE clone — **never push `main`, never touch the build loops' working clones**
(`/cc-ci`, `/cc-ci-adv`) or their in-flight state.
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
> ```
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
> ```
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
> two CVSS-9.8 RCEs" finding.
### 5. VERIFY each PR on the CI server (deterministic; still never merge)
A PR is only "working" once **cc-ci verifies it green** (operator rule) — dogfood the CI that found
the bug. Verification is deterministic (the harness), not an AI judgement.
@@ -0,0 +1,91 @@
---
name: cve-check-and-upgrade
description: Security-driven upgrade run. Does a full /cve-check sweep first (per-image advisory scan of every recipe's available upgrade, with adjudication), then runs /recipe-upgrade ONLY on the recipes whose upgrade fixes at least one CVE — worst severity first — opening a verified recipe PR for each, and finally publishes one report covering both the sweep and the PRs. Recipes with no CVEs are left alone; that is the point. NEVER merges. Invoke as /cve-check-and-upgrade [recipe ...] [--min-severity high] [--capacity N] [--dry-run].
---
# cve-check-and-upgrade
`/upgrade-all` upgrades everything that *has* an upgrade. **This upgrades what has a reason.** It runs
the `/cve-check` sweep, then spends CI time only on the recipes where an upgrade actually closes a
vulnerability, handling the worst first.
Use it when you want to act on security rather than churn the whole fleet: after a vendor announcement,
when CI capacity is short, or between weekly runs. When you only want to *know*, use `/cve-check`. When
you want everything current regardless of CVEs, use `/upgrade-all`.
**Creates PRs. Never merges.** Every PR is verified green on cc-ci and left for a human.
## Arguments
- `<recipe> …` — restrict the whole run to these recipes.
- `--min-severity critical|high|medium|low` — only upgrade recipes whose fixed CVEs reach this
severity. Default **`low`** (any CVE at all justifies the upgrade). `--min-severity high` is the
useful "just the urgent ones" setting.
- `--capacity N` — subagent pool size; defaults to the live `DRONE_RUNNER_CAPACITY` (the drone
runner's slots), matching `/upgrade-all`'s rolling-pool behaviour.
- `--dry-run` — do the whole sweep and print exactly which recipes *would* be upgraded and why, then
stop without spawning a single upgrade. **Publishes no report and opens no PR.**
## Procedure
### 1. Sweep — run `/cve-check` in full
Follow `.claude/skills/cve-check/SKILL.md` steps 15 exactly: candidate list, per-image upgrade windows,
the advisory scan per recipe, pass-2 adjudication of anything undecided, and the severity classification.
**Do not publish its report** — this run produces one combined report at the end instead.
Keep, per recipe: the windows scanned, the CVE count, the CVE ids with severities, and whether the count
is a floor (undetermined advisories remain) or unknown.
### 2. Decide what to upgrade
`RECIPES_TO_UPGRADE` = recipes where **the scan found ≥1 CVE** at or above `--min-severity`.
Deliberate exclusions, each recorded in the report with its reason:
- **0 CVEs** — an upgrade may exist, but nothing security-relevant. Left alone; that is the point of
this skill. `/upgrade-all` is what sweeps those up.
- **`external` tier** — swept for visibility, **never upgraded here**; someone else maintains it. Flag
it loudly in the report if it has a critical, since the action is to tell them, not to open a PR.
- **`UPTODATE`** — nothing available.
- **count `?` / UNKNOWN** — do **not** upgrade blind, and do **not** treat it as clean. Put it in the
Addendum as needing a look. An unknown is a gap in our knowledge, not evidence of safety.
Order the queue by **worst severity first** (critical → high → …), count breaking ties. If `--dry-run`,
print this queue with each recipe's CVE ids and severities, and STOP here.
### 3. Upgrade each one — via `/recipe-upgrade` subagents
Run `/recipe-upgrade <recipe>` per queued recipe as a **subagent**, in the queue order above, as a
**rolling pool** keeping `--capacity` (default `DRONE_RUNNER_CAPACITY`) running at once and starting the
next as each finishes — the same concurrency discipline as `/upgrade-all` §3, and safe for the same
reason (per-run recipe trees + app-domain locks).
Each subagent does the full job: plan, implement the bump, verify green on cc-ci with `!testme`, and
open a recipe PR. **Default mode — no `--with-tests`**: a genuinely stale test gets an explanatory PR
comment, not a test edit.
**Tell each subagent which CVEs justify its upgrade**, with ids and severities, so the PR description
says why it exists. That is most of this skill's value to a reviewer: a PR that names the CVSS-9.8 RCE
it closes gets merged today, an unexplained version bump waits a fortnight.
Collect per recipe: PR url + number, the `!testme` verdict and build number, and any failure.
### 4. Report — one page covering sweep AND PRs
Write `/tmp/cve-spec-<DATE>.json` per `/cve-check` step 6, with these differences:
- Rows for upgraded recipes carry the real `ci` (`build N ✓` / `RED N · <stage>`) + `ci_url`, and
`pr`/`pr_url`. `status` is the CI verdict (`GREEN`/`FAILED`/`STALE`); the live PR-status column
derives itself from `recipe` + `pr`.
- Rows for swept-but-not-upgraded recipes keep `PENDING`/`UPTODATE` with empty `ci`/`pr`, and a
`notes` reason (`0 CVEs — not upgraded`, `external — maintained elsewhere`, `below --min-severity`).
- Include `changes[]` — one entry per recipe that got a PR, describing what the upgrade changes **and
the CVEs it closes**.
- Keep `"kind": "cve"`: it titles the page "The Recipe Report — CVE check" and files it as
`cve-<DATE>.html`, alongside the weekly editions in the same archive index.
Then render + publish exactly as `/cve-check` step 7, and verify as its step 8. Print the report URL,
`N swept · M upgraded · K PRs green · J failed`, and `CVE CHECK AND UPGRADE COMPLETE`.
## Guardrails
- **NEVER merge.** Create and verify; a human merges. Never push to true upstream.
- **Never weaken a test** to make a PR green, and never edit a test without `--with-tests`.
- **Never upgrade a recipe whose CVE count is unknown** on the assumption it is fine — surface it.
- **Never upgrade an `external` recipe** here, even with a critical; report it instead.
- **Public-safe report only** — no secrets, tokens, internal hostnames, raw logs, or spend figures.
- If the sweep finds **nothing** at or above `--min-severity`, that is a good outcome: publish the
report saying so and open no PRs. Do not manufacture work.
+204
View File
@@ -0,0 +1,204 @@
---
name: cve-check
description: Fleet-wide CVE sweep WITHOUT upgrading anything. For every recipe cc-ci deploys, works out what upgrade is available (current pinned tag → newest supported tag, per image including sidecars), runs the deterministic advisory scan over that window, adjudicates whatever the scan could not decide, and publishes a CVE report to report.ci.commoninternet.net as cve-<DATE>.html. READ-ONLY — opens no PRs, edits no recipes, runs no CI, merges nothing. Answers "what are we exposed to that an upgrade would fix?" in minutes rather than the hours a full upgrade run takes. Invoke as /cve-check [recipe ...] [--weekly-only].
---
# cve-check
A **security sweep, not an upgrade run.** It answers one question for every recipe cc-ci deploys:
> If we upgraded this recipe today, how many CVEs would that fix, and how bad are they?
It is the cheap, safe half of `/upgrade-all`: the same version research and the same advisory scan,
with **no implementation, no CI, and no PRs**. Use it when you want the security picture now — after a
vendor announcement, before deciding what to prioritise, or between weekly runs. When you want the PRs
too, use **`/cve-check-and-upgrade`**.
**Read-only, absolutely.** Never edit a recipe, never open or comment on a PR, never merge, never
deploy. The only thing it writes is its own log and the published report page.
## Arguments
- `<recipe> …` — sweep only these recipes (else every recipe in `cc-ci-plan/used-recipes.md`).
- `--weekly-only` — skip rows tagged `external`. **Off by default on purpose**: an `external` recipe is
still deployed and still exposes us, so a security sweep that silently skipped it would misreport the
fleet's exposure. Externals are swept and clearly marked "maintained elsewhere" in the report.
## Procedure
> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
> `abra` needs a TTY. Wrap every abra call: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
> (`git` and other commands do NOT need the wrapper.)
### 1. Build the candidate list
Read `cc-ci-plan/used-recipes.md` — the canonical inventory. Take every row (both tiers), recording the
tier per recipe; with `--weekly-only`, drop the `external` rows. An explicit recipe argument overrides
any skip.
### 2. Per recipe — establish the upgrade window WITHOUT upgrading
This is `/recipe-upgrade` step 1's research, stopping before it implements anything.
> ⚠️ **The same four things that silently skip recipes apply here — handle ALL FOUR:**
> 1. **pseudo-TTY** — per the box above.
> 2. **go-git auth to git.autonomic.zone** — recipes on the private mirror FATA
> `authentication required: Unauthorized`. Bake creds into origin first (idempotent, only when
> origin is on git.autonomic.zone):
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`
> 3. **dirty worktree** — usually just the untracked cc-ci overlay; `git stash -u` before, `stash pop`
> after. Only a genuinely dirty TRACKED tree is a skip.
> 4. **tag+digest pins abra cannot parse** — abra FATAs and aborts the WHOLE recipe (immich). Do not
> hand-check the registry; run the resolver, which is abra-independent and covers every image:
> ```
> python3 /srv/cc-ci/cc-ci-plan/resolve-images.py <recipe> --ssh cc-ci --table
> ```
> It reports, per image, `newest_within_major` (the compatibility-safe pick) and
> `newest_same_shape` (the newest of that tag's form). **Use `newest_within_major` unless you have
> checked the app supports the major jump** — immich's postgres tag encodes the pg major plus the
> vectorchord/pgvectors versions immich-server is built against, so taking the newest would break
> the deploy. `all_resolved: false` means an image could NOT be resolved — that is a `?`, never a 0.
**Reconcile the mirror from true upstream FIRST — ALWAYS, no exceptions** — one command,
`cc-ci-plan/reconcile-upstream.sh <recipe>... | --all`. This is the same reconcile
`/upgrade-all` does. Do not skip it in the name of keeping the sweep read-only: skipping it makes you
research a stale checkout, and on the first real run that produced **two recipes with no survey output
at all**, which is indistinguishable from "no upgrades" unless you check. It is safe — recipe work
lives in **branches**, never directly on `main`, so a force-sync of `main` to upstream discards
nothing; it also auto-closes mirror PRs whose changes upstream has already merged.
> ### ⚠️ The default branch may be `master`, not `main` — check, do not assume
> Several coopcloud recipes keep a **stale `main` alongside the real default `master`**. gitea is one:
> `main` sits at 1.24.2-rootless while `master` has 1.27.1-rootless plus the merged PRs and the 3.6.3
> release. Reading `main` there tells you the recipe is three releases behind and missing two CVSS-9.8
> RCE fixes — a false alarm that reads exactly like a real one. Resolve the default branch from the
> API (`/api/v1/repos/coop-cloud/<recipe>` → `default_branch`) before reading any file, and never
> `git reset --hard origin/main` on a checkout that tracks `master`.
**Cross-check abra with the resolver.** abra is the primary source, but it silently contributes
nothing for images it cannot parse, and it reported "no new versions" for images that did have them
(mumble v1.6.870-0 → -4). Run `resolve-images.py` for every recipe and take the UNION of the two: on
the first real sweep the resolver found upgrades abra missed entirely in five recipes, one of which
(plausible's clickhouse) carried four CVEs.
Then read versions:
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
script -qec "abra recipe fetch $R --force -n" /dev/null; \
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
```
For each recipe produce **one window per image**: `current pinned tag → newest supported tag`. You need
the sidecars (redis, postgres, nginx …), not just the app — a sidecar bump is where discourse's only
CRITICAL came from, and an image with no window is not counted at all.
- **No upgrade available** → the recipe is `UPTODATE`; its CVE count is **`0`**, not `?`. There is
nothing an upgrade could fix. Record it and move on.
- **No output at all is NOT "no upgrade".** An abra call that times out, FATAs, or prints nothing
leaves the recipe **unverified** — treat it as a distinct outcome, never fold it into up-to-date.
Re-run it, and if it still yields nothing, resolve the versions by direct registry check (box item 4).
Only report `?` once BOTH the abra check and the direct check have failed. On the first real run this
distinction was the difference between two false zeros and the truth (both recipes turned out fine,
but nothing in the survey said so).
### 2c. Know which recipes CANNOT see CVEs at all
```
python3 cc-ci-plan/audit-sources.py --security-sources
```
A recipe whose sources yield **no CVE data at all** cannot produce a meaningful `0` — nothing was
measured, the same way a missing registry file cannot. Render those as **`?`**, not `0`.
**The fleet is currently at zero such recipes.** The last two — `mattermost-lts` (empty advisory
feed, client-side-rendered bulletins) and `mumble` (nothing published anywhere) — were fixed by
declaring an NVD CPE in their registry:
```
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
```
**If this sweep ever reports a blind recipe again, that is the fix**: find the product's CPE at
nvd.nist.gov and add the line. Prefer a real advisory feed or an attributable changelog when one
exists — NVD lags the vendor — but a lagging source beats no source, and it turns a `?` into a
number.
An *unparseable page* is NOT the same thing: it is harmless when the same project also publishes an
advisory feed (redis, gitea, minio, clickhouse all do). Only "no usable source for this image" counts.
### 3. Run the advisory scan over that window
```
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app> --to <new-app> \
[--image <name>=<old>:<new>]...
```
**One call per recipe with every image in it** — the count is a union across images, and the
UNKNOWN guarantee only holds when a single run sees them all. Paste the markdown block verbatim into
the per-recipe log at `/srv/cc-ci/.cc-ci-logs/cve-check/<DATE>/<recipe>.md`.
### 4. Adjudicate what the scan could not decide (pass 2)
If the block reports advisories it **could NOT judge**, or the count is **UNKNOWN**, re-run with
`--adjudicate` and decide each open case yourself:
```
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> … --adjudicate
```
Answer **FIXED / NOT-FIXED / STILL-UNKNOWN** per case, each with a one-line reason **citing the
evidence shown** — never from memory of the project, which is the exact failure that let two CVSS-9.8
gitea RCEs be published as "none". Every FIXED is added to the count; pass 1's number is a floor. The
block also lists what pass 1 already decided — if a verdict looks wrong given its evidence, say so.
Record your verdicts in the per-recipe log so the number is auditable.
### 5. Classify severity and priority
For each recipe collect the CVE ids with **severities** (the scan gives them, with GHSA ids). Sort the
report rows by what an operator should deal with first:
1. recipes with a **critical**, then **high**, then anything else with CVEs (more CVEs higher within a band);
2. then `?` (a count that could not be established — investigate, do not ignore);
3. then recipes with an upgrade available but **0** CVEs;
4. then `UPTODATE`.
Severity outranks raw count: 2 CVSS-9.8 RCEs matter more than 120 medium plugin advisories.
### 6. Write the report spec
`/tmp/cve-spec-<DATE>.json`, same shape as `/recipe-report` (see `recipe-report.py`'s header), with:
- **`"kind": "cve"`** — titles the page "The Recipe Report — CVE check" and files it as
`cve-<DATE>.html`. It appears in the SAME archive index as the weekly editions, suffixed
"— CVE check" so the two are told apart at a glance. Without this field you would overwrite that
date's weekly edition.
- `date`, `subtitle` "CVE check <human date>",
- `lead`**one short paragraph**: fleet exposure in a sentence and what to do first.
- `table[]` — every recipe swept. `recipe`; `change` = the window you scanned, e.g.
`1.27.0 → 1.27.1 · redis 7.4 → 8.10`; `status` = `UPTODATE` when nothing is available, else
`PENDING` (an upgrade exists and is not yet taken); **`cve`** = the count (integer, `?` only per the
rules below); `notes` = severity mix, whether the number is a floor, and `maintained elsewhere` for
`external` rows. **Leave `ci`/`pr` empty — nothing was built and no PR exists.**
- `addendum[]` — real anomalies only: registry URLs that failed, recipes whose window could not be
established, a scan whose count is a floor with many undetermined advisories.
- `security[]` — one entry per **critical/high** finding: recipe · CVE id(s) + severity · what it fixes
· **which image** it is in. Name the image: `CVE-2025-49844` is a redis flaw, and an operator reading
"discourse" needs to know that.
- `changes[]`**omit** (nothing changed; there are no PRs).
**`?` must stay RARE.** Use it only when a scan ran and reported genuinely failed sources, or the count
came back UNKNOWN and adjudication could not settle it. Never `none` for an unknown — a blank reads as
clean. A recipe with no upgrade available is `0`, not `?`. Many `?` is a bug for the Addendum.
### 7. Render and publish — via the script only
```
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py render /tmp/cve-spec-<DATE>.json /tmp/cve-<DATE>.html
python3 /srv/cc-ci/cc-ci-plan/recipe-report.py publish /tmp/cve-<DATE>.html <DATE> cve
```
All layout is owned by `recipe-report.py`. Never hand-write or post-process HTML; if `render` errors,
fix the spec JSON and re-render. **Public page — no secrets, tokens, internal hostnames, raw logs, or
any billing/spend figures.**
### 8. Verify and stop
`curl -fsS https://report.ci.commoninternet.net/cve-<DATE>.html` renders and the index lists it. Print
the URL, a one-line summary (`N recipes swept · M with CVEs · K critical`), and `CVE CHECK COMPLETE`,
then go idle. One-shot — do not loop, and do not start upgrading anything.
## Guardrails
- **Read-only.** No PRs, no edits, no merges, no deploys, no CI runs. If a recipe looks urgent, say so
in the report — do not act on it. `/cve-check-and-upgrade` is the skill that acts.
- **Never report `0` for something you could not scan.** `0` means checked-and-clean; unknown is `?`.
- A count with undetermined advisories is a **floor** — say so in the notes rather than rounding away.
- **Public-safe output only.**
+14
View File
@@ -0,0 +1,14 @@
---
name: help
description: Operator orientation - lists every skill available on this orchestrator with what it does and when to reach for it, grouped by purpose (status, weekly maintenance, host updates, tests, enrollment, recovery, reporting), plus a "what do you want to do?" guide for common situations. Read-only. Invoke as /help.
---
# help (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/help/SKILL.md`**
Read that file for the full content. This `.claude/skills/` copy is kept as a
thin pointer for Claude Code compatibility; opencode loads the canonical
definition from `.opencode/skills/` directly.
+21 -48
View File
@@ -57,60 +57,33 @@ tailscale ping -c 3 <host-alias>
If the host still does not come back, continue.
## 3. Request the Hetzner console
## 3. Use the repo recovery tools (permanent home — do NOT rebuild these in /tmp)
Request a remote console session:
The API and console tooling live in **`scripts/recovery/`** (see its README for the condensed
10-minute drill, proven 2026-08-03):
```bash
curl -s -X POST \
-H "Authorization: Bearer ${HCLOUD_TOKEN}" \
-H "Content-Type: application/json" \
"https://api.hetzner.cloud/v1/servers/<SERVER_ID>/actions/request_console"
# API: status / actions / reboot / reset / poweroff / poweron / rescue-on / rescue-off / console
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci status
python3 /srv/cc-ci-orch/scripts/recovery/hetzner.py cc-ci actions 10
# Shell-only console access (fresh console session + websocat bridge + vncdotool,
# venv auto-bootstrapped at ~/.cache/hetzner-console-venv):
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci screenshot /tmp/console.png
bash /srv/cc-ci-orch/scripts/recovery/hetzner-console.sh cc-ci key Down Down Return
```
The API returns:
Known server names: `cc-ci` (134485294), `orchestrator` (134487234). Token: `HCLOUD_TOKEN`
env or `/srv/cc-ci/.hcloud-token` (0600, not in git; prefer per-incident revocable tokens,
and never paste tokens into a chat transcript).
- `wss_url`
- `password`
If you have a browser, use the Hetzner console directly.
If you only have shell access, you can still drive it locally because the console is **raw VNC over
websocket**.
## 4. Shell-only console access (websocket VNC bridge)
Install temporary tools:
```bash
nix shell nixpkgs#websocat -c websocat --version
python3 -m venv /tmp/opencode/hetzner-console-venv
/tmp/opencode/hetzner-console-venv/bin/pip install --disable-pip-version-check pillow websocket-client vncdotool
```
Bridge the websocket console to a local VNC TCP port:
```bash
nohup nix shell nixpkgs#websocat -c \
websocat -b -E tcp-l:127.0.0.1:5905 '<WSS_URL>' \
>/tmp/opencode/hetzner-websockify.log 2>&1 &
```
Validate the RFB banner:
```bash
python3 - <<'PY'
import socket
s=socket.socket(); s.settimeout(5); s.connect(('127.0.0.1',5905))
print(repr(s.recv(32)))
PY
```
Expected:
```text
b'RFB 003.008\n'
```
Notes that used to cost time:
- Each console command requests a **fresh** console session — old sessions die on hard reset,
and the websocat bridge is single-connection anyway.
- A GRUB one-shot/default for a NixOS generation needs the **submenu id `1>N`** (top level:
0 = default entry, 1 = the "All configurations" submenu). A bare index silently falls back
to the default entry. Clear any grubenv override after the next `switch` regenerates
grub.cfg — indices shift.
Capture a screenshot from the console:
+14
View File
@@ -0,0 +1,14 @@
---
name: recipe-enroll
description: Add a NEW recipe to cc-ci's maintained set, end to end — create + sync the recipe-maintainers mirror from coopcloud upstream, author a real test suite (health floor + non-vacuous recipe-specific tests incl. a create-an-object round-trip), enroll it in the !testme bridge (POLL_REPOS) and the weekly /upgrade-all inventory (used-recipes.md weekly row + upstream registry), then VERIFY the whole thing: full harness suite green with the new tests, bridge deployed (test-before-switch) and healthy. Opens the cc-ci PR for visibility and merges it directly once verification is green (the skill invocation is the authorization); the report lists merged PR links + what changed. Invoke as /recipe-enroll <recipe>.
---
# recipe-enroll (thin wrapper)
The canonical definition of this skill lives in the **opencode** position:
**`.opencode/skills/recipe-enroll/SKILL.md`**
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
thin pointer for Claude Code compatibility; opencode loads the canonical
definition from `.opencode/skills/` directly.
+29
View File
@@ -37,6 +37,35 @@ keeps every weekly edition looking the same regardless of which model writes the
- **Security analysis.** Scan the per-recipe `upgrade_notes_md` + the summary (and use your own
knowledge of the version bumps) for upgrades that fix **CVEs / security issues**. For each recipe,
**count the CVEs** the PR fixes — this drives both the table's `cve` column and the priority sort.
- **ADDITIONALLY, and never instead:** each per-recipe log carries an `### Advisory scan
(deterministic pre-step)` block (from `cc-ci-plan/advisory-scan.py` — GitHub Security
Advisories + vendor security pages + OSV, with severities and fixed-in versions). Treat its
CVE list as a **further source** and report the **UNION** of it and what you found by reading.
Its entries are machine-derived with advisory IDs, so prefer them for CVE ids / severities /
fixed-in versions, and cite the GHSA where present in the Security Bulletin. If the block
lists **failed sources**, the count is **not** authoritative: render the cve cell as `?`
(unknown), never `none` — a blank that reads as "clean" is exactly how two CVSS-9.8 gitea
RCEs were reported as "none" on 2026-08-07.
- **`?` must stay RARE — it means "we tried and could not tell", not "we didn't look".** Use it
ONLY when a scan ran and reported genuinely failed sources, **or when the scan block says
COUNT UNKNOWN**. In that case the scan's `0` means *not determined*: publish `?` and say so in
the notes; publishing `0` would assert a clean bill of health nothing supports. Note a
**version-scheme change is no longer a reason for `?`** — the scan resolves semver→calver jumps
(discourse 3.5.3 → 2026.7.1) by falling back to advisory publish dates and reports a real number.
- **A count with undetermined advisories is a FLOOR.** If the scan block says N advisories
"could NOT be judged", report the number but say in the notes that it is a floor — those
advisories are neither fixed nor safe, they are unmeasured. Never round them away.
- **Counts span every image, each judged by its own window.** A scan block lists one line per
image with its version range and classification method; the headline is their union. So a
recipe's count legitimately includes **sidecar** CVEs (discourse's 128 = 123 app + 5 redis).
When a sidecar contributes a critical/high, name the image in the bulletin — CVE-2025-49844 is
a redis flaw, not a discourse one, and an operator reading "discourse" needs to know that.
(The scan headline itself now says `UNKNOWN` rather than a number in that case.)
In particular: a recipe with **no upgrade this run** (up-to-date/skipped) has nothing an
upgrade could have fixed — report `0`, not `?`. A recipe with a clean scan reports its number (including `0`). Benign notes in a scan
block (`no-advisories-published`, `skipped: template URL`) are NOT failures and must not
trigger `?`. If you find yourself rendering `?` for many recipes, that is a bug to report in
the Addendum, not a normal outcome.
Anything **critical/high** also gets a `security` bulletin entry (recipe · CVE id(s) + severity ·
what it fixes · PR link); be specific about severity and what's exposed if not merged.
- **Lead — ONE short paragraph.** A tight, concrete opener in opus's voice: fleet state in a sentence
+56
View File
@@ -157,6 +157,56 @@ On cc-ci's `~/.abra/recipes/<recipe>` (wrap every abra call per the pseudo-TTY b
`open-recipe-pr.sh`). Do **not** push to upstream; the version bump + tag + publish are the operator's
final `abra recipe release` step.
### 2a. Advisory scan (deterministic; ADDITIVE — run it, never skip it)
Run the deterministic scanner for the exact upgrade window and **paste its markdown block verbatim
into the per-recipe log**:
```
python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app-version> --to <new-app-version> \
[--image <name>=<old>:<new>]...
```
**Pass an `--image` for EVERY sidecar you upgraded** (redis, postgres, nginx …), not just the app —
each image is judged by its own versions, and an image you don't name is not counted at all. Repeat
the flag for each one and pass them **all in a single call** (the count is a union across images).
e.g. discourse moving app 3.5.3→2026.7.1 *and* redis 7.4→8.10:
```
... --from 3.5.3 --to 2026.7.1 --image redis=7.4:8.10
```
→ 140 CVEs (123 app + 17 redis), where the redis seventeen include a **critical** (CVE-2025-49844)
that is invisible if the sidecar is left out. `<name>` is substring-matched against source repo names,
so make it specific enough to hit exactly one.
**If the scan reports advisories it could NOT judge, re-run it with `--adjudicate`.** That is a second
pass: it collects each open case's full evidence (advisory prose, references, affected ranges, every
release naming the CVE) and asks YOU to decide FIXED / NOT-FIXED / STILL-UNKNOWN with a reason citing
that evidence. The deterministic number is a **floor** — add every FIXED to the count. Say
STILL-UNKNOWN rather than inferring from memory, and never record an undecided CVE as unaffected.
It also shows what pass 1 already decided; if a verdict looks wrong given its evidence, say so.
It queries, per recipe: the **GitHub Security Advisories API** for every source repo in
`cc-ci-plan/upstream/<recipe>.md` (CVE + GHSA + severity + vulnerable/patched ranges, so
"fixed by THIS upgrade" is computed, not guessed), every **vendor release/security URL** in that
registry (fetched + regex-scanned for CVE ids), and **OSV** where a package mapping exists.
**This does NOT replace your own release-note reading — it is an ADDITIONAL evidence source.** Do
exactly what you did before, then union the two: the CVE count you report is the union of the CVEs
you found in the notes and the CVEs the scan found. Never let the scan lower a count you established
by reading.
Why it exists: gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). Both are named
only in the vendor's blog security section — the GitHub *release notes* mention neither — so the
release-note read found one unrelated minor item and the weekly report printed a CVE count of "1",
then "none". Advisory databases lagged too (OSV 404'd on both; NVD's API had neither by CPE, id, or
keyword), which is why the GitHub advisory API and the vendor pages lead.
If the scanner reports **failed sources**, say so in the log — an incomplete scan must not read as
a clean one. If a vendor publishes security notes at a URL the registry lacks (gitea's
`blog.gitea.com`), **add it to `cc-ci-plan/upstream/<recipe>.md`** so the next scan sees it.
### 2b. Direct deploy + inspect on cc-ci — live feedback BEFORE CI (recipe-maintainer style)
Before opening the PR / running `!testme`, deploy the WIP recipe **directly** on the cc-ci server and
watch it converge — the way recipe-maintainer tests on `cctest`. This gives you **live logs +
@@ -262,6 +312,12 @@ test change, and a test change is **gated by `--with-tests`**:
Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
stale test; operator to decide).
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
0. **READ `tests/STYLE.md` in the cc-ci repo FIRST.** It is the rulebook for changing a test, and
it is written against the failures this pipeline has actually produced. The two that matter most
here: **set state up through the app's own interface, never its database** (a plausible fixture
that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks),
and **gate on version rather than writing a fixture that supports both** — old-version tests can
simply be deleted, since the older version is only exercised through the upgrade tier.
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
@@ -102,6 +102,27 @@ if [ "${MODE}" != "--reconcile-only" ]; then
DIVERGED=$(git log --oneline origin/main..HEAD 2>/dev/null || true)
[ -n "${DIVERGED}" ] || { echo "ERROR: HEAD has no commits beyond origin/main. Nothing to PR."; exit 1; }
LATEST_MSG=$(git log -1 --pretty=%s HEAD)
# --- Keep the LOCAL work current with the freshly-synced upstream main (anti-drift) ---
# The push path below grafts `HEAD^{tree}` WHOLESALE onto the PR branch. If this checkout is not
# based on the upstream main we just synced, every upstream change made since the branch was cut
# is silently ABSENT from the pushed tree — the PR (and the CI that verifies it) then describes a
# tree that will never deploy. Observed on gitea PR #5 (2026-08-10): its base predated upstream's
# "BREAKING CHANGE: remove forgejo", so `!testme` verified a forgejo-bearing tree while main had
# dropped it. Merge upstream in FIRST, and fail loudly rather than paper over a conflict.
if ! git merge-base --is-ancestor "${NEW_MAIN_SHA}" HEAD; then
echo "→ Local work predates upstream main (${NEW_MAIN_SHA:0:8}) — merging upstream in first..."
if ! GIT_AUTHOR_NAME="${GITEA_USERNAME}" GIT_AUTHOR_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
GIT_COMMITTER_NAME="${GITEA_USERNAME}" GIT_COMMITTER_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
git merge --no-edit "${NEW_MAIN_SHA}" >/dev/null 2>&1; then
git merge --abort 2>/dev/null || true
echo "ERROR: cannot auto-merge upstream main (${NEW_MAIN_SHA:0:8}) into the local ${RECIPE} work."
echo " Upstream changed files this upgrade also touches. Resolve by hand in"
echo " ${RECIPE_DIR}, then re-run. Refusing to push a tree that omits upstream changes."
exit 1
fi
echo " ✓ upstream merged into the local work"
fi
fi
# --- Reconcile open PRs against the freshly-synced upstream main ---
@@ -165,9 +186,19 @@ if git rev-parse --verify --quiet "refs/remotes/gitea/${BRANCH}" >/dev/null; the
if [ "$(git rev-parse 'HEAD^{tree}')" = "$(git rev-parse "${EXIST_TIP}^{tree}")" ]; then
echo "→ '${BRANCH}' already has this exact tree — nothing new to push (will still re-test)."
else
# Parent the new commit on the branch tip AND (when the branch predates it) on upstream main, so
# the recorded HISTORY matches the tree we are pushing. Without the second parent the merge-base
# stays stale: git would later treat upstream's post-branch changes as "removed by this PR" and a
# merge could revert them (the gitea #5 / forgejo-removal drift, 2026-08-10). No force-push: this
# is still a fast-forward from the branch tip.
EXTRA_PARENT=()
if ! git merge-base --is-ancestor "${NEW_MAIN_SHA}" "${EXIST_TIP}"; then
EXTRA_PARENT=(-p "${NEW_MAIN_SHA}")
echo " (also parenting on upstream main ${NEW_MAIN_SHA:0:8} — branch predated it)"
fi
ONTOP=$(GIT_AUTHOR_NAME="${GITEA_USERNAME}" GIT_AUTHOR_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
GIT_COMMITTER_NAME="${GITEA_USERNAME}" GIT_COMMITTER_EMAIL="${GITEA_USERNAME}@git.autonomic.zone" \
git commit-tree "$(git rev-parse 'HEAD^{tree}')" -p "${EXIST_TIP}" -m "${LATEST_MSG}")
git commit-tree "$(git rev-parse 'HEAD^{tree}')" -p "${EXIST_TIP}" "${EXTRA_PARENT[@]}" -m "${LATEST_MSG}")
echo "→ Adding the new work on top of '${BRANCH}' (fast-forward, no force-push)..."
git push gitea "${ONTOP}:refs/heads/${BRANCH}"
fi
@@ -73,6 +73,29 @@ done
# 5) Stray exited containers (debug one-shots) — best-effort prune.
docker container prune -f >/dev/null 2>&1 || true
# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
# "failure" and looked like recipe regressions.
#
# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
# `image prune -a` removes only images no container references, so anything deployed (infra +
# warm-* canonicals) is untouched; anything else is re-pulled on demand.
#
# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
# data-warm and their volumes are legitimately dangling between runs.
DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
else
echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
fi
if [ "$removed" -eq 0 ]; then
echo "== orphan sweep: clean (nothing to remove) =="
else
+4
View File
@@ -28,3 +28,7 @@ master-age.txt
# Python bytecode cache
__pycache__/
*.pyc
# Local API tokens — never committed (advisory-scan / hetzner recovery)
.github-token
.hcloud-token
+1 -1
View File
@@ -1,3 +1,3 @@
[submodule "references/recipe-maintainer"]
path = references/recipe-maintainer
url = https://git.autonomic.zone/recipe-maintainers/recipe-maintainer
url = ssh://git@git.autonomic.zone:2222/recipe-maintainers/autonomic-recipe-maintainer.git
+156
View File
@@ -0,0 +1,156 @@
---
name: cc-ci-status
description: Comprehensive read-only status check of the whole cc-ci system - how the recent weekly upgrade runs went and whether their reports published, which recipes/tests are stale, how long since the server + orchestrator host flake updates, open recipe PRs (flagging CVE-carrying PRs that have been open too long), host health (failed units, disk, timers, bridge/!testme path), ending with a verdict (ALL HEALTHY or a findings list) and recommended next steps mapped to the skills to invoke. Never changes anything - it only reads and reports. Invoke as /cc-ci-status.
---
# cc-ci-status
One comprehensive, **read-only** status pass over the cc-ci system. Output ends with either
**`ALL HEALTHY`** or a prioritized findings list, each finding paired with the skill to invoke.
Nothing here mutates state — no restarts, no deploys, no merges.
## Checks (run all; collect findings, don't stop at the first)
### 1. Weekly upgrade runs — recency + outcome
```
ls -t /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | head -3
head -20 <latest> # the Summary block: Considered/green/stale/Failed/Skipped
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
```
- **Overdue** if the newest report is >8 days old, or the timer is inactive/missing → recommend
`systemctl start cc-ci-upgrade-all.service` (or investigate the timer) / `/upgrade-all`.
- **Failed entries** in the latest report → recommend `/recipe-upgrade <recipe>` per entry (or
`/ci-test-review` if the failure is harness-side).
- A run currently in flight (tmux `cc-ci-upgrader` session live) is NOT a finding — report it as
in-progress and skip staleness checks that depend on its output.
### 2. Report publishing — report.ci.commoninternet.net
```
curl -s -o /dev/null -w '%{http_code}' https://report.ci.commoninternet.net/
ls -t /var/lib/cc-ci-reports/week-*.html | head -2 # via ssh cc-ci
```
- Index must be 200 and there must be a `week-*.html` at least as new as the last **completed**
upgrade run (a completed run without a matching page = report generation broke → recommend
`/recipe-report` / inspect `launch-report.py`).
### 3. Stale recipes / stale tests
- Latest report's "PRs where a test looks stale" section + carry-over notes.
- Any entries → recommend `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <recipe>
--with-tests` (single).
- Also check `/srv/cc-ci/.cc-ci-logs/tests-update-*.md` recency — if stale tests were reported
weeks ago and no tests-update run since, say so.
### 4. Open recipe PRs — especially CVE-carrying ones that linger
Enumerate open PRs across `recipe-maintainers/*` (Gitea API, creds in `/srv/cc-ci/.testenv`):
```
GET /repos/recipe-maintainers/<repo>/pulls?state=open # repos = the used-recipes.md inventory + cc-ci
```
For each open PR: age (now created_at), and whether the PR title/body/report row mentions
**CVE** / security patch.
- **CVE-carrying PR open >14 days** → HIGH-priority finding: name the PR, the CVE context, and
the blocker (commonly a stale test — check the report row) → recommend the unblocking skill
(`/cc-ci-tests-update`) plus "operator: review + merge <PR>".
- Non-CVE PRs open >30 days → low-priority note ("operator review backlog: N PRs").
- Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
as failures.
### 5. Server + orchestrator host update recency
For BOTH hosts:
```
# cc-ci server: ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
# orchestrator: nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
git ls-remote https://github.com/NixOS/nixpkgs <channel> # current tip
ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
```
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
`/cc-ci-orchestrator-update`.
### 6. Host + service health (both machines)
```
ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
```
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
HIGH: recommend `hetzner-server-recovery`.
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
> pruned them), the filesystem filled during a run, and the harness died at startup with
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
- **!testme actually produces results** (the check that would have caught the above days earlier):
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
it reads as a recipe regression rather than a sick host. Builds 12361242 all failed that way.
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
and the drone step log"*. The step log lives in drone's sqlite
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
(distroless container, swarm secret).
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
> inode (it recurred afterwards — that apparent fix was coincidence).
>
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
> if it becomes disruptive.
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
recommend refreshing the bridge secret + redeploy (test-before-switch).
### 7. Maintained-set consistency (quick)
- Every `weekly` row in `used-recipes.md` has `tests/<recipe>/` on cc-ci AND a
`recipe-maintainers/<recipe>` mirror AND is in bridge `POLL_REPOS`; mismatches → recommend
finishing enrollment (`/recipe-enroll <recipe>` covers all touchpoints).
## Output format
```
# cc-ci status — <date>
## Verdict: ALL HEALTHY | N findings (M high-priority)
## Weekly upgrades: <last run date + one-line outcome; next timer firing>
## Report site: <ok/broken + latest page>
## Stale tests: <none | list>
## Open PRs: <count; CVE-carrying + age flagged FIRST; ready-to-merge list>
## Host updates: server <rev, N days behind tip> · orchestrator <rev, N days>
## Health: server <failed/disk/services> · orchestrator <failed/disk/sessions> · bridge <ok/401s>
## Recommended next steps
1. <finding> → /<skill> (or operator action)
```
When a finding is that the fleet's **security exposure is unknown** — the last weekly run failed or
is stale, so nobody has scanned for CVEs recently — the recommended step is **`/cve-check`** (read-only,
minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recommend
**`/cve-check-and-upgrade`** (add `--min-severity high` when only the urgent ones matter). Prefer
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
producing results.json, disk under
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).
## Guardrails
- **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
- Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
reported once, at the root cause.
@@ -83,8 +83,29 @@ failure (AI — this is the `ci-test-review` step-3 diagnosis):
changed upstream, what the test currently asserts.
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
> ### ⚠️ RECONCILE FROM UPSTREAM FIRST — always, before any PR work or upgrade check
> ```
> cc-ci-plan/reconcile-upstream.sh <recipe>... # or --all
> ```
> Deterministic, idempotent, and safe (recipe work lives in branches, never on mirror `main`). It
> force-syncs each mirror to coopcloud's **default branch — resolved from the API, `main` OR
> `master`** — and closes any mirror PR whose changes upstream already merged. Skipping it has cost
> us three distinct ways: mailu #6 was reported as the fix for two internet-facing CVEs while
> upstream had already merged AND released it; a stale mirror makes a survey report "no upgrades
> available" so the recipe drops out of the weekly run; and reading the wrong branch on a recipe with
> a stale `main` beside a live `master` (gitea) manufactures a false "three releases behind, missing
> two CVSS-9.8 RCEs" finding.
### 2. For each stale test — author the minimal test update (AI; never weaken)
> **Read `tests/STYLE.md` in the cc-ci repo before writing the update.** It is the rulebook for test
> changes, written from failures this pipeline actually produced. Most load-bearing: set state up
> through the app's **own interface, never its database** (a plausible fixture that INSERTed rows
> passed on v2 and silently broke on v3 — 202 acks, rows in postgres, nothing ingested — and held the
> recipe RED for six weeks), **gate on version rather than supporting both** (old-version tests can be
> deleted; the older version is only exercised via the upgrade tier), and correct the fixture or the
> wait but **never the assertion**.
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
Swarm). For each `STALE_TESTS` entry:
@@ -0,0 +1,26 @@
---
name: cctest-init-instance
description: "[recipe-maintainer/cctest] Deploy all maintained recipes to the active test instance from scratch (Wraps the autonomic-recipe-maintainer skill /init-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-init-instance.)"
---
# cctest-init-instance (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/init-instance/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
+26
View File
@@ -0,0 +1,26 @@
---
name: cctest-intro
description: "[recipe-maintainer/cctest] Explain what this project is and how to get started (Wraps the autonomic-recipe-maintainer skill /intro; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-intro.)"
---
# cctest-intro (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/intro/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-new-recipe-guide
description: "[recipe-maintainer/cctest] Guide for developing a new Co-op Cloud recipe from scratch (Wraps the autonomic-recipe-maintainer skill /new-recipe-guide; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-new-recipe-guide.)"
---
# cctest-new-recipe-guide (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/new-recipe-guide/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-opencode-sync
description: "[recipe-maintainer/cctest] Ensure every Claude skill has a corresponding OpenCode skill alias (Wraps the autonomic-recipe-maintainer skill /opencode-sync; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-opencode-sync.)"
---
# cctest-opencode-sync (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/opencode-sync/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-check
description: "[recipe-maintainer/cctest] Fetch a Co-op Cloud recipe and check for available upgrades (Wraps the autonomic-recipe-maintainer skill /recipe-check; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-check.)"
---
# cctest-recipe-check (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-check/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-create-pr
description: "[recipe-maintainer/cctest] Push local recipe commits to git.autonomic.zone and open a PR against an upstream-synced main branch (Wraps the autonomic-recipe-maintainer skill /recipe-create-pr; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-create-pr.)"
---
# cctest-recipe-create-pr (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-create-pr/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-deploy
description: "[recipe-maintainer/cctest] Deploy the local recipe checkout to the test instance (Wraps the autonomic-recipe-maintainer skill /recipe-deploy; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-deploy.)"
---
# cctest-recipe-deploy (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-deploy/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-guidelines
description: "[recipe-maintainer/cctest] Guidelines for all recipe operations including local change preservation, version format, and secrets (Wraps the autonomic-recipe-maintainer skill /recipe-guidelines; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-guidelines.)"
---
# cctest-recipe-guidelines (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-guidelines/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-init
description: "[recipe-maintainer/cctest] Create a new test instance and recipe-info for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-init; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-init.)"
---
# cctest-recipe-init (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-init/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-logging
description: "[recipe-maintainer/cctest] Logging instructions for maintaining detailed operation logs in the logs directory (Wraps the autonomic-recipe-maintainer skill /recipe-logging; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-logging.)"
---
# cctest-recipe-logging (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-logging/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-new-tag
description: "[recipe-maintainer/cctest] Bump the recipe version and create an annotated git tag (Wraps the autonomic-recipe-maintainer skill /recipe-new-tag; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-new-tag.)"
---
# cctest-recipe-new-tag (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-new-tag/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-overview
description: "[recipe-maintainer/cctest] Check all maintained recipes and recommend what to upgrade (Wraps the autonomic-recipe-maintainer skill /recipe-overview; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-overview.)"
---
# cctest-recipe-overview (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-overview/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-review
description: "[recipe-maintainer/cctest] Review a recipe for Co-op Cloud best practices (Wraps the autonomic-recipe-maintainer skill /recipe-review; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-review.)"
---
# cctest-recipe-review (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-review/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-test-all
description: "[recipe-maintainer/cctest] Run tests for all maintained recipes, deploying each one at a time (Wraps the autonomic-recipe-maintainer skill /recipe-test-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-all.)"
---
# cctest-recipe-test-all (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-all/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-test-backup
description: "[recipe-maintainer/cctest] Test backing up and restoring a recipe's test instance (Wraps the autonomic-recipe-maintainer skill /recipe-test-backup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-backup.)"
---
# cctest-recipe-test-backup (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-backup/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-test-new
description: "[recipe-maintainer/cctest] Test a recipe's first-time initialization from scratch (Wraps the autonomic-recipe-maintainer skill /recipe-test-new; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-new.)"
---
# cctest-recipe-test-new (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-new/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-test-update
description: "[recipe-maintainer/cctest] Test upgrading a recipe's test instance using abra app deploy (Wraps the autonomic-recipe-maintainer skill /recipe-test-update; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test-update.)"
---
# cctest-recipe-test-update (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test-update/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-test
description: "[recipe-maintainer/cctest] Run all tests for a Co-op Cloud recipe (Wraps the autonomic-recipe-maintainer skill /recipe-test; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-test.)"
---
# cctest-recipe-test (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-test/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-upgrade-apply
description: "[recipe-maintainer/cctest] Execute a planned recipe upgrade — apply changes, deploy, test, commit/tag (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-apply; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-apply.)"
---
# cctest-recipe-upgrade-apply (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-apply/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-upgrade-cron-all
description: "[recipe-maintainer/cctest] Autonomous weekly upgrade run — overview all recipes, upgrade each end-to-end (sequentially by default, parallel with --parallel), open PRs (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-cron-all; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-cron-all.)"
---
# cctest-recipe-upgrade-cron-all (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-cron-all/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-upgrade-full
description: "[recipe-maintainer/cctest] Plan and apply a recipe upgrade end-to-end, no human review in the middle (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-full; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-full.)"
---
# cctest-recipe-upgrade-full (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-full/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-upgrade-plan
description: "[recipe-maintainer/cctest] Create a detailed upgrade plan for a recipe (Wraps the autonomic-recipe-maintainer skill /recipe-upgrade-plan; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upgrade-plan.)"
---
# cctest-recipe-upgrade-plan (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upgrade-plan/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-recipe-upstream
description: "[recipe-maintainer/cctest] From a git.autonomic.zone review-PR URL, fetch the branch + tag locally and emit the commands to open the upstream PR on git.coopcloud.tech (Wraps the autonomic-recipe-maintainer skill /recipe-upstream; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-recipe-upstream.)"
---
# cctest-recipe-upstream (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/recipe-upstream/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-setup-sandbox
description: "[recipe-maintainer/cctest] Guide for setting up a sandboxed environment to run the agent with recipe-maintainer (Wraps the autonomic-recipe-maintainer skill /setup-sandbox; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-setup-sandbox.)"
---
# cctest-setup-sandbox (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/setup-sandbox/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-switch-default-instance
description: "[recipe-maintainer/cctest] Switch the default test instance (b1cc or t1cc) for all recipe operations (Wraps the autonomic-recipe-maintainer skill /switch-default-instance; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-switch-default-instance.)"
---
# cctest-switch-default-instance (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/switch-default-instance/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-sync-secrets
description: "[recipe-maintainer/cctest] Sync Docker secrets from the test server into recipe-info/testsecrets/ (Wraps the autonomic-recipe-maintainer skill /sync-secrets; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-sync-secrets.)"
---
# cctest-sync-secrets (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/sync-secrets/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-t1cc-start
description: "[recipe-maintainer/cctest] Provision the t1cc DigitalOcean test server and deploy Traefik (Wraps the autonomic-recipe-maintainer skill /t1cc-start; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-start.)"
---
# cctest-t1cc-start (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/t1cc-start/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-t1cc-stop
description: "[recipe-maintainer/cctest] Destroy the t1cc DigitalOcean test server via terraform (Wraps the autonomic-recipe-maintainer skill /t1cc-stop; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-t1cc-stop.)"
---
# cctest-t1cc-stop (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/t1cc-stop/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-test-context-reset
description: "[recipe-maintainer/cctest] Undeploy all apps from the test server except traefik (Wraps the autonomic-recipe-maintainer skill /test-context-reset; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-context-reset.)"
---
# cctest-test-context-reset (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/test-context-reset/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
@@ -0,0 +1,26 @@
---
name: cctest-test-setup
description: "[recipe-maintainer/cctest] Verify the test environment is configured correctly (Wraps the autonomic-recipe-maintainer skill /test-setup; runs against the cctest test server + ARM sandbox, not cc-ci. Invoke as /cctest-test-setup.)"
---
# cctest-test-setup (cctest wrapper)
**Canonical procedure:** `references/recipe-maintainer/.opencode/skills/test-setup/SKILL.md`
— read it and follow it. This wrapper only sets context + policy.
**Context:** this is an **autonomic-recipe-maintainer (ARM)** skill. It operates on the
recipe-maintainer **cctest** test server / local abra sandbox — NOT on the cc-ci CI server or
its shared swarm. Execute with the submodule as your working directory:
`cd /srv/cc-ci-orch/references/recipe-maintainer`. If the ARM environment is not yet
configured on this host (`settings.toml` from `settings.toml.example`, sandbox/test instances),
run `/cctest-intro` / `/cctest-setup-sandbox` first.
**Unified policy (same as cc-ci — no differences):**
- **Recipe PRs are NEVER merged by an agent.** Every flow ends at an open PR; the operator
reviews and merges. This is ARM's own rule too ("PRs are reviewed and merged manually by a
human afterwards — never pushes to upstream or merges anything"); ARM's "no human review in
the middle" wording refers only to skipping the mid-run plan confirmation, not to merging.
- Never touch cc-ci infrastructure (the CI server, its swarm, `/root/*` clones, the weekly
timers) from an ARM skill — cc-ci work goes through the cc-ci skills.
- The submodule is **pinned**: don't commit into it from here; upstream ARM changes arrive via
a deliberate submodule bump + `scripts/gen-cctest-skills.py` regeneration.
+110
View File
@@ -0,0 +1,110 @@
---
name: help
description: Operator orientation - lists every skill available on this orchestrator with what it does and when to reach for it, grouped by purpose (status, weekly maintenance, host updates, tests, enrollment, recovery, reporting), plus a "what do you want to do?" guide for common situations. Read-only. Invoke as /help.
---
# help
Orient the operator: what this orchestrator can do, via which skill, and what's sensible next.
**First, enumerate the live roster** (skills get added — don't trust this file's list blindly):
```
ls /srv/cc-ci-orch/.opencode/skills/ # canonical definitions (one dir per skill)
```
For any skill not described below, read its `SKILL.md` frontmatter description and include it.
Then present the roster grouped as follows, and close with the situation guide.
## The roster (curated descriptions — merge with the live listing)
**Status & orientation**
- **/cc-ci-status** — the comprehensive read-only health/status check: weekly-run outcomes,
report publishing, stale tests, CVE-PR aging, host update recency, service health, bridge
`!testme` path. Ends `ALL HEALTHY` or prioritized findings each mapped to a skill. **Start
here when unsure.**
- **/help** — this orientation.
**Weekly maintenance (recipes)**
- **/upgrade-all** — the weekly sweep: survey every `weekly` recipe, open verified upgrade PRs,
write the summary + report. Runs on a timer (`cc-ci-upgrade-all.timer`, Fri 02:00 UTC);
invoke manually to run it now.
- **/recipe-upgrade <recipe>** — the same pipeline for ONE recipe (plan → bump → verify green →
PR). `--with-tests` also fixes that recipe's stale test.
- **/recipe-report** — (re)generate the weekly report page for report.ci.commoninternet.net.
**Keeping the PR list honest**
- **/cc-ci-cleanup** — reconciles every mirror from true upstream (which alone closes PRs upstream
already merged), then closes the open recipe PRs that can no longer merge or were never meant to
(CI sweep artifacts, obsolete bumps, superseded duplicates) and reports what is actually blocking
the ones that should land. Never merges.
**Security (CVEs)**
- **/cve-check** — fleet-wide CVE sweep with **no upgrading**: for every recipe, work out what
upgrade is available (per image, sidecars included), scan it for CVEs, and publish a CVE report.
Read-only and quick — the "what are we exposed to?" answer without an upgrade run.
- **/cve-check-and-upgrade** — the same sweep, then open verified PRs **only** for the recipes whose
upgrade actually fixes a CVE, worst severity first. `--min-severity high` for just the urgent ones.
Never merges.
**Tests**
- **/cc-ci-tests-update** — fleet-wide stale-test cleanup: find tests broken by legitimate
upstream changes, fix without weakening, verify, merge the test PRs.
- **/ci-test-review** — diagnose a specific red CI run: classify recipe bug vs stale test vs
CI-server bug, then fix on the right side.
**cc-ci itself**
- **/cc-ci-server-update** — bump the CI **server** host's nixpkgs/sops-nix, deploy with
build → `nixos-rebuild test` → switch + health gate, PR merged on green.
- **/cc-ci-orchestrator-update** — same for **this** orchestrator host (self-update caveats).
- **/cc-ci-update** — both of the above plus /cc-ci-tests-update in one pass.
- **/ci-dev-workflow** — harness/CI-server development discipline (changing cc-ci itself).
**Enrollment**
- **/recipe-enroll <recipe>** — add a NEW recipe to the maintained set end-to-end: mirror,
test suite, bridge + inventory enrollment, full-suite-green verification, bridge deploy.
**Recovery**
- **hetzner-server-recovery** — when a Hetzner host is unreachable over SSH/tailscale: API
reboot, rescue mode, GRUB generation selection (submenu ids are `1>N`), console access.
**Recipe-maintainer toolkit (`/cctest-*` — the ARM sandbox + cctest test server, NOT cc-ci)**
The full autonomic-recipe-maintainer skill set, vendored as a pinned submodule
(`references/recipe-maintainer`) and exposed with the `cctest-` prefix — ~30 skills for
hands-on recipe work against the recipe-maintainer **cctest** test server and local abra
sandbox: `/cctest-intro` (start here), `/cctest-recipe-overview`, `/cctest-recipe-init`,
`/cctest-recipe-deploy`, `/cctest-recipe-test*`, `/cctest-recipe-upgrade-plan|apply|full`,
`/cctest-new-recipe-guide`, sandbox/instance management (`/cctest-setup-sandbox`,
`/cctest-t1cc-start|stop`), and more — enumerate with `ls .opencode/skills | grep ^cctest-`.
**Rule of thumb:** verifying/shipping against the CI pipeline → the cc-ci skills above;
exploratory or hands-on recipe development on a test instance → `/cctest-*`. Policy is
unified: recipe PRs are never agent-merged on either side (operator reviews + merges), and
ARM skills never touch cc-ci infra. After a submodule bump run `scripts/gen-cctest-skills.py`.
## "What do you want to do?"
| Situation | Do this |
|---|---|
| "How is everything?" | `/cc-ci-status` |
| "Run the weekly upgrades now" | `/upgrade-all` (or `systemctl start cc-ci-upgrade-all.service`) |
| "Upgrade just <recipe>" | `/recipe-upgrade <recipe>` |
| "The report site is stale/missing a week" | `/recipe-report` |
| "The open PR list is a mess / what should I merge?" | `/cc-ci-cleanup` |
| "What CVEs are we exposed to right now?" | `/cve-check` (read-only, no PRs) |
| "A CVE just dropped — check and patch it" | `/cve-check-and-upgrade` (add `--min-severity high` to skip the noise) |
| "Is <recipe> vulnerable?" | `/cve-check <recipe>` |
| "Tests are red because upstream changed" | `/cc-ci-tests-update` (fleet) or `/recipe-upgrade <r> --with-tests` |
| "A CI run failed and I don't know why" | `/ci-test-review` |
| "Update the CI server OS/deps" | `/cc-ci-server-update` |
| "Update this orchestrator's OS/deps" | `/cc-ci-orchestrator-update` |
| "Add <recipe> to what we maintain" | `/recipe-enroll <recipe>` |
| "A host is unreachable" | `hetzner-server-recovery` |
| "Hack on / bootstrap a recipe in a sandbox" | `/cctest-recipe-init`, `/cctest-recipe-deploy`, `/cctest-recipe-test` |
| "New to recipe work, where do I start?" | `/cctest-intro` |
| "What needs my review?" | `/cc-ci-status` → its open-PR section lists CVE-urgent + ready-to-merge PRs |
**Standing conventions** (all skills follow these): PRs are opened for visibility and merged
directly once verified (invocation = authorization) — except recipe upgrade PRs, which stay
operator-merged; `nixos-rebuild test` before any `switch`; never weaken a test; single-writer
branches; serialize deploy-heavy work on the shared swarm.
If several things need doing, run `/cc-ci-status` first — its findings come pre-prioritized
with the skill to invoke for each.
+137
View File
@@ -0,0 +1,137 @@
---
name: recipe-enroll
description: Add a NEW recipe to cc-ci's maintained set, end to end — create + sync the recipe-maintainers mirror from coopcloud upstream, author a real test suite (health floor + non-vacuous recipe-specific tests incl. a create-an-object round-trip), enroll it in the !testme bridge (POLL_REPOS) and the weekly /upgrade-all inventory (used-recipes.md weekly row + upstream registry), then VERIFY the whole thing: full harness suite green with the new tests, bridge deployed (test-before-switch) and healthy. Opens the cc-ci PR for visibility and merges it directly once verification is green (the skill invocation is the authorization); the report lists merged PR links + what changed. Invoke as /recipe-enroll <recipe>.
---
# recipe-enroll
Enroll a coopcloud recipe as a **maintained** recipe: mirrored, test-covered, `!testme`-triggerable,
and picked up by the weekly `/upgrade-all`. This is the full end-to-end path — worked example:
the 2026-08-03 wordpress enrollment (cc-ci PR #14).
**"Maintained" = ALL of:**
1. Mirror `recipe-maintainers/<recipe>` exists, `main` = coopcloud upstream main, tags synced.
2. `tests/<recipe>/` enrolled in the cc-ci repo (this is what `/upgrade-all` + `ci-test-review`
enumerate).
3. `POLL_REPOS` in `nix/modules/bridge.nix` includes the mirror (the `!testme` bridge).
4. `cc-ci-plan/used-recipes.md` has a `<recipe> weekly` row (orchestrator repo — `/upgrade-all`
skips recipes without it or tagged `external`).
5. `cc-ci-plan/upstream/<recipe>.md` registry entry (release-notes URLs + standing notes).
## Preconditions
- The recipe exists upstream: `ssh cc-ci 'script -qec "abra recipe fetch <recipe>" /dev/null'`
succeeds (every recipe must have a coop-cloud correspondent).
- `GITEA_*` creds in `/srv/cc-ci/.testenv` (orchestrator side; the cc-ci host does NOT have this
file — inject creds over stdin when running helper scripts there, see step 2).
- The shared Swarm is quiescent for step 6 (verification deploys the recipe) — do NOT verify
concurrent with `/upgrade-all` or other verify runs; author everything first, verify when clear.
## Procedure
### 1. Survey the recipe (read-only)
On cc-ci, after `abra recipe fetch <recipe>`, read `~/.abra/recipes/<recipe>/`:
- `compose.yml` — services + images (for the upstream registry), healthcheck (`start_period`
informs `DEPLOY_TIMEOUT`), traefik labels.
- `.env.sample` — is the app self-initializing, or does a fresh deploy sit in a setup wizard
(e.g. wordpress without `POST_DEPLOY_CMDS core_install`)? The tests must handle the state a
fresh CI deploy actually lands in.
- Auth model — how will a test create an object? (REST + token, session login, XML-RPC, …)
Check overlay configs (htaccess/nginx templates) for blocked endpoints before relying on one.
### 2. Create + sync the mirror
Create `recipe-maintainers/<recipe>` (Gitea API: `POST /orgs/recipe-maintainers/repos`,
`{"name":…,"private":true,"default_branch":"main","auto_init":false}`), then force-sync from
coopcloud with the existing helper **run on cc-ci with creds injected via stdin** (the host has
no `.testenv`):
```
set -a; . /srv/cc-ci/.testenv; set +a
{ printf 'export GITEA_USERNAME=%q GITEA_PASSWORD=%q GITEA_URL=%q\n' "$GITEA_USERNAME" "$GITEA_PASSWORD" "$GITEA_URL";
cat /srv/cc-ci-orch/.claude/skills/recipe-upgrade/open-recipe-pr.sh; } \
| ssh cc-ci 'bash -s -- <recipe> --reconcile-only'
```
Expect: repo created (or exists), `main` force-synced to upstream, published tags pushed.
### 3. Author the test suite (dedicated cc-ci clone + branch)
```
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests-<recipe>
cd … && git checkout -b test/<recipe>-enroll-$(date -u +%Y%m%d)
```
`tests/<recipe>/` contents (template: `tests/uptime-kuma/`, `tests/wordpress/`):
- **`recipe_meta.py`** — `HEALTH_PATH` / `HEALTH_OK` (accept the fresh-deploy state, e.g. a
302 to a setup wizard), `DEPLOY_TIMEOUT` (recipe healthcheck `start_period` + DB init +
first-boot copy, be generous), `HTTP_TIMEOUT`, `WARM_CANONICAL = True` (canon §2.B — all
recipes enroll as data-warm canonicals, operator 2026-06-17).
- **`custom/`** — the health floor + **≥2 recipe-specific, non-vacuous tests**, one of which is
the §4.3 **create-an-object + read-it-back** (write through the app's real API, read back via
a *different* path where possible — e.g. wordpress: XML-RPC write → REST read → permalink
HTML). If the app needs setup/auth, put it in a recipe-local `_<recipe>.py` helper
(idempotent `ensure_*` so test ordering doesn't matter; run-scoped class-B credentials —
the app is destroyed at teardown). Design assertions to name the broken layer (e.g. assert
both the rewrite-dependent and rewrite-independent API routes separately).
- **`PARITY.md`** — table of tests × what's verified × why non-vacuous; note there's no
recipe-maintainer parity corpus if so.
- Lint with the **repo dev-shell ruff**: `nix develop -c ruff check tests/<recipe>/ && nix
develop -c ruff format tests/<recipe>/` (pre-existing drift in other files is not yours).
### 4. Bridge enrollment (same branch)
`nix/modules/bridge.nix`: append `,recipe-maintainers/<recipe>` to the `POLL_REPOS=` CSV.
### 5. Inventory + registry (orchestrator repo, direct to main)
- `cc-ci-plan/used-recipes.md`: add `<recipe> weekly` row (alphabetical).
- `cc-ci-plan/upstream/<recipe>.md`: images table (source repo + releases/changelog links) +
standing notes (setup-wizard behavior, auth caveats, known upgrade traps).
- Commit + push (doc commits go direct to main in this repo).
### 6. Verify end-to-end — full suite GREEN with the new tests (swarm-serialized)
Open the cc-ci PR first (visibility): `TITLE=… BODY_FILE=… bash
/srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh`. The body: what's enrolled,
the test design rationale, the verify + deploy plan.
Then, when the swarm is clear:
```
ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/<recipe>-enroll-<date> \
ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \
cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true'
RECIPE=<recipe> REMOTE_ROOT=/root/cc-ci-test-verify \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh # no REF — recipe main
```
Required: **cold full-suite green** (install/upgrade/backup/restore/custom). Iterate the tests
(bounded, ≤3 attempts) if red — fix the TESTS to match real app behavior, never weaken. Clean up
`/root/cc-ci-test-verify` after.
### 7. Deploy the bridge change to the cc-ci host (test-before-switch)
The `POLL_REPOS` change only takes effect via a host rebuild. Per `/cc-ci-server-update` steps
5a-e: stage the branch to `/root/cc-ci-deploy` (+ secrets copy), `nixos-rebuild build`, detached
`nixos-rebuild test`, verify reachable + healthy, then `switch`. Confirm the bridge is polling
the new repo: `ssh cc-ci 'docker service inspect ccci-bridge_app --format "{{json
.Spec.TaskTemplate.ContainerSpec.Env}}"' | grep <recipe>` and the bridge task is 1/1 with no
auth errors in its logs (a stale Gitea secret 401s silently — see the 2026-08-03 finding).
### 8. Merge + report
Merge the cc-ci PR (invocation = authorization; PR is the visible record — comment the
verification evidence first). Report to the operator: merged PR link + change summary, the
verify log path, and the note that the **next weekly `/upgrade-all` picks the recipe up
automatically** (it enumerates `tests/<recipe>/` dirs × `weekly` rows).
## Guardrails
- **Full-suite green is the enrollment gate** — an enrolled-but-red recipe poisons every future
sweep. Don't merge on partial green.
- **Never weaken**: the tests assert the app's real current behavior, incl. asserting removed
auth paths are rejected where that's the upstream intent.
- **Single-writer**: dedicated clones/branches; never push `main` of cc-ci; never touch
`/root/builder-clone` or the loops' clones; `/root/cc-ci-test-verify` is yours — remove after.
- **Serialize on the swarm**: authoring is free, verification + bridge deploy wait for
`/upgrade-all`/other runs to finish.
- **abra over ssh needs the pseudo-TTY wrap**: `ssh cc-ci 'script -qec "abra …" /dev/null'`.
- **PRs for visibility, merged directly once verified**; failed enrollment leaves the PR open
with an explanatory comment and the report says exactly what's missing.
+26
View File
@@ -115,3 +115,29 @@ When the orchestrator, Builder, or assistant makes intentional repository change
promptly and push them to `git.autonomic.zone` in append-only fashion (never force-push). Match the
existing commit author and message style in this repo. Do not bundle unrelated worktree changes you
did not make; stage only the intended files.
## Ship as PRs, merge them yourself, operator reviews retrospectively
**This applies to the two INFRASTRUCTURE repos — `recipe-maintainers/cc-ci-orchestrator` (here) and
`recipe-maintainers/cc-ci` (the CI product).** For work in either:
1. Branch, don't commit straight to `main`.
2. Open a PR with a description written to be read **after** the fact: what changed, why, and what
evidence says it works (test output, a verified run, a before/after number). The PR *is* the
review artifact and the historical record.
3. **Merge it yourself once it is verified** — do not wait for review. The invocation is the
authorization; blocking on review would stall the pipeline these repos exist to run.
4. The operator reviews **retrospectively**, from the PR.
So the PR is not a gate — it is how the work stays legible. A PR that merely says "fix scanner" has
failed at its only job.
> ### This does NOT extend to RECIPE repos
> Recipe PRs — any `coop-cloud/<recipe>` or its `recipe-maintainers/<recipe>` mirror — are
> **created and verified but NEVER merged by an agent**. Those change what deploys on other people's
> infrastructure, so a human merges them. The split is deliberate: agents own the tooling, the
> operator owns the recipes.
If work has already landed on `main` without a PR, do not rewrite published history to fix it.
Create a branch pinned at the pre-work commit and open the PR against that, so the diff is still
reviewable and merging only advances the pointer (see PRs #2-#5, 2026-08-11).
+79
View File
@@ -791,3 +791,82 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
deploys to the cc-ci server and the orchestrator host — `test` leaves the bootloader/profile
untouched so a reboot always recovers. Codified in the cc-ci-server-update skill (step 5d) and
AGENTS.md (orchestrator rebuild instructions).
## 2026-08-03 ~21:20 UTC — /cc-ci-tests-update + first /cc-ci-orchestrator-update (backup orchestrator)
- /cc-ci-tests-update: both carry-over stale tests fixed via new recipe-local OIDC session-login
helper, verified GREEN cold full-suite paired with their recipe PRs, MERGED: cc-ci PR #12
(lasuite-docs, impress v5.4.0 Bearer removal) + PR #13 (lasuite-meet, meet v1.22.0 hardening).
Comments left on recipe PRs #7/#8 (operator-owned; merging them ⇒ green — both carry the nginx
1.31.3 CVE batch). Fresh 20-recipe sweep skipped in favor of the immediately-following
/upgrade-all (recorded in .cc-ci-logs/tests-update-2026-08-03.md). discourse #6 remains an
operator migration decision (not stale-test).
- /cc-ci-orchestrator-update (first run of the new skill): host bumped 5b4f72e→531670d
(26.05 tip), gen 34, test-before-switch honored, PR cc-ci-orchestrator#1 merged.
Log: .cc-ci-logs/orchestrator-update-2026-08-03.md
- Next: kicking off the weekly /upgrade-all (opencode ZEN glm-5.2) and monitoring to completion
incl. report generation, per operator instruction.
## 2026-08-04 ~17:15 UTC — weekly run COMPLETE + bridge fixed + wordpress live + session hygiene
- Weekly /upgrade-all 2026-08-03 finished after 3 provider-billing interruptions (Go monthly wall
resets ~Aug 22; ZEN balance twice). Final: 20 considered, 15 PRs opened/extended, 12 GREEN,
3 pending (discourse stale test; keycloak+mailu RED on a HARNESS canonical-baseline-404
regression from the 2026-08-03 nixpkgs bump — open task), 0 failed. Public page
week-2026-08-03.html published + privacy-linted (billing wording neutralized — standing rule).
- Model economics: subagents configured to opencode/deepseek-v4-pro (opencode.jsonc agent.general;
main sessions stay glm-5.2; report glm-5.2). NOTE: not yet confirmed live — the finish-run's
subagents still ran glm; verify on the next run that agent.general.model is honored.
- ccci-bridge FIXED: gitea-token swarm secret _v1 was frozen pre-rotation (ensure_secret is
create-once); nix now pins _v3 minted from current sops at deploy. Zero 401s; !testme restored.
- wordpress ENROLLED end-to-end and MERGED (cc-ci PR #14): suite verified GREEN cold full-suite,
bridge deployed via test-before-switch, POLL_REPOS includes wordpress. Joins next weekly survey.
- launch-upgrader watchdog bug FIXED + CONFIRMED (pinned session id + direct-GET validation;
archive-rename at launch). Root cause: /session API rows carry no time fields — sort degraded
to list order and resumed the wrong (giant) session, killing healthy runs.
- Session naming convention live + durable in all three launchers: exactly one canonical
cc-ci-upgrader / cc-ci-supervisor / cc-ci-report; ALL other top-level sessions renamed
archive-<title> (113 archived in the sweep; 33 restyled earlier).
- OPEN: harness canonical-baseline 404 (keycloak/mailu re-verify after fix); confirm ds4-pro
subagent config on next run; mattermost-lts 11.9.0 is innovation-release (EOL 2026-10-15) —
operator decides ESR vs innovation.
## 2026-08-04 ~18:00 UTC — all three pending weekly-run PRs unblocked (operator ask)
- keycloak #5: harness canonical-baseline 404 NOT reproducible post re-activation; repro build
#1199 full green; cc-ci/testme=success reflected. mailu #6: re-verify #1200 hit a cold-pull
900s deploy timeout, retry #1202 green; reflected. (Missing-runs-dir anomaly = install-stage
failure artifact, not a runner bug.)
- discourse #6: TWO stale-test roots fixed in cc-ci PR #15 (MERGED, verified level 5/5 paired
with the recipe PR head): (1) new UPGRADE_BASE_FLOOR recipe_meta key + resolver support —
excludes structurally-invalid upgrade bases (0.8.x bitnami family) while resolution stays
dynamic; declared skip when nothing ≥ floor; (2) the upgrade faithfulness test's hardcoded
discourse/discourse:3.5.3 pin → version-agnostic official-family assertion. Never-weaken held.
- Debug gitea tokens minted for the drone triggers all deleted (ids 55-57); the two 401s during
verification were transient git.autonomic.zone blips (sops bridge token verified valid, 200).
- Operator review queue: keycloak #5, mailu #6, discourse #6 all green + reflected.
## 2026-08-10 — 2026-08-07 weekly run FINISHED + three real bugs fixed
- The 2026-08-07 run did all 18 per-recipe upgrades then died at the summary step on the provider's
monthly cap (05:26 UTC) — and then sat unfinished and UNREPORTED for 3 days. Operator raised the
cap; a finish-run wrote upgrade-all-2026-08-07.md and published week-2026-08-07.html
(privacy-linted; only false positive is Ghost's own "billing search" changelog text).
Result: 17 surveyed, 14 GREEN, 0 new failures, 2 up-to-date, 1 long-standing cross-major hold.
- BUG 1 (3-day silence) — supervisor progress gate. _run_pids() substring-matched the WHOLE cmdline
for the session name, and an agent's kickoff PROMPT is an argv element: the supervisor's own
billing-hung agent contained "cc-ci-upgrader" and matched as a live upgrader run. It read its own
corpse as health ~60 times. Now matches FLAG VALUES (--title/-s) only, AND progress requires the
SESSION TREE to have advanced — a live-but-idle proc is not progress (that proc spun 3 days
emitting nothing). Billing-walled runs are REPORTED, never killed (operator policy): they may
resume when the wall lifts. Commit d441c6c.
- BUG 2 (my regression from the 2026-08-04 pinning work) — launch-report.py never cleared/re-pinned
its session id, so the shared watchdog resolved a PREVIOUS report session that already carried
RECIPE REPORT COMPLETE, declared "run completed" and exited in 3 min, leaving the live report
unwatched. start() now archives titles, clears the pin, and re-pins after launch (same contract
as the upgrader). Commit 242a6d9.
- BUG 3 — the deepseek subagent config never bound; the entire 2026-08-07 run billed as glm (17/17
subagents). Root cause is PLACEMENT, not the .jsonc extension (that parsed fine): launcher-started
sessions pass no --dir so they inherit the opencode SERVE process's project
(/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
real launcher) and landed in a different project.
+386
View File
@@ -0,0 +1,386 @@
# Advisory scan — specification
What `cc-ci-plan/advisory-scan.py` does, step by step, and why each step exists. This documents the
implementation as it stands (2026-08-11); if you change the code, change this file in the same commit.
`cc-ci-plan/test-advisory-scan.py` is the executable half of this spec — every rule below is asserted
there.
**Role.** A per-recipe CVE detector run as a **pre-step of `/recipe-upgrade`** (step 2a). It is
**strictly additive**: it never replaces the release-note reading the upgrade agent already does. The
CVE count reported for a recipe is the **union** of what the agent read and what this scan found; the
scan may never *lower* a count established by reading.
**Why it exists.** gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). The weekly
report printed gitea's CVE count as `1`, then `none`. The upgrade agent had read the GitHub *release
notes*, which name neither — both were announced only in the vendor's blog security section — and the
report then derived security content from those notes plus model knowledge, which predates the CVEs.
Nothing in the pipeline queried an advisory source. This scan closes that hole.
---
## Two passes
| | Pass 1 — measure | Pass 2 — judge (`--adjudicate`) |
|---|---|---|
| Who | Pure Python, no model | The calling agent, a model |
| Does | Collects evidence and decides every case it can by arithmetic | Weighs the collected evidence on cases arithmetic cannot settle |
| Output | A count, or `UNKNOWN` | FIXED / NOT-FIXED / STILL-UNKNOWN per open case |
| Rule | Deterministic and reproducible | May only **raise** the count, never lower it |
**Prefer pass 1.** Every case pass 1 decides is one that reproduces identically next week. Pass 2 exists
for evidence that is prose rather than data — a fallback, not a co-equal stage. When a class of case
keeps landing in pass 2, the fix is a new deterministic method in pass 1. §4c is exactly that: it moved
12 redis advisories out of pass 2 and into arithmetic.
---
## Inputs
```
advisory-scan.py <recipe> [--from <version>] [--to <version>]
[--image <name>=<from>:<to>]... [--adjudicate] [--json] [--registry DIR]
advisory-scan.py <recipe> --compose-to <URL> [--compose-from <URL>] # windows derived, not typed
```
| Input | Meaning |
|---|---|
| `<recipe>` | Recipe name; selects `cc-ci-plan/upstream/<recipe>.md` (the per-recipe URL registry) |
| `--from` / `--to` | The **primary app image's** version window being upgraded across |
| `--image NAME=FROM:TO` | A **sidecar image and the versions it moved between** (repeatable, all in ONE call). `NAME` is substring-matched against source repo names. Malformed values warn on stderr and are skipped. Without it that image's advisories stay unclassified. |
| `--compose-to URL` | **Derive every window by diffing this compose against its baseline**, instead of typing `--from/--to/--image`. Point it at a PR's `compose.yml`. |
| `--compose-from URL` | Baseline for the above. Default: the same repo's **default branch, resolved from the API** — never assumed to be `main`. |
| `--adjudicate` | Run pass 2: append the evidence dossier for judgement |
| `--registry` | Registry dir; also `CCCI_UPSTREAM_REGISTRY` |
| `GITHUB_TOKEN` / `GITHUB_TOKEN_FILE` | Read-only token; **rate limit only** (60/hr anonymous → 5000/hr). Default file `/srv/cc-ci/.github-token`, mode 600. Public advisories need **no scopes**. |
Exit code is always 0 — this is informational. Failures are *reported*, never raised.
---
# Pass 1 — deterministic
## Step 1 — Collect source URLs from the registry
Read `cc-ci-plan/upstream/<recipe>.md` and extract every `http(s)://…` URL.
**Trailing markdown punctuation is stripped** (`` ` `` `'` `"` `*` `.` `,` `;` `:` `>` `)`). The registry
is markdown, so URLs appear inside backticks and quotes; capturing the punctuation produced fetches of
`https://docs.n8n.io/release-notes/\`` which 404, and made immich and n8n render `?` for no real reason.
> **Registry hygiene matters.** The scan can only look where the registry points. Two classes of defect
> have been found and fixed by running it: a **wrong URL** (`pgautoupgrade/pgautoupgrade`, which 404s —
> the repo is `pgautoupgrade/docker-pgautoupgrade`) and a **missing** one (gitea's CVEs are announced at
> `blog.gitea.com`, which the registry didn't list). When a vendor publishes security notes somewhere
> the registry lacks, add it.
## Step 2 — Query the sources
Three source classes, each recording **its own status** so *"checked, none found"* is never confused
with *"not checked"*.
### 2a. GitHub Security Advisories — PRIMARY
For every `github.com/<owner>/<repo>` URL in the registry:
`GET /repos/<owner>/<repo>/security-advisories`.
Captured per advisory: `cve_id`, `ghsa_id`, `severity`, `summary`, **`description`**, `published_at`,
and **all** `vulnerabilities[]` entries' `vulnerable_version_range` + `patched_versions` (joined `;`).
- **All entries, not just the first.** An advisory carries one entry **per patched release line** —
n8n patches three (1.123.32, 2.17.4, 2.18.1). Reading only `vulnerabilities[0]` silently dropped the
line a deployment was actually on, and misclassified CVE-2026-42231/42232 as out-of-window.
- **Pagination via the `Link rel="next"` cursor**, to exhaustion (cap 20 hops). This endpoint returns
at most 100 rows **and ignores `?page=`** — it re-returns the same rows, which silently truncates busy
projects. discourse has 286 advisories; a single page cannot even cover one upgrade window.
- **The description is kept from this response.** It is already present here, and pass 2 needs the
prose; re-fetching it per advisory would cost one request each.
- **HTTP 404 ⇒ `no-advisories-published`** — a benign absence (many sidecar images publish none), **not**
a failure. Conflating the two pushed nearly every recipe to `?` and destroyed the signal.
### 2b. Vendor release / security pages
Every other registry URL is fetched, HTML-stripped, and scanned for `CVE-\d{4}-\d{4,7}`, keeping ±160
characters of context per hit.
URLs containing `<`, `>`, `{`, `}`, `VERSION`, or `vX.Y.Z` are **skipped as templates** — they are
human documentation (`…/changelog/v<VERSION>/`), not fetchable, and counting them as failures is wrong.
This is the source that would have caught gitea: the vendor blog names both CVEs, the GitHub release
page names neither.
**When the page is a changelog organised by release, each CVE is attributed to the release heading it
appears under** (`Changes with nginx 1.31.3`, `## v1.31.3`, …) and that becomes its fixed-in version.
Without this, a project that publishes no advisory feed can never contribute a CVE:
> **nginx publishes NO GitHub security advisories.** Every nginx CVE we can see comes from
> `nginx.org/en/CHANGES`. Scraping ids out of it without attributing them to a release left them with
> no patched version, so they were never classifiable — and every nginx bump in the fleet reported
> **0** forever. nginx is a sidecar in most recipes. Measured: `1.31.1 → 1.31.3` fixes **six** CVEs
> (three in .2, three in .3); lasuite-docs#7 went 0 → 6 and lasuite-drive#6 went 0 → 3 on this alone.
A changelog CVE is tied to a window by the **image name appearing in the page URL** (window `nginx` ↔
`nginx.org/...`). A CVE found on a vendor page with no attributable release still has no version data,
so pass 1 cannot place it — it goes to pass 2 (§6).
### 2c. NVD by CPE — the fallback for projects that publish nothing
Declared per recipe in the registry as `nvd-cpe: <image-key> = <cpe:2.3:...>`.
> **Why it exists.** Two recipes could not see CVEs *at all*: `mattermost-lts` (empty GitHub advisory
> feed, security bulletins rendered client-side so a text sweep finds nothing) and `mumble` (nothing
> published anywhere the registry points). Their scans returned `?` — nothing measured. NVD is
> CPE-indexed and carries structured ranges, so it answers where the vendor does not: mattermost
> 10.5.0 → 10.12.4 now scores **165**, and mumble finds `CVE-2025-71264` (fixed 1.6.870).
Two range forms, both used:
| NVD field | meaning | how it is judged |
|---|---|---|
| `versionEndExcluding X` | fixed in X exactly | a normal patched version (§4a) |
| `versionEndIncluding X` | affected **up to and including** X; fix version unpublished | fixed when the upgrade crosses X, i.e. `from ≤ X < to` |
**NVD lags the vendor** — it had neither gitea CVSS-9.8 RCE at publication — so this is a fallback,
never a replacement for 2a/2b. Unauthenticated calls are rate-limited (~5/30s), hence the retry.
### 2d. OSV.dev — supplementary
Only when the recipe has an entry in `OSV_PACKAGES` (ecosystem + package) and a version is given.
> **Measured, not assumed.** For the two gitea CVEs, OSV **404'd on both** and returned only Go
> *dependency* advisories for the package; NVD's API had them by neither CPE, CVE id, nor keyword.
> **Advisory databases lag the vendor**, which is why 2a and 2b lead and this is supplementary.
## Step 3 — Union
All findings merge into one CVE map: id → `{sources[], severity, ghsa, vulnerable_range, patched,
description, published_at, url, cvss, context}`. A CVE seen by several sources keeps them all.
## Step 4 — Classify against the upgrade window
Two invariants govern this step, both learned from a wrong answer in production.
> **A. Every image is judged by its OWN versions.** The app repo uses `--from/--to`; each sidecar uses
> its own `--image NAME=FROM:TO`. **Pass them all in ONE invocation** — the count is a union across
> images, and the UNKNOWN guarantee in B only holds when a single run sees every one. An image with no
> window is **not** classified; its advisories are listed as unclassified so they stay visible without
> inflating the count. Each window is classified independently, so one may use version ranges while
> another falls back to dates.
> *Why:* discourse once reported **133**, of which **34 were redis CVEs** — including
> `CVE-2021-21309`, patched in redis 6.0.11 in 2021 — counted purely because 6.0.11 sits numerically
> inside discourse's `3.5.3 → 2026.7.1` range. The fix is not to ignore sidecars but to give each one
> the versions it actually moved through: with `--image redis=7.4:8.10`, discourse scores
> **140 = 123 (app, by date) + 17 (redis)**, and the redis seventeen include `CVE-2025-49844`,
> **critical**, invisible while sidecars went uncounted.
>
> **B. Never emit a number you cannot justify.** If no method can order a window, the count is
> `null` / `UNKNOWN`, never `0`. A `0` in a security column asserts safety. Equally, an advisory that
> cannot be judged is **indeterminate** (§4d) — never silently counted as "not fixed".
### 3b. Deriving the windows from a compose diff (`--compose-to`)
Typing `--from/--to/--image` by hand means someone has to remember that the recipe also bumped its
redis. That is how sidecar CVEs went uncounted for months. This mode reads the windows off the diff:
1. Fetch both compose files (baseline = the repo's **default branch from the API**, since several
recipes keep a stale `main` beside a live `master`).
2. Parse `{service: (image-repo, tag)}` — keyed by **service, not image repo**, because an upgrade
may change the repo itself (plausible moved `plausible/analytics` →
`ghcr.io/plausible/community-edition`; keyed by repo that reads as one image vanishing and an
unrelated one appearing, losing the app window entirely).
3. Every service whose tag or repo changed becomes a window. The `app` service drives `--from/--to`
(coop-cloud convention: it is the recipe's primary image); the rest become `--image` windows.
Unchanged images produce no window — inventing one would be a false count.
4. The derived windows are printed to stderr before the scan, so the inputs are auditable.
Image names are matched against advisory sources **both ways** — an image name is often longer than
its source repo (`clickhouse/clickhouse-server` vs `ClickHouse/ClickHouse`) and sometimes shorter
(`redis` vs `redis/redis`).
Verified on plausible PR #5: from the compose URL alone it derives `v2.0.0 → v3.2.1` plus
`clickhouse-server 23.4.2.11-alpine → 24.12-alpine`, and reports **6** — identical to the
hand-specified args.
`--from/--to/--image` remain available for finer-grained checks (scanning a window that is not a
literal compose diff, e.g. "what would the compatibility-safe target fix?").
### 4a. By patched version (preferred — exact)
**A fix on the line you are upgrading FROM was already yours.** Projects that maintain several lines
patch them all at once: mattermost fixed `CVE-2025-11794` in 10.11.4, 10.12.1 *and* 10.5.12. An
upgrade 10.11.22 → 10.12.4 crosses 10.12.1, so a naive window test counts it — but 10.11.22 is
already past 10.11.4, so the deployment had the fix before the upgrade. Counting it credits the
upgrade with work it did not do. This check is **skipped for placeholder versions** (`7.4.X` parses
to a bare `7.4`, which would read as "already fixed at 7.4" and silently drop a real fix — exactly
how redis `CVE-2024-46981` was lost when the rule was first added).
`patched_versions` is a **range expression** (`">= 2.18.1"`), possibly several joined by `;`. Extract
every version-looking token; the advisory is **fixed-by-this-upgrade** if **any** patched version `p`
satisfies `from < p <= to` — exclusive lower (a fix already in the version you were on is not this
upgrade's doing), inclusive upper.
**Comparison is zero-padded to equal length**, so `18` == `18.0` == `18.0.0` as semver means it.
Without padding, plain tuple order makes `(18,) < (18,0)`, i.e. a fix in 18.0 falls *outside* a window
ending at 18 — and bare major tags are the norm for sidecars (`postgres:18`, `redis:8-alpine`). Padding
is permissive at the lower bound and conservative at the upper: with `to = 18`, a fix in `18.5` is
**not** counted, because nothing proves which 18.x a floating tag resolved to.
### 4b. By advisory publish date (fallback — temporal)
Used **only** when 4a cannot be trusted: a **version-scheme change**, detected as the leading version
component jumping by ≥ `SCHEME_JUMP` (100) — e.g. semver `3.5.3` → calver `2026.7.1`.
Version strings are unorderable across such a jump (`2025.12.2` compares "newer" than `3.5.3` while
shipping earlier), but **release dates always order**. So:
1. Resolve `--from` and `--to` to **git tag dates** on that source (tries `v<version>` then
`<version>`; annotated tag → tagger date, else commit date).
2. An advisory counts as fixed when `date_from < published_at <= date_to` — the same exclusive/inclusive
boundaries as 4a, so the two methods agree at the edges.
This reproduces, automatically, the hand count that established discourse `3.5.3` (2025-12-30) →
`2026.7.1` (2026-07-31) = **123 CVEs**.
*Assumption:* the vendor publishes advisories at fix time (true for discourse). The count includes
**first-party plugin advisories** where the vendor files them on the same repo — which is why a
plugin-rich project scores far higher than a monolith, not a statement about relative security.
### 4c. By release notes naming the CVE (rescue — for advisories with no fix version)
Applied to advisories 4a/4b could not decide, **before** giving up on them. Fetch the source repo's
GitHub **releases** (cached per repo, 4 pages) and find every tag whose notes **name the CVE id**. If
any such tag falls inside the window by 4a's rule, the advisory is fixed by this upgrade, and the
naming tags are recorded in `fix_versions_from_release_notes` as the citation.
> **Why this is not optional.** Vendors routinely publish an advisory with `patched_versions: "TBD"`
> and then name the CVE in the release notes of every branch that got the fix. **All 12** redis
> advisories crossed by discourse's redis bump are exactly this shape — `TBD`, or a placeholder like
> `7.4.X`, with an open-ended `vulnerable_version_range` (`All`, `>= 7.0.0`) — yet each is named in
> concrete releases (`CVE-2025-32023` → 6.2.19, 7.2.10, 7.4.5, 8.0.3, 8.2.0). Without this method
> discourse's redis contribution reads 5; with it, 17. Six of the twelve are high severity.
### 4d. Otherwise — indeterminate, not excluded
An advisory from a **windowed** source that none of 4a4c could decide — no usable `patched_versions`,
an open-ended vulnerable range, and no release note naming it — is recorded as **indeterminate**. It is:
- **not** added to the count (nothing justifies counting it), and
- **not** treated as unaffected (nothing justifies dismissing it either).
It is listed prominently, the headline reads *"(at least — see undetermined below)"*, and it becomes an
input to pass 2. Silently excluding these is the same defect class as printing `0` for an unscanned
recipe, one level down.
### 4e. Unorderable window
If neither 4a nor 4b can order a window at all, `count_known = false`, `cve_count_fixed = null`, and the
headline reads **"CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count"** with an
explicit *"This is NOT zero"*. If **any** requested window is unorderable the whole count is suppressed;
a partial number would understate a security figure.
## Step 5 — Output
Markdown (default) for pasting into the per-recipe upgrade log, or `--json`.
| Field | Meaning |
|---|---|
| `cve_count_fixed` | Union across all windows, or **`null`** if any window was unorderable |
| `count_known` | Distinguishes "counted zero" from "could not count" |
| `cve_count_indeterminate` / `indeterminate[]` | Judged by nothing; a **floor marker** on the count |
| `resolved_by_release_notes` | CVE → the tags that named it (4c citations) |
| `windows` | Every source classified, with its from/to |
| `classified_by` | **Per source**: which method decided it |
| `date_window` | **Per source**, when 4b was used |
| `fixed_by_this_upgrade[]` | CVE ids, with severity / GHSA / fixed-in per id |
| `unclassified[]` | Seen but not attributable (other images, or vendor-page-only) |
| `sources[]` / `sources_failed[]` / `sources_benign[]` | Per-source status; only genuine failures in `failed` |
---
# Pass 2 — adjudication (`--adjudicate`)
## Step 6 — What gets judged
Two kinds of open case, both real gaps rather than noise:
1. **Indeterminate** (§4d) — from an image *with* a window, but no fix version is knowable anywhere.
2. **Vendor-page-only** — a CVE seen only on a vendor security page, with no structured advisory
behind it. **gitea's two CVSS-9.8 RCEs are this shape.** They carry no version data, so no
arithmetic can place them, but the page prose usually states the fixed release.
## Step 7 — The evidence dossier
Pass 1 collects; pass 2 judges. Nothing in the dossier interprets — it assembles what was *measured*,
so the judgement is made against evidence rather than recollection. That distinction is the whole
point: the original failure was a report leaning on model knowledge that predated the CVEs, with no
source queried at all.
Per open case: severity, CVSS, sources, its window, why it is undecided, `patched_versions` and
`vulnerable_version_range` **as published**, summary, full description, references, affected ranges
with `first_patched_version`, and **every release tag naming the CVE** — whether or not in window, since
the model may reason about branch lines the arithmetic deliberately will not.
**Pass 2 also sees every decision pass 1 made** — a compact table of counted and excluded advisories
with the evidence behind each verdict. A deterministic verdict can still be wrong (a mis-parsed range,
a release note that mentions a CVE without fixing it), and only a reader with the evidence in front of
it can catch that. Silence means agreement.
## Step 8 — The verdict contract
For each open case: **FIXED** / **NOT-FIXED** / **STILL-UNKNOWN**, each with a one-line reason
**citing the evidence shown**. Every FIXED is added to the recipe's count — pass 1's number is a floor,
not a total. If the evidence does not settle it, **STILL-UNKNOWN**: do not infer from memory of the
project, and never record an undecided CVE as unaffected.
**No silent caps.** `MAX_ADJUDICATE` (25) and `MAX_REVIEW_ROWS` (400) bound the output; whenever either
truncates, the block says how many were dropped and that the unshown remain undetermined.
---
## How consumers must read it
`/recipe-report` renders the `cve` column from the **union** of this scan and the agent's own reading:
- a clean scan → its number, **including `0`**;
- **failed sources**, or `UNKNOWN` → **`?`**, never `none` — a blank reads as "clean", which is exactly
how two CVSS-9.8 gitea RCEs were published as `none`;
- **no upgrade this run** → `0`, not `?` — nothing an upgrade could have fixed;
- a count with **indeterminate advisories** → the number is a floor; say so in the notes;
- benign notes → never `?`.
`?` must stay **rare**: it means *we tried and could not tell*, not *we did not look*. A rash of `?` is
a bug to raise in the report's Addendum, not a normal outcome — every instance so far traced to a defect
in this tool or stale registry data.
## Testing
`test-advisory-scan.py` — 58 offline tests (fixtures, no network) plus 6 live regressions against the
counts published in week-2026-08-07. **The offline tier covers pass 1 only, by design**: pass 2's
judgement is a model's and cannot be asserted deterministically. What *is* tested about pass 2 is the
part that stays deterministic — which cases it selects, and that truncation is always announced.
```
python3 test-advisory-scan.py # offline
python3 test-advisory-scan.py --live # + historic report numbers
```
`audit-advisory-scan.py` re-derives the counts with a **separate** semver implementation and its own
release fetch, then diffs against the scanner. Run it after changing classification; it is what caught
the 12 undercounted redis CVEs.
## Known limits
1. **Versions must be supplied per image.** An image with no `--image` is not counted — the scan will
not guess a version range it was not given. `/recipe-upgrade` passes one per image it bumped.
2. **Date-based counts are temporal**, not exact — they assume publish-at-fix-time.
3. **Registry-bound.** Unlisted vendor security pages are invisible; the scan cannot know what it was
never pointed at.
4. **`NAME` matching is substring-against-source-name**, so a short or generic name can attach to more
than one repo (`postgres` matches `discourse/discourse-postgres`). The primary source is claimed
first and cannot be stolen. A name matching **nothing** is silently ignored — a typo costs coverage
without warning.
5. **Release-note rescue (4c) trusts that naming implies fixing.** A release note that merely mentions a
CVE would be read as fixing it. Pass 2's review table exists partly to catch this.
6. **Rate limit** without a token is 60/hr — a full weekly sweep will exhaust it and degrade to failed
sources (visibly, but degraded).
+1283
View File
File diff suppressed because it is too large Load Diff
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
"""Independent audit of advisory-scan's counts.
Deliberately does NOT reuse the scanner's classifier. Re-parses patched versions with a separate
semver implementation and re-derives membership, then diffs against what the scanner concluded.
Anything the two disagree on is a miscategorization in one of them.
"""
import importlib.util, json, re, sys
spec = importlib.util.spec_from_file_location("A", "/srv/cc-ci-orch/cc-ci-plan/advisory-scan.py")
A = importlib.util.module_from_spec(spec); spec.loader.exec_module(A)
REG = "/srv/cc-ci-orch/cc-ci-plan/upstream"
def sv(s):
"""Independent semver parse: strict 3-tuple, missing parts are 0."""
m = re.match(r"^\s*v?(\d+)(?:\.(\d+))?(?:\.(\d+))?", s or "")
if not m:
return None
return tuple(int(x) if x else 0 for x in m.groups())
def in_window(f, t, patched_expr):
"""Independent membership: any patched token strictly above f and at most t."""
kf, kt = sv(f), sv(t)
for tok in re.findall(r"\d+(?:\.\d+)*", patched_expr or ""):
c = sv(tok)
if c and kf and kt and kf < c <= kt:
return True
return False
_RELS = {}
def fetch_releases(owner_repo):
"""Independent releases fetch — deliberately NOT the scanner's cache or pagination helper."""
if owner_repo in _RELS:
return _RELS[owner_repo]
import urllib.request
tok = None
try:
tok = open("/srv/cc-ci/.github-token").read().strip()
except OSError:
pass
h = {"User-Agent": "audit", "Accept": "application/vnd.github+json"}
if tok:
h["Authorization"] = f"Bearer {tok}"
out, url, pages = [], f"https://api.github.com/repos/{owner_repo}/releases?per_page=100", 0
while url and pages < 4:
req = urllib.request.Request(url, headers=h)
with urllib.request.urlopen(req, timeout=45) as r:
rows = json.load(r)
link = r.headers.get("Link", "") or ""
out += [(x.get("tag_name") or "", (x.get("body") or "") + " " + (x.get("name") or ""))
for x in rows]
url = None
for part in link.split(","):
if 'rel="next"' in part:
url = part.split(";")[0].strip().strip("<>")
pages += 1
_RELS[owner_repo] = out
return out
def audit(recipe, vf, vt, images=None, label=""):
rep = A.scan(recipe, vf, vt, REG, images)
print(f"\n{'='*78}\n{recipe} {vf}{vt} {label}\n{'='*78}")
print(f"scanner count = {rep['cve_count_fixed']} known={rep['count_known']} "
f"failed_sources={rep['sources_failed']}")
counted = set(rep["fixed_by_this_upgrade"])
by_src = {}
for cve, e in rep["cves"].items():
by_src.setdefault(e["sources"][0], []).append((cve, e))
windows = rep["windows"]
total_mismatch = 0
for src, win in windows.items():
f, t = win["from"], win["to"]
method = rep["classified_by"][src]
rows = by_src.get(src, [])
scanner_here = {c for c, e in rows if c in counted}
print(f"\n ── {src} ({f}{t}) method={method} advisories={len(rows)}")
if "publish date" in method:
dw = rep["date_window"][src]
indep = {c for c, e in rows
if e.get("published_at") and dw["from"] < e["published_at"] <= dw["to"]}
print(f" date window {dw['from'][:10]}{dw['to'][:10]}")
else:
indep = {c for c, e in rows if in_window(f, t, e.get("patched"))}
# Independently redo the release-note method: fetch the repo's releases ourselves and
# confirm a tag NAMING the CVE really does fall inside (f, t].
kf, kt = sv(f), sv(t)
owner_repo = src.split(":", 1)[1]
rels = fetch_releases(owner_repo)
for c, e in rows:
if c in indep:
continue
naming = [tag for tag, body in rels if c in body]
if any(kf < sv(tag) <= kt for tag in naming if sv(tag)):
indep.add(c)
missed = indep - scanner_here
extra = scanner_here - indep
print(f" scanner counted {len(scanner_here)} | independent {len(indep)}"
f" | missed_by_scanner {len(missed)} | over_counted {len(extra)}")
if missed:
print(f" !! MISSED: {sorted(missed)}")
for c in sorted(missed):
print(f" {c} patched={dict(rows)[c].get('patched')!r}")
if extra:
print(f" !! OVER-COUNTED: {sorted(extra)}")
for c in sorted(extra):
print(f" {c} patched={dict(rows)[c].get('patched')!r}")
total_mismatch += len(missed) + len(extra)
# Anything counted that belongs to NO window would be a leak.
leaked = {c for c in counted if not any(s in rep["cves"][c]["sources"] for s in windows)}
if leaked:
print(f"\n !! COUNTED BUT OUTSIDE EVERY WINDOW: {sorted(leaked)}")
total_mismatch += len(leaked)
# Unclassified entries that belong to a WINDOWED source would mean a judged CVE was dropped.
dropped = [c for c in rep["unclassified"]
if any(s in rep["cves"][c]["sources"] for s in windows)]
if dropped:
print(f"\n !! UNCLASSIFIED DESPITE HAVING A WINDOW: {sorted(dropped)[:10]}")
total_mismatch += len(dropped)
unwindowed = {}
for cve in rep["unclassified"]:
unwindowed.setdefault(rep["cves"][cve]["sources"][0], []).append(cve)
if unwindowed:
print("\n unclassified by source (expected: images with no --image given):")
for s, cs in sorted(unwindowed.items()):
print(f" {len(cs):4d} {s}")
print(f"\n VERDICT: {'CLEAN' if total_mismatch == 0 else f'{total_mismatch} DISAGREEMENTS'}")
return total_mismatch, rep
if __name__ == "__main__":
bad = 0
bad += audit("gitea", "1.27.0", "1.27.1")[0]
bad += audit("discourse", "3.5.3", "2026.7.1")[0]
bad += audit("discourse", "3.5.3", "2026.7.1", [("redis", "7.4", "8.10")], "+redis sidecar")[0]
bad += audit("keycloak", "26.7.0", "26.7.1")[0]
bad += audit("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")], "+redis")[0]
bad += audit("n8n", "1.123.0", "2.18.1")[0]
print(f"\n\n{'#'*78}\nOVERALL: {'CLEAN — no disagreements' if bad == 0 else f'{bad} DISAGREEMENTS'}")
sys.exit(1 if bad else 0)
+314
View File
@@ -0,0 +1,314 @@
#!/usr/bin/env python3
"""audit-sources — are we still looking in the right place for each recipe's updates?
A recipe tracks an image repo and a set of registry URLs. Upstreams move: they rename the image,
switch registry, archive the GitHub repo, or split a community edition out of the original. When that
happens nothing errors — the old repo simply stops receiving tags, and the recipe looks "up to date"
forever while real releases happen somewhere else.
plausible is the worked example. It tracked `plausible/analytics` on Docker Hub; upstream moved to
`ghcr.io/plausible/community-edition`. The old repo still exists and still serves v2.0.0, so every
survey said "no upgrades available" while v3 shipped elsewhere.
This reports the signals that catch that, per image and per registry URL:
* IMAGE GONE QUIET — newest tag is older than --quiet-days (default 365). The single strongest
signal that releases moved somewhere else.
* DEPRECATION WORDING — the registry description says deprecated / moved / no longer maintained.
* GITHUB REPO ARCHIVED — upstream archived it.
* GITHUB REPO RENAMED — the API redirects to a different owner/name than we ask for.
* GITHUB REPO GONE — 404.
Everything is a SIGNAL, not a verdict: a genuinely stable image (mumble, custom-html) can be quiet
for good reason. The output is for a human to judge, so each finding says what was measured.
audit-sources.py [recipe ...] [--ssh HOST] [--quiet-days N] [--json]
"""
from __future__ import annotations
import argparse
import importlib.util
import json
import os
import re
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone
HERE = os.path.dirname(os.path.abspath(__file__))
_spec = importlib.util.spec_from_file_location("resolve_images", os.path.join(HERE, "resolve-images.py"))
RI = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(RI)
# advisory-scan supplies the source-fetching + changelog-attribution used by --security-sources
_aspec = importlib.util.spec_from_file_location("advisory_scan", os.path.join(HERE, "advisory-scan.py"))
A = importlib.util.module_from_spec(_aspec)
_aspec.loader.exec_module(A)
REGISTRY_DIR = os.environ.get("CCCI_UPSTREAM_REGISTRY", os.path.join(HERE, "upstream"))
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
DEPRECATION_RE = re.compile(
r"\b(deprecat|no longer maintain|unmaintained|superseded|moved to|migrated to|"
r"has moved|discontinued|end.of.life|archived)\b", re.I)
def _days_since(iso: str | None) -> int | None:
if not iso:
return None
try:
d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
except ValueError:
return None
return (datetime.now(timezone.utc) - d).days
def hub_repo_meta(repo: str) -> dict:
"""Docker Hub repo metadata: when it was last pushed to, and how it describes itself."""
try:
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}", RI._hub_auth())
except urllib.error.HTTPError as e:
return {"status": f"HTTP {e.code}"}
except Exception as e: # noqa: BLE001
return {"status": f"{type(e).__name__}"}
text = f"{d.get('description') or ''}\n{d.get('full_description') or ''}"
m = DEPRECATION_RE.search(text)
return {"status": "ok", "last_updated": d.get("last_updated"),
"deprecation_hint": (m.group(0) if m else None),
"archived": bool(d.get("is_archived") or d.get("status") == "inactive")}
def github_repo_meta(owner: str, repo: str) -> dict:
"""GitHub repo state — archived, renamed (the API answers with the CURRENT full_name), or gone."""
hdrs = {"Accept": "application/vnd.github+json"}
tok = RI._gh_token()
if tok:
hdrs["Authorization"] = f"Bearer {tok}"
try:
d = RI._json(f"https://api.github.com/repos/{owner}/{repo}", hdrs)
except urllib.error.HTTPError as e:
return {"status": f"HTTP {e.code}"}
except Exception as e: # noqa: BLE001
return {"status": f"{type(e).__name__}"}
asked, got = f"{owner}/{repo}".lower(), (d.get("full_name") or "").lower()
return {"status": "ok", "archived": bool(d.get("archived")), "pushed_at": d.get("pushed_at"),
"renamed_to": (d.get("full_name") if got and got != asked else None),
"description": d.get("description") or ""}
def newest_tag_date(registry: str, repo: str, tag: str) -> str | None:
"""When was the repo's newest same-shape tag pushed? Docker Hub only (it dates its tags)."""
if registry not in ("docker.io", "registry-1.docker.io"):
return None
try:
d = RI._json(f"https://hub.docker.com/v2/repositories/{repo}/tags"
f"?page_size=100&ordering=last_updated", RI._hub_auth())
except Exception: # noqa: BLE001
return None
want = RI.shape(tag)
for row in d.get("results", []):
if RI.shape(row.get("name") or "") == want:
return row.get("last_updated")
return (d.get("results") or [{}])[0].get("last_updated")
def security_source_audit(recipe: str) -> list[dict]:
"""Per source: are its CVEs USABLE, or merely visible?
The nginx lesson. nginx publishes no GitHub advisories; all its CVEs live in nginx.org/en/CHANGES.
The scan saw them and could do nothing with them, because nothing said which release fixed which
CVE — so every nginx bump in the fleet reported 0. Attribution (advisory-scan §2b) fixed that for
changelogs organised by release, but a page that lists CVEs with NO release structure is still a
blind spot: visible, uncountable. This finds those.
Per source: `advisory-feed` (structured, best), `changelog` (CVEs attributable to a release),
`unattributable` (CVEs present but no release structure — BLIND), or `no-cve-data`.
"""
urls, _ = _registry_urls(recipe)
out = []
# NVD CPE entries are a first-class source: for projects publishing nothing machine-readable
# (mattermost, mumble) they are the ONLY structured source, and omitting them here made two
# recipes look permanently blind after they had been fixed.
for key, cpe in A.registry_cpes(recipe, REGISTRY_DIR):
e = A.nvd_advisories(cpe, key)
n = len(e.get("advisories") or [])
out.append({"source": e["source"] + f" ({cpe.split(':')[4]}/{cpe.split(':')[3]})",
"kind": "advisory-feed" if n else "no-cve-data",
"status": e["status"], "cves": n, "usable": n})
for entry in A.github_advisories(urls):
out.append({"source": entry["source"], "kind": "advisory-feed",
"status": entry["status"], "cves": len(entry.get("advisories") or []),
"usable": len(entry.get("advisories") or [])})
for entry in A.vendor_pages(urls):
if entry["status"].startswith("skipped"):
continue
n = len(entry.get("cves") or [])
attributed = len(entry.get("fixed_in") or {})
kind = ("no-cve-data" if n == 0 else
"changelog" if attributed else "unattributable")
out.append({"source": entry["source"], "kind": kind, "status": entry["status"],
"cves": n, "usable": attributed})
return out
def audit_recipe(recipe: str, ssh: str | None, quiet_days: int) -> dict:
out = {"recipe": recipe, "findings": [], "images": [], "sources": []}
try:
refs = (RI.compose_images_ssh(recipe, ssh, "~/.abra/recipes") if ssh
else RI.compose_images(recipe, RI.RECIPE_DIR))
except Exception as e: # noqa: BLE001
out["findings"].append({"level": "error", "what": f"could not read compose: {e}"})
return out
for ref in refs:
if "${" in ref:
continue
info = RI.parse_ref(ref)
row = {"ref": ref, "registry": info["registry"], "repo": info["repo"], "tag": info["tag"]}
if info["registry"] in ("docker.io", "registry-1.docker.io"):
meta = hub_repo_meta(info["repo"])
row.update(meta)
newest = newest_tag_date(info["registry"], info["repo"], info["tag"])
row["newest_tag_pushed"] = newest
age = _days_since(newest)
row["newest_tag_age_days"] = age
if age is not None and age > quiet_days:
out["findings"].append({
"level": "warn", "what": "image has gone quiet",
"detail": f"{info['repo']}: newest {RI.shape(info['tag'])}-shaped tag pushed "
f"{age} days ago — releases may have moved elsewhere"})
if meta.get("deprecation_hint"):
out["findings"].append({
"level": "warn", "what": "registry text suggests deprecation",
"detail": f"{info['repo']}: says {meta['deprecation_hint']!r}"})
if meta.get("archived"):
out["findings"].append({"level": "warn", "what": "registry repo archived/inactive",
"detail": info["repo"]})
out["images"].append(row)
urls, reg_path = ([], None)
try:
urls, reg_path = _registry_urls(recipe)
except Exception: # noqa: BLE001
pass
if reg_path is None:
out["findings"].append({"level": "warn", "what": "no upstream registry file",
"detail": f"cc-ci-plan/upstream/{recipe}.md is missing — the advisory "
f"scan has nowhere to look"})
seen = set()
for u in urls:
m = re.match(r"https?://github\.com/([^/]+)/([^/#?]+)", u)
if not m:
continue
owner, repo = m.group(1), m.group(2).removesuffix(".git")
if (owner, repo) in seen:
continue
seen.add((owner, repo))
meta = github_repo_meta(owner, repo)
row = {"repo": f"{owner}/{repo}", **meta}
age = _days_since(meta.get("pushed_at"))
row["pushed_age_days"] = age
out["sources"].append(row)
if meta.get("status") != "ok":
out["findings"].append({"level": "warn", "what": "registry source unreachable",
"detail": f"{owner}/{repo}: {meta['status']}"})
continue
if meta.get("renamed_to"):
out["findings"].append({"level": "alert", "what": "GitHub repo has MOVED",
"detail": f"{owner}/{repo} now answers as {meta['renamed_to']}"})
if meta.get("archived"):
out["findings"].append({"level": "alert", "what": "GitHub repo is ARCHIVED",
"detail": f"{owner}/{repo} — upstream development has stopped here"})
if age is not None and age > quiet_days:
out["findings"].append({"level": "warn", "what": "GitHub repo quiet",
"detail": f"{owner}/{repo}: last push {age} days ago"})
return out
def _registry_urls(recipe: str):
path = os.path.join(REGISTRY_DIR, f"{recipe}.md")
if not os.path.exists(path):
return [], None
text = open(path).read()
urls = []
for u in re.findall(r"https?://[^\s)|\]]+", text):
u = u.rstrip("`'\"*.,;:>)")
if u and u not in urls:
urls.append(u)
return urls, path
def all_recipes() -> list[str]:
out = []
for ln in open(USED_RECIPES):
ln = ln.strip()
if not ln or ln.startswith("#") or ln.startswith("`"):
continue
parts = ln.split()
if len(parts) >= 2 and parts[1] in ("weekly", "external"):
out.append(parts[0])
return out
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("recipes", nargs="*")
ap.add_argument("--ssh", default=None)
ap.add_argument("--quiet-days", type=int, default=365)
ap.add_argument("--json", action="store_true")
ap.add_argument("--security-sources", action="store_true",
help="audit whether each recipe's CVE sources are USABLE (structured advisory "
"feed / release-attributable changelog) or merely visible")
a = ap.parse_args()
recipes = a.recipes or all_recipes()
if a.security_sources:
# What matters is whether the RECIPE can see CVEs at all — not whether some individual page
# is unparseable. A page with no release structure is harmless when the same project also
# publishes an advisory feed (redis, gitea, minio, clickhouse all do); it is only a blind
# spot when nothing else covers that project.
blind_recipes, noisy = [], 0
for r in recipes:
rows = security_source_audit(r)
feeds = [x for x in rows if x["kind"] == "advisory-feed" and x["cves"] > 0]
logs = [x for x in rows if x["kind"] == "changelog"]
unattr = [x for x in rows if x["kind"] == "unattributable"]
noisy += len(unattr)
usable = len(feeds) + len(logs)
if usable == 0:
blind_recipes.append(r)
print(f"!! {r}: NO USABLE CVE SOURCE — {len(unattr)} unparseable page(s), "
f"0 advisory feeds, 0 attributable changelogs")
for x in rows:
print(f" {x['kind']:15} {x['source'][:64]} ({x['cves']} CVEs)")
else:
print(f"OK {r}: {len(feeds)} advisory-feed(s), {len(logs)} changelog(s)"
+ (f", {len(unattr)} unparseable page(s) (redundant — covered by a feed)"
if unattr else ""))
for x in logs:
print(f" changelog {x['source'][:62]} ({x['usable']}/{x['cves']})")
print(f"\n{len(recipes)} recipes · {len(blind_recipes)} with NO usable CVE source"
+ (f": {', '.join(blind_recipes)}" if blind_recipes else "")
+ f" · {noisy} unparseable page(s) elsewhere (harmless where a feed covers them)")
return 0
reports = [audit_recipe(r, a.ssh, a.quiet_days) for r in recipes]
if a.json:
print(json.dumps(reports, indent=2))
return 0
alerts = 0
for rep in reports:
fs = rep["findings"]
mark = "OK " if not fs else ("!! " if any(f["level"] == "alert" for f in fs) else " ? ")
print(f"{mark} {rep['recipe']}")
for f in fs:
alerts += f["level"] == "alert"
print(f" [{f['level']}] {f['what']}: {f.get('detail','')}")
print(f"\n{len(reports)} recipes audited · "
f"{sum(len(r['findings']) for r in reports)} findings · {alerts} alerts")
return 0
if __name__ == "__main__":
sys.exit(main())
+32
View File
@@ -70,6 +70,34 @@ def start(mode, date):
log(f"{SESSION} busy with a report — leaving it"); return
log(f"{SESSION} exists (idle/leftover) — killing first"); kill_session(); time.sleep(1)
# Unique-name invariant (same as upgrader/supervisor): archive-rename every older session
# titled SESSION so the one this launch creates is the ONLY 'cc-ci-report' in the web UI.
# Archived names start with 'archive-' (operator convention 2026-08-04).
#
# AND clear the stale session PIN. The pin file is what the shared watchdog resolves via
# lu._session_id(); if a PREVIOUS run's pin survives, the watchdog inspects that old (already
# DONE_MARKER-bearing) session, declares "run completed" and exits within one poll — leaving
# the new run unwatched. That regression silently un-watchdogged the 2026-08-07 finish-run's
# report step (pin dated 2026-08-04). The pin is re-established after launch, below.
_lu = None
_prev_ids = set()
try:
os.environ["UPGRADER_SESSION"] = SESSION # scope the shared helpers to THIS session name
import importlib.util as _ilu
_spec = _ilu.spec_from_file_location(
"launch_upgrader", os.path.join(os.path.dirname(os.path.realpath(__file__)), "launch-upgrader.py"))
_lu = _ilu.module_from_spec(_spec); _spec.loader.exec_module(_lu)
_lu._archive_stale_titles(SESSION)
_rows = _lu._server_get("/session") or []
_rows = _rows if isinstance(_rows, list) else _rows.get("data", [])
_prev_ids = {s.get("id") for s in _rows}
try:
_lu.STATE_SID_FILE.unlink()
except OSError:
pass
except Exception as e:
log(f" (archive-rename/pin-clear skipped: {e})")
kf = Path(LOG_DIR) / f".kickoff-{SESSION}.txt"
kf.write_text(build_kickoff(date))
model_flag = f"--model '{MODEL}'" if MODEL else ""
@@ -90,6 +118,10 @@ def start(mode, date):
log(f"starting {SESSION} (backend={BACKEND}, tier={TIER}, model={MODEL}, date={date or 'today'})")
subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", cwd, cmd])
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_DIR}/{SESSION}.log'"])
if BACKEND == "opencode" and _lu is not None:
# Re-establish the pin on THIS run's session, so the watchdog (spawned below) resolves the
# new session rather than falling back to a title lookup or a stale pin.
_lu._pin_new_session(_prev_ids)
if BACKEND == "opencode":
if OPENCODE_SHARE:
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net +public --share link")
+24 -3
View File
@@ -98,6 +98,12 @@ def spawn_supervisor(sid, reason):
Path(LOG_DIR).mkdir(parents=True, exist_ok=True)
if _sup_alive():
_sup_kill(); time.sleep(1)
# Unique-name invariant (same as the upgrader): archive-rename every older session titled
# SUP_SESSION so the one this launch creates is the ONLY 'cc-ci-supervisor' in the web UI.
try:
lu._archive_stale_titles(SUP_SESSION)
except Exception:
pass
kf = Path(LOG_DIR) / f".kickoff-{SUP_SESSION}.txt"
kf.write_text(build_kickoff(sid, reason))
share = "--share" if OPENCODE_SHARE else ""
@@ -137,9 +143,24 @@ def _gate():
# a live `opencode run … -s <sid> --attach` proc, or a log touched within the stall window.
pids = lu._run_pids(sid)
idle = lu._session_idle_min()
if pids or (idle is not None and idle < lu.STALL_MIN):
via = f"{len(pids)} live run proc(s)" if pids else f"log idle {idle:.0f}m < {lu.STALL_MIN:.0f}m"
return False, sid, f"upgrader run progressing ({via}) — leaving it"
advancing = idle is not None and idle < lu.STALL_MIN
# PROGRESS REQUIRES THE SESSION TO BE ADVANCING — a live proc alone is not enough. A run walled
# by the provider keeps its process alive and spinning while emitting nothing (2026-08-07: 3
# days, zero output). Treating "proc exists" as progress is what deadlocked the gate.
if advancing:
return False, sid, f"upgrader run progressing (session advanced {idle:.0f}m ago) — leaving it"
if pids and lu._billing_blocked():
# NEVER kill it: it may resume when the wall lifts, and its context is the run's state.
# Surface it loudly instead — this is an operator-actionable condition, not self-healing.
return False, sid, (
f"run BLOCKED on a provider billing/usage wall ({len(pids)} proc(s) alive, session idle "
f"{idle:.0f}m) — NOT killing (resumable once the wall lifts); operator action required"
)
if pids:
log(
f"note: {len(pids)} live run proc(s) but session idle {idle:.0f}m ≥ "
f"{lu.STALL_MIN:.0f}m — treating as stalled, not progressing"
)
# The per-run watchdog owns PROMPT recovery (resume on proc-death/stall) and is the single writer
# while it lives. Defer to it — it gives up (exits its tmux) only after MAX_RESUMES fail, i.e. the
# run is stuck in a way a bare resume can't fix (e.g. disk-full). THEN the supervisor takes over.
+141 -11
View File
@@ -184,6 +184,18 @@ def start(mode="use-or-create"):
if SESSION == "cc-ci-upgrader":
prereclaim_cc_ci()
# Unique-name invariant: the run we are about to launch becomes THE 'cc-ci-upgrader' in the
# web UI; every older run gets an archive title. Also snapshot existing ids so the new
# session can be pinned unambiguously after launch.
_archive_stale_titles()
_prev_rows = _server_get("/session") or []
_prev_rows = _prev_rows if isinstance(_prev_rows, list) else _prev_rows.get("data", [])
_prev_ids = {s.get("id") for s in _prev_rows}
try:
STATE_SID_FILE.unlink()
except OSError:
pass
kf = Path(LOG_DIR) / f".kickoff-{SESSION}.txt"
kf.write_text(build_kickoff())
@@ -217,6 +229,8 @@ def start(mode="use-or-create"):
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION,
f"cat >> '{LOG_DIR}/{SESSION}.log'"])
log(f"started. attach: tmux attach -t {SESSION} log: {LOG_DIR}/{SESSION}.log")
if BACKEND == "opencode":
_pin_new_session(_prev_ids)
# For the opencode backend, spawn a watchdog that auto-resumes the run if the opencode
# usage-limit (429) stalls it mid-run (it does NOT self-resume). See watchdog().
if BACKEND == "opencode" and os.environ.get("UPGRADER_WATCHDOG", "1") == "1":
@@ -246,13 +260,95 @@ def _server_get(path):
except Exception:
return None
STATE_SID_FILE = Path(LOG_DIR) / f".{SESSION}-session-id"
def _server_patch(path, body):
req = _ureq.Request(OPENCODE_SERVER + path, method="PATCH",
headers={"Content-Type": "application/json"},
data=_json.dumps(body).encode())
try:
with _ureq.urlopen(req, timeout=15) as r:
return r.status
except Exception:
return None
def _db_created_ms(sid):
"""Session creation time from the opencode sqlite DB — the /session API rows carry NO
time fields, which is exactly how the 2026-08-04 watchdog resumed the WRONG (old, giant)
session: sorting on a missing key degraded to server list order."""
try:
import sqlite3
db = sqlite3.connect("file:" + os.path.expanduser(
"~/.local/share/opencode/opencode.db") + "?mode=ro", uri=True)
row = db.execute("SELECT time_created FROM session WHERE id=?", (sid,)).fetchone()
db.close()
return row[0] if row else 0
except Exception:
return 0
def _archive_stale_titles(title=None, label=None):
"""Rename every existing top-level session with the given canonical title to a dated
archive title, so EXACTLY ONE session ever carries the canonical name (the one the next
launch creates). Keeps the run trivially findable in the opencode web UI and makes the
title lookup in _session_id() unambiguous. Old runs stay browsable under
'<label> — <date>'. Also used by launch-supervisor.py for its own session name."""
title = title or SESSION
# Archive names always start with 'archive-' (operator convention 2026-08-04) so they
# sort/filter together in the web UI: 'archive-<original-title> — <date>'.
label = label or f"archive-{title}"
rows = _server_get("/session") or []
rows = rows if isinstance(rows, list) else rows.get("data", [])
for s in rows:
if s.get("title") == title and not (s.get("parentID") or s.get("parentId")):
created = _db_created_ms(s["id"])
d = datetime.fromtimestamp(created / 1000).strftime("%Y-%m-%d") if created else "unknown-date"
_server_patch(f"/session/{s['id']}", {"title": f"{label} {d}"})
log(f" archived old session {s['id'][:20]}'{label} {d}'")
def _session_id():
"""Newest top-level opencode session titled like SESSION (the run we manage)."""
"""The opencode session this launcher manages. Prefers the pinned id recorded at launch
(LOG_DIR/.{SESSION}-session-id) — title lookup is only the fallback, and thanks to
_archive_stale_titles() at most one top-level session carries the title. Never trust
server list order (see _db_created_ms)."""
try:
pinned = STATE_SID_FILE.read_text().strip()
# Validate directly — the /session LIST is paginated (~100 rows), so membership
# scans miss older/newer sessions; a direct GET is authoritative.
if pinned and (_server_get(f"/session/{pinned}") or {}).get("id") == pinned:
return pinned
except OSError:
pass
rows = _server_get("/session") or []
rows = rows if isinstance(rows, list) else rows.get("data", [])
cands = [s for s in rows if s.get("title") == SESSION and not (s.get("parentID") or s.get("parentId"))]
cands.sort(key=lambda s: (s.get("time") or {}).get("created") or 0, reverse=True)
return cands[0]["id"] if cands else None
cands.sort(key=lambda s: _db_created_ms(s.get("id")), reverse=True)
if cands:
try:
STATE_SID_FILE.write_text(cands[0]["id"])
except OSError:
pass
return cands[0]["id"]
return None
def _pin_new_session(prev_ids, wait_sec=45):
"""After launching a fresh run, discover the NEW top-level SESSION-titled session (one not
in prev_ids) and pin its id to the state file, so the watchdog can never grab an old one."""
deadline = _time.time() + wait_sec
while _time.time() < deadline:
rows = _server_get("/session") or []
rows = rows if isinstance(rows, list) else rows.get("data", [])
for s in rows:
if (s.get("title") == SESSION and not (s.get("parentID") or s.get("parentId"))
and s.get("id") not in prev_ids):
try:
STATE_SID_FILE.write_text(s["id"])
except OSError:
pass
log(f" pinned managed session id {s['id'][:20]}{STATE_SID_FILE.name}")
return s["id"]
_time.sleep(3)
log(" WARNING: could not discover the new session id to pin (title lookup remains the fallback)")
return None
def _session_idle_min():
"""Minutes since the managed run last ADVANCED — measured across the whole session TREE (the
@@ -321,25 +417,55 @@ def _limit_retry_after():
return 0
def _run_pids(sid=None):
"""PIDs of live `opencode run` procs for THIS session (via /proc scan — never matches self)."""
"""PIDs of live `opencode run` procs for THIS session (via /proc scan — never matches self).
Matches on FLAG VALUES (`--title <SESSION>` / `-s|--session <sid>`), never a substring of the
whole cmdline. The substring form caused the 2026-08-07 three-day deadlock: an agent's kickoff
PROMPT is passed as an argv element, and the supervisor's prompt text contains the literal
"cc-ci-upgrader", so the supervisor's own (billing-hung) agent matched as a live upgrader run —
the hourly gate then read its own corpse as "run progressing" and no-opped ~60 times."""
me, out = os.getpid(), []
for p in os.listdir("/proc"):
if not p.isdigit() or int(p) == me:
continue
try:
cl = open(f"/proc/{p}/cmdline", "rb").read().split(b"\0")
cl = [c for c in open(f"/proc/{p}/cmdline", "rb").read().split(b"\0") if c]
except Exception:
continue
joined = b" ".join(cl)
if not (b"opencode" in joined and b"run" in cl and b"--attach" in cl):
if not cl or b"opencode" not in cl[0] or b"run" not in cl or b"--attach" not in cl:
continue
# Scope to THIS managed session only: a fresh run carries `--title <SESSION>`, a resumed
# run carries `-s <sid>`. Without this, the report watchdog would kill the idle upgrader
# run (and vice-versa) since both are `opencode run … --attach`.
if SESSION.encode() in joined or (sid and sid.encode() in joined):
def _flag(names):
for i, a in enumerate(cl[:-1]):
if a in names:
return cl[i + 1]
return None
title = _flag((b"--title",))
s_val = _flag((b"-s", b"--session"))
if title == SESSION.encode() or (sid and s_val == sid.encode()):
out.append(int(p))
return out
def _billing_blocked(window_bytes=4000):
"""True when the tail of this run's log shows a provider billing/usage wall.
Such a process must NEVER be killed: it may resume when the wall lifts, and killing it can
lose in-flight work. It is also NOT progress — it can spin for days emitting nothing (observed
2026-08-07..10), so the gate reports it as BLOCKED and leaves recovery to the operator."""
try:
with open(LOG_FILE, "rb") as f:
f.seek(0, os.SEEK_END)
f.seek(max(0, f.tell() - window_bytes))
tail = f.read().decode(errors="replace").lower()
except OSError:
return False
return any(
s in tail
for s in ("spending limit", "insufficient balance", "usage limit", "usagelimiterror")
)
def _completed():
# Done only when the MODEL signs off with DONE_MARKER as its FINAL word: the marker in the LAST
# assistant TEXT (prose) message. This guards THREE false-positives that each abandoned a run:
@@ -399,6 +525,10 @@ def resume(reason="manual"):
f"--model '{MODEL}' {share} --attach '{OPENCODE_SERVER}' --dir '{WORKDIR}' \"$(cat '{kf}')\"")
subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", WORKDIR, cmd])
subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_FILE}'"])
try:
STATE_SID_FILE.write_text(sid) # a resume continues the SAME session — keep it pinned
except OSError:
pass
log(f"resume: relaunched {SESSION} (session {sid})")
# Every resume must be self-healing: ensure a watchdog is watching this run. Skip if one is
# already alive — notably when the watchdog ITSELF called resume (it lives in {SESSION}-watchdog),
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env python3
"""pr-survey — deterministic facts about every open recipe PR, for /cc-ci-cleanup to judge.
Open recipe PRs rot in specific, detectable ways. This gathers the evidence; it does NOT decide
anything — closing a PR is a judgement the skill makes, with these facts in hand.
RUN `reconcile-upstream.sh --all` FIRST. Every signal below is measured against the mirror's `main`,
and an unreconciled mirror makes all of them wrong: on 2026-08-11 three PRs (discourse #6 carrying
140 CVEs, keycloak #6 carrying 12, n8n #5) looked pending against a stale mirror while upstream had
already merged them. This tool refuses to guess about that — see `reconciled_recently`.
Per PR:
behind_main commits on main not in the branch — the "out of date" measure
ahead commits on the branch not on main
mergeable gitea's own verdict (false = conflicts, needs a rebase)
diff_files files the PR touches (0 = nothing left to merge)
adds_images the `+ image:` lines it introduces
already_in_main those `+ image:` lines ALREADY present in main -> the bump landed another way
obsolete true when every image it adds is already in main (nothing to contribute)
ci newest `!testme` verdict + build number parsed from the PR comments
branch_kind upgrade / fix / ci-artifact (`ci/*` sweep + probe branches) / other
age_days, stale_days (since last update)
pr-survey.py [recipe ...] [--json]
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timezone
HERE = os.path.dirname(os.path.abspath(__file__))
USED_RECIPES = os.path.join(HERE, "used-recipes.md")
TESTENV = os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")
NS = "recipe-maintainers"
def _env() -> dict:
e = {}
try:
for ln in open(TESTENV):
ln = ln.strip()
if "=" in ln and not ln.startswith("#"):
k, v = ln.split("=", 1)
e[k] = v.strip().strip('"').strip("'")
except OSError:
pass
return e
ENV = _env()
GITEA = os.environ.get("GITEA_URL") or ENV.get("GITEA_URL", "git.autonomic.zone")
_AUTH = base64.b64encode(
f"{os.environ.get('GITEA_USERNAME') or ENV.get('GITEA_USERNAME','')}:"
f"{os.environ.get('GITEA_PASSWORD') or ENV.get('GITEA_PASSWORD','')}".encode()
).decode()
def _get(path: str, raw: bool = False):
req = urllib.request.Request(
f"https://{GITEA}{path}",
headers={"Authorization": f"Basic {_AUTH}", "User-Agent": "cc-ci-pr-survey"},
)
with urllib.request.urlopen(req, timeout=60) as r:
body = r.read()
return body.decode(errors="replace") if raw else json.loads(body)
def _days(iso: str | None) -> int | None:
if not iso:
return None
try:
d = datetime.fromisoformat(iso.replace("Z", "+00:00"))
except ValueError:
return None
return (datetime.now(timezone.utc) - d).days
def _branch_kind(ref: str) -> str:
if ref.startswith("ci/"):
return "ci-artifact" # regall/cfold sweeps + testme probes; never meant to merge
if ref.startswith("upgrade"):
return "upgrade"
if re.match(r"^(fix|feat|chore|revert)", ref):
return "fix"
return "other"
def _main_images(recipe: str) -> set[str]:
"""Image refs pinned on the mirror's main — the baseline a PR is judged against."""
out = set()
for f in ("compose.yml",):
try:
txt = _get(f"/{NS}/{recipe}/raw/branch/main/{f}", raw=True)
except Exception: # noqa: BLE001
continue
for m in re.finditer(r"^\s*image:\s*[\"']?([^\"'\s]+)", txt, re.M):
out.add(m.group(1))
return out
def _ci_verdict(recipe: str, number: int) -> dict:
"""Newest cc-ci !testme outcome recorded on the PR."""
try:
cs = _get(f"/api/v1/repos/{NS}/{recipe}/issues/{number}/comments?limit=100")
except Exception: # noqa: BLE001
return {"verdict": "unknown", "build": None}
for c in reversed(cs):
b = c.get("body") or ""
if "cc-ci:testme" not in b:
continue
m = re.search(r"/cc-ci/(\d+)", b)
if "" in b or "passed" in b:
return {"verdict": "passed", "build": m.group(1) if m else None}
if "" in b or "failure" in b:
return {"verdict": "failed", "build": m.group(1) if m else None}
if "" in b or "in progress" in b:
return {"verdict": "running", "build": m.group(1) if m else None}
return {"verdict": "never-run", "build": None}
def survey_pr(recipe: str, pr: dict, main_images: set[str]) -> dict:
n = pr["number"]
head = pr["head"]["ref"]
row = {
"recipe": recipe, "number": n, "title": pr.get("title", ""), "head": head,
"url": pr.get("html_url"), "branch_kind": _branch_kind(head),
"age_days": _days(pr.get("created_at")), "stale_days": _days(pr.get("updated_at")),
"mergeable": pr.get("mergeable"),
}
try:
row["behind_main"] = _get(
f"/api/v1/repos/{NS}/{recipe}/compare/{urllib.parse.quote(head, safe='')}...main"
).get("total_commits", 0)
row["ahead"] = _get(
f"/api/v1/repos/{NS}/{recipe}/compare/main...{urllib.parse.quote(head, safe='')}"
).get("total_commits", 0)
except Exception: # noqa: BLE001
row["behind_main"], row["ahead"] = None, None
# A FAILED diff fetch must never look like an empty diff: gitea#4 404s on .diff (force-pushed
# branch) and would otherwise be flagged EMPTY-DIFF and closed — while being a verified, green,
# needed fix. Unknown is its own state.
diff = None
try:
body = _get(f"/{NS}/{recipe}/pulls/{n}.diff", raw=True)
if body.lstrip().startswith(("diff --git", "From ")) or not body.strip():
diff = body
except Exception: # noqa: BLE001
diff = None
row["diff_files"] = None if diff is None else len(re.findall(r"^diff --git ", diff, re.M))
adds = re.findall(r"^\+\s*image:\s*[\"']?([^\"'\s]+)", diff or "", re.M)
row["adds_images"] = sorted(set(adds))
row["already_in_main"] = sorted({i for i in set(adds) if i in main_images})
# Nothing left to contribute: it touches files but every image it introduces is already pinned.
# Only claim obsolete when the diff was actually READ. No diff, no verdict.
row["obsolete"] = diff is not None and bool(adds) and set(adds).issubset(main_images)
row["ci"] = _ci_verdict(recipe, n)
return row
def all_recipes() -> list[str]:
out = []
for ln in open(USED_RECIPES):
p = ln.split()
if len(p) >= 2 and not ln.startswith(("#", "`")) and p[1] in ("weekly", "external"):
out.append(p[0])
return out
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("recipes", nargs="*")
ap.add_argument("--json", action="store_true")
a = ap.parse_args()
rows = []
for r in (a.recipes or all_recipes()):
try:
prs = _get(f"/api/v1/repos/{NS}/{r}/pulls?state=open&limit=50")
except urllib.error.HTTPError as e:
rows.append({"recipe": r, "error": f"HTTP {e.code}"})
continue
if not prs:
continue
mi = _main_images(r)
for pr in prs:
rows.append(survey_pr(r, pr, mi))
if a.json:
print(json.dumps(rows, indent=2))
return 0
print(f"{len(rows)} open PR(s)\n")
for x in sorted(rows, key=lambda z: (z.get("recipe", ""), z.get("number", 0))):
if x.get("error"):
print(f" {x['recipe']}: {x['error']}")
continue
flags = []
if x["obsolete"]:
flags.append("OBSOLETE(images already in main)")
if x["branch_kind"] == "ci-artifact":
flags.append("CI-ARTIFACT")
if x["diff_files"] == 0:
flags.append("EMPTY-DIFF")
if x["diff_files"] is None:
flags.append("DIFF-UNREADABLE(do not close on this)")
if x["mergeable"] is False:
flags.append("CONFLICTS")
if (x["behind_main"] or 0) > 0:
flags.append(f"BEHIND-{x['behind_main']}")
print(f" {x['recipe']}#{x['number']:<3} {x['title'][:52]}")
print(f" {x['branch_kind']:12} age={x['age_days']}d idle={x['stale_days']}d "
f"ci={x['ci']['verdict']}({x['ci']['build'] or '-'}) files={x['diff_files'] if x['diff_files'] is not None else '?'}")
if x["adds_images"]:
print(f" adds: {', '.join(i.split('/')[-1] for i in x['adds_images'][:4])}")
if flags:
print(f" >> {' | '.join(flags)}")
return 0
if __name__ == "__main__":
sys.exit(main())
+45 -14
View File
@@ -9,7 +9,11 @@ Subcommands (the /recipe-report agent runs them around its own review/classifica
survey [DATE] JSON of the run + every recipe's open PRs + CI verdict + per-recipe upgrade
notes (breaking-change/CVE analysis), and the /upgrade-all summary.
render SPEC.json OUT.html render the agent's report spec -> a self-contained newspaper HTML page
publish OUT.html DATE copy to cc-ci:/var/lib/cc-ci-reports/week-DATE.html and regen the archive index
publish OUT.html DATE [KIND] copy to cc-ci:/var/lib/cc-ci-reports/<KIND>-DATE.html and regen the
archive index. KIND is `week` (default, the weekly /recipe-report) or `cve`
(a /cve-check advisory sweep). BOTH kinds appear in the SAME archive index,
newest first, each row suffixed "full" or "CVE check"; the distinct
filename prefix just stops a sweep overwriting a weekly edition.
Page order: short lead → the full wire table (priority-sorted, CVEs column) → Addendum → Security
Bulletin → per-recipe "What changed".
@@ -44,6 +48,10 @@ LOGDIR = "/srv/cc-ci/.cc-ci-logs"
TESTENV = "/srv/cc-ci/.testenv"
INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
HOST_REPORTS = "/var/lib/cc-ci-reports"
# Both kinds live in ONE archive, distinguished by a suffix on a common title.
# prefix -> (page title, index label)
KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),
"cve": ("The Recipe Report — CVE check", "{d} — CVE check")}
def _env():
@@ -215,8 +223,16 @@ def _table(rows, repo_url=None):
if repo_url and r.get("recipe") in repo_url:
name = f'<a href="{repo_url[r["recipe"]]}">{name}</a>'
cve = r.get("cve")
cve_cell = (f'<span class="cve">{int(cve)}</span>' if isinstance(cve, (int, float)) and cve
else '<span class="muted">none</span>')
# "?" = advisory scan absent or had failed sources → count NOT authoritative. Per the
# /recipe-report guardrail this must NEVER render as "none" (a blank-that-reads-clean is
# exactly how two CVSS-9.8 gitea RCEs were misreported as "none" on 2026-08-07). A positive
# int is the confirmed CVE count; 0/omit is a confirmed-clean scan.
if isinstance(cve, str) and cve.strip() == "?":
cve_cell = '<span class="muted" title="advisory scan incomplete or absent — CVE count unknown">?</span>'
elif isinstance(cve, (int, float)) and cve:
cve_cell = f'<span class="cve">{int(cve)}</span>'
else:
cve_cell = '<span class="muted">none</span>'
ci = _esc(r.get("ci"))
if r.get("ci_url"):
ci = f'<a href="{_esc(r["ci_url"])}">{ci}</a>'
@@ -270,8 +286,10 @@ def _mast():
def render(spec_path, out_path):
s = json.load(open(spec_path))
kind = s.get("kind", "week")
title = KINDS.get(kind, KINDS["week"])[0]
gen = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
sub = s.get("subtitle", "Week of " + s["date"])
sub = s.get("subtitle", ("Week of " if kind == "week" else "CVE check ") + s["date"])
lead = s.get("lead", "") or ""
# Auto-link recipe-name mentions in the lead to their mirror repos.
gitea = _env().get("GITEA_URL", "git.autonomic.zone")
@@ -285,7 +303,9 @@ def render(spec_path, out_path):
f'<span>report.ci.commoninternet.net</span><span>{gen}</span></div>'
f'<div class="lead">{lead}</div>')
# 1) the full wire — every recipe, in the agent's recommended priority order (CVEs first); CVEs column.
body += f'<h2>The full wire — every recipe, in priority order</h2>{_table(s.get("table"), repo_url)}'
wire = ("The full wire — every recipe, in priority order" if kind == "week"
else "Advisory sweep — every recipe, worst first")
body += f'<h2>{wire}</h2>{_table(s.get("table"), repo_url)}'
# 2) addendum — special issues to look into (normal-size header); omitted entirely if there are none.
add = [a for a in (s.get("addendum") or []) if str(a).strip()]
if add:
@@ -298,19 +318,30 @@ def render(spec_path, out_path):
# 4) what changed — a short section per recipe that has a PR
if s.get("changes"):
body += f'<h2>What changed</h2>{_changes(s.get("changes"), repo_url)}'
body += (f'<footer>The Recipe Report · generated {gen} · '
body += (f'<footer>{title} · generated {gen} · '
f'<a href="https://ci.commoninternet.net/">dashboard</a> · <a href="./">archive</a></footer>')
open(out_path, "w").write(_page("The Recipe Report — " + s["date"], body))
open(out_path, "w").write(_page(f"{title} · " + s["date"], body))
print("wrote", out_path)
def publish(html_path, date):
page = f"week-{date}.html"
def publish(html_path, date, kind="week"):
if kind not in KINDS:
print(f"unknown kind {kind!r}; expected one of {', '.join(KINDS)}"); sys.exit(2)
page = f"{kind}-{date}.html"
subprocess.run(["ssh", "cc-ci", f"cat > {HOST_REPORTS}/{page}"], input=open(html_path, "rb").read(), check=True)
listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/week-*.html 2>/dev/null"],
capture_output=True, text=True).stdout.split()
dates = sorted({os.path.basename(p)[5:-5] for p in listing}, reverse=True)
lis = "\n".join(f'<li><a href="week-{d}.html">Week of {d}</a><span class="d">{d}</span></li>' for d in dates)
# One index over BOTH families, newest first, each row labelled by its kind — an operator looking
# for "the latest security picture" should not have to know which skill produced which page.
entries = []
for k in KINDS:
listing = subprocess.run(["ssh", "cc-ci", f"ls -1 {HOST_REPORTS}/{k}-*.html 2>/dev/null"],
capture_output=True, text=True).stdout.split()
for pth in listing:
d = os.path.basename(pth)[len(k) + 1:-5]
if re.fullmatch(r"\d{4}-\d{2}-\d{2}", d):
entries.append((d, k))
lis = "\n".join(
f'<li><a href="{k}-{d}.html">{KINDS[k][1].format(d=d)}</a><span class="d">{d}</span></li>'
for d, k in sorted(set(entries), reverse=True))
idx = _page("The Recipe Report — Archive", _mast() +
'<div class="dateline"><span>Weekly review of Co-op Cloud recipe upgrades &amp; CI</span>'
'<span>report.ci.commoninternet.net</span></div>'
@@ -328,7 +359,7 @@ def main():
elif cmd == "render":
render(a[1], a[2])
elif cmd == "publish":
publish(a[1], a[2])
publish(a[1], a[2], a[3] if len(a) > 3 else "week")
else:
print(__doc__); sys.exit(2)
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env bash
# reconcile-upstream — sync recipe mirrors from TRUE upstream. Run this FIRST, always.
# ----------------------------------------------------------------------------------
# Every recipe we maintain is a MIRROR of a coopcloud recipe. Work done against a stale
# mirror is wasted or wrong, in three ways we have actually hit:
#
# 1. A PR whose changes upstream ALREADY MERGED. mailu #6 (2024.06.57 + redis 8.10,
# two internet-facing Roundcube CVEs) sat open and was reported as the fix for
# those CVEs — while upstream had merged and released it as 3.1.3+2024.06.57. The
# work was done; only our mirror was behind.
# 2. A survey that reads the stale mirror and reports "no upgrades available", so a
# recipe silently drops out of the weekly run.
# 3. Reading the WRONG BRANCH. Several coopcloud recipes keep a stale `main` beside
# the real default `master` — gitea's `main` is at 1.24.2-rootless while `master`
# has 1.27.1-rootless plus the merged PRs. Reading `main` there says the recipe is
# three releases behind and missing two CVSS-9.8 RCE fixes, which reads exactly
# like a real finding. open-recipe-pr.sh resolves the default branch itself
# (main OR master) — never hand-pick one.
#
# This is deterministic: it force-syncs each mirror's `main` to upstream's default
# branch and closes any mirror PR whose changes are already upstream. No AI judgement.
#
# reconcile-upstream.sh <recipe>... # specific recipes
# reconcile-upstream.sh --all # every recipe in used-recipes.md
#
# Safe to run repeatedly; a mirror already in sync is a no-op. Recipe work lives in
# BRANCHES, never on mirror `main`, so force-syncing `main` discards nothing.
set -o errexit -o nounset -o pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ORCH="$(dirname "$HERE")"
SSH="${SSH:-cc-ci}"
TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"
[ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
set -a; . "$TESTENV"; set +a
: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
if [ "${1:-}" = "--all" ]; then
mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
else
[ "$#" -gt 0 ] || { echo "usage: reconcile-upstream.sh <recipe>... | --all" >&2; exit 2; }
RECIPES=("$@")
fi
synced=0; closed=0; failed=0
for r in "${RECIPES[@]}"; do
echo "── $r"
if out="$(ssh "$SSH" "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s $r --reconcile-only" < "$RECONCILE" 2>&1)"; then
printf '%s\n' "$out" | grep -E "Force-syncing|already in sync|closed PR|still open|✓" | sed 's/^/ /' || true
synced=$((synced + 1))
closed=$((closed + $(printf '%s' "$out" | grep -c "closed PR" || true)))
else
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
echo " ✗ FAILED — do NOT proceed against this mirror until it reconciles"
failed=$((failed + 1))
fi
done
echo
echo "reconcile-upstream: ${synced} mirror(s) synced, ${closed} already-upstream PR(s) closed, ${failed} failed"
[ "$failed" -eq 0 ]
+459
View File
@@ -0,0 +1,459 @@
#!/usr/bin/env python3
"""resolve-images — what version is each of a recipe's images on, and what is newest?
An abra-independent version resolver. `abra recipe upgrade` is the normal path, but it has a hard
failure mode: an image pinned with BOTH a tag and a digest makes it FATA and abandon the WHOLE
recipe — even images it already parsed. immich pins two that way:
ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf6…
docker.io/valkey/valkey:9@sha256:3acc…
so immich contributes NO version data at all and silently drops out of every survey. That is
indistinguishable from "up to date" unless a human notices the missing row — which is exactly how it
kept getting skipped, and why a CVE sweep reported it as unknown.
This reads the compose files directly and queries the registries itself, so a digest pin is just a
digest pin. Output is JSON (default) or a table.
resolve-images.py <recipe> [--ssh HOST] [--recipe-dir DIR] [--table] [--only IMAGE]
The cc-ci host has no python3, so `--ssh cc-ci` reads the compose files from that host's checkout
over ssh and does the resolving locally. That keeps the source of truth the SAME tree abra and CI
use, rather than a second copy that can drift.
TAG SHAPES. Registries mix wildly different tag conventions in one repo, so "newest" is meaningless
without a shape. Each tag is reduced to a signature by replacing digit runs with '#':
v3.1.0 -> v#.#.#
1.27.1-rootless -> #.#.#-rootless
8.10-alpine -> #.#-alpine
14-vectorchord0.4.3-pgvectors0.2.0 -> #-vectorchord#.#.#-pgvectors#.#.#
Only tags sharing the CURRENT pin's shape are candidates. That keeps `-alpine` on `-alpine`, and
stops a `latest`/`release`/`sha-…` tag from ever being proposed as an upgrade.
TWO ANSWERS, NOT ONE. It reports `newest_same_shape` AND `newest_within_major` (same leading number).
For a plain app image they usually agree. For a compatibility-pinned sidecar they do not, and taking
the max would be wrong: immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
versions that immich-server is built against, so jumping pg major because a newer tag exists breaks
the deployment. The caller picks; this tool refuses to guess and shows both.
"""
from __future__ import annotations
import argparse
import glob
import shlex
import subprocess
import gzip
import json
import os
import re
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
UA = "cc-ci-resolve-images (+https://git.autonomic.zone/recipe-maintainers/cc-ci)"
TIMEOUT = int(os.environ.get("RESOLVE_IMAGES_TIMEOUT", "45"))
RECIPE_DIR = os.environ.get("ABRA_RECIPE_DIR", os.path.expanduser("~/.abra/recipes"))
MAX_TAG_PAGES = int(os.environ.get("RESOLVE_IMAGES_MAX_PAGES", "40"))
IMAGE_RE = re.compile(r"""^\s*image:\s*["']?([^"'\s]+)["']?\s*$""", re.M)
RETRIES = int(os.environ.get("RESOLVE_IMAGES_RETRIES", "4"))
def _fetch(url: str, headers: dict | None = None) -> bytes:
"""GET with backoff on rate limits.
Docker Hub throttles anonymous clients hard, and a sweep re-reads the same popular repos
(nginx, redis, postgres) for recipe after recipe. A 429 mid-sweep used to surface as
'unresolved', which is indistinguishable from a real lookup failure — so retry, and let the
per-repo cache below remove most of the requests entirely."""
h = {"User-Agent": UA, "Accept-Encoding": "gzip"}
h.update(headers or {})
delay = 2.0
for attempt in range(RETRIES):
try:
with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=TIMEOUT) as r:
raw = r.read()
if r.headers.get("Content-Encoding") == "gzip":
raw = gzip.decompress(raw)
return raw
except urllib.error.HTTPError as e:
if e.code in (429, 503) and attempt < RETRIES - 1:
time.sleep(delay)
delay *= 2
continue
raise
raise RuntimeError("unreachable")
_HUB_JWT: list = []
def _hub_auth() -> dict:
"""Authenticated Docker Hub calls get a far higher rate limit than anonymous ones.
Credentials come from /srv/cc-ci/.testenv (DOCKERHUB_USERNAME / DOCKERHUB_TOKEN), the same pair
the CI host already uses. Absent creds are fine — the sweep just runs anonymous and slower."""
if _HUB_JWT:
return _HUB_JWT[0]
env = {}
try:
for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
if "=" in ln and not ln.strip().startswith("#"):
k, v = ln.strip().split("=", 1)
env[k] = v.strip().strip("\"'")
except OSError:
pass
u = os.environ.get("DOCKERHUB_USERNAME") or env.get("DOCKERHUB_USERNAME")
t = os.environ.get("DOCKERHUB_TOKEN") or env.get("DOCKERHUB_TOKEN")
hdrs = {}
if u and t:
try:
body = json.dumps({"username": u, "password": t}).encode()
req = urllib.request.Request("https://hub.docker.com/v2/users/login",
data=body, method="POST",
headers={"Content-Type": "application/json", "User-Agent": UA})
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
tokj = json.load(r).get("token")
if tokj:
hdrs = {"Authorization": f"JWT {tokj}"}
except Exception: # noqa: BLE001 — anonymous is a valid fallback
hdrs = {}
_HUB_JWT.append(hdrs)
return hdrs
def _json(url: str, headers: dict | None = None):
return json.loads(_fetch(url, headers))
def shape(tag: str) -> str:
"""Signature of a tag with every digit run replaced by '#'. See module docstring."""
return re.sub(r"\d+", "#", tag)
def vkey(tag: str) -> tuple:
"""Ordering key: every number in the tag, in order. '1.27.10' > '1.27.9'; text ignored."""
return tuple(int(x) for x in re.findall(r"\d+", tag))
def parse_ref(ref: str) -> dict:
"""Split an image reference into registry / repo / tag / digest."""
digest = None
if "@" in ref:
ref, _, digest = ref.partition("@")
host, repo, tag = "docker.io", ref, "latest"
# A leading component is a REGISTRY only when there is a path after it. Without the slash test,
# a bare `postgres:15.18` looks like host "postgres:15.18" because of the tag's colon — which
# silently sent every library image to a nonexistent registry.
if "/" in ref:
first = ref.split("/")[0]
if "." in first or ":" in first or first == "localhost":
host, _, repo = ref.partition("/")
if ":" in repo.split("/")[-1]:
repo, _, tag = repo.rpartition(":")
if host == "docker.io" and "/" not in repo:
repo = f"library/{repo}" # bare `redis` is really `library/redis`
return {"registry": host, "repo": repo, "tag": tag, "digest": digest}
HUB_RECENT_PAGES = int(os.environ.get("RESOLVE_IMAGES_HUB_PAGES", "10"))
def _hub_tag_exists(repo: str, tag: str) -> bool:
try:
_json(f"https://hub.docker.com/v2/repositories/{repo}/tags/{tag}", _hub_auth())
return True
except Exception: # noqa: BLE001
return False
def _hub_tags(repo: str) -> list[str]:
"""Recently-pushed tags, newest first.
Popular Docker Hub repos carry many thousands of tags, so a full enumeration is impractical —
but it is also unnecessary: a tag NEWER than the one we run must have been pushed AFTER it, so
ordering by last_updated and reading a bounded recent window is sufficient to find any upgrade.
(ghcr offers no ordering, which is why that path needs a different strategy.)
"""
tags, url = [], (f"https://hub.docker.com/v2/repositories/{repo}/tags"
f"?page_size=100&ordering=last_updated")
auth = _hub_auth()
for _ in range(HUB_RECENT_PAGES):
d = _json(url, auth)
tags += [r["name"] for r in d.get("results", [])]
url = d.get("next")
if not url:
break
return tags
def _oci_bearer(host: str, repo: str) -> dict:
"""Token for an OCI registry, discovered from its own auth challenge.
Registries do NOT share a token endpoint. ghcr answers at /token?scope=…&service=ghcr.io, but
lscr.io and dock.mau.dev advertise different realms, and assuming ghcr's shape made both 401 —
which then read as "could not resolve" rather than "asked the wrong URL". The registry tells us
where to go in its WWW-Authenticate header; use that."""
try:
urllib.request.urlopen(
urllib.request.Request(f"https://{host}/v2/{repo}/tags/list?n=1",
headers={"User-Agent": UA}), timeout=TIMEOUT)
return {} # no auth needed
except urllib.error.HTTPError as e:
if e.code != 401:
return {}
chal = e.headers.get("WWW-Authenticate", "") or ""
except Exception: # noqa: BLE001
return {}
if not chal.lower().startswith("bearer"):
return {}
parts = dict(re.findall(r'(\w+)="([^"]*)"', chal))
realm = parts.get("realm")
if not realm:
return {}
q = {"service": parts.get("service", host), "scope": parts.get("scope", f"repository:{repo}:pull")}
url = realm + ("&" if "?" in realm else "?") + urllib.parse.urlencode(q)
try:
tok = (_json(url) or {}).get("token") or (_json(url) or {}).get("access_token")
return {"Authorization": f"Bearer {tok}"} if tok else {}
except Exception: # noqa: BLE001
return {}
def _oci_tags(host: str, repo: str) -> list[str]:
"""Tags from any OCI/v2 registry, with challenge-derived auth and Link pagination.
ghcr paginates hard — immich-server has >40,000 tags — and a truncated listing silently hides
the newest release line, so follow the cursor and let the caller's integrity check catch a read
that never reached the current pin."""
hdrs = _oci_bearer(host, repo)
tags, url = [], f"https://{host}/v2/{repo}/tags/list?n=1000"
for _ in range(MAX_TAG_PAGES):
req = urllib.request.Request(url, headers={"User-Agent": UA, **hdrs})
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
tags += (json.load(r) or {}).get("tags") or []
link = r.headers.get("Link", "") or ""
m = re.search(r'<([^>]+)>;\s*rel="next"', link)
if not m:
break
nxt = m.group(1)
url = f"https://{host}{nxt}" if nxt.startswith("/") else nxt
return tags
def _gh_token() -> str | None:
tok = os.environ.get("GITHUB_TOKEN")
if tok:
return tok.strip()
try:
return open(os.environ.get("GITHUB_TOKEN_FILE", "/srv/cc-ci/.github-token")).read().strip() or None
except OSError:
return None
def github_release_tags(owner: str, repo: str, max_pages: int = 4) -> list[str]:
"""Release tag names for a GitHub repo, newest first.
FALLBACK for registries whose tag listing cannot be enumerated. ghcr has no ordering and no
server-side filter, and immich-machine-learning carries >40,000 tags — a full read is impractical
and a partial read silently hides the newest release line. The project's RELEASES are ordered,
small, and authoritative: container tags track them. (The GitHub Packages API would answer this
directly but needs a scoped token; this scan's token deliberately has none.)
"""
hdrs = {"Accept": "application/vnd.github+json"}
tok = _gh_token()
if tok:
hdrs["Authorization"] = f"Bearer {tok}"
out = []
for page in range(1, max_pages + 1):
try:
rows = _json(f"https://api.github.com/repos/{owner}/{repo}/releases"
f"?per_page=100&page={page}", hdrs)
except Exception: # noqa: BLE001
break
if not rows:
break
out += [r.get("tag_name") or "" for r in rows]
return [t for t in out if t]
def _release_fallback_repos(registry: str, repo: str) -> list[tuple[str, str]]:
"""Candidate GitHub repos whose releases track this image's tags."""
if "ghcr.io" not in registry:
return []
parts = repo.split("/")
if len(parts) < 2:
return []
owner, name = parts[0], parts[-1]
cands = [(owner, name)]
# ghcr.io/immich-app/immich-machine-learning is built from immich-app/immich.
if name.startswith(owner.split("-")[0]):
cands.append((owner, owner.split("-")[0]))
return cands
_TAG_CACHE: dict[tuple[str, str], tuple[list[str], str | None]] = {}
def list_tags(registry: str, repo: str) -> tuple[list[str], str | None]:
if (registry, repo) in _TAG_CACHE:
return _TAG_CACHE[(registry, repo)]
res = _list_tags_uncached(registry, repo)
_TAG_CACHE[(registry, repo)] = res
return res
def _list_tags_uncached(registry: str, repo: str) -> tuple[list[str], str | None]:
try:
return (_hub_tags(repo) if registry in ("docker.io", "registry-1.docker.io")
else _oci_tags(registry, repo)), None
except urllib.error.HTTPError as e:
return [], f"HTTP {e.code}"
except Exception as e: # noqa: BLE001
return [], f"{type(e).__name__}: {e}"
def resolve(ref: str) -> dict:
"""Current pin -> newest same-shape tag, and newest within the current major."""
if "${" in ref or "$(" in ref:
# The tag is a compose variable (ghost pins `ghost:${IMAGE_VERSION}-alpine`). Its real value
# lives in .env, not here. Report it as skipped, never as a failed lookup.
return {**parse_ref(ref), "ref": ref, "shape": None, "candidates": 0,
"newest_same_shape": None, "newest_within_major": None,
"upgrade_available": False,
"status": "skipped: templated ref (tag comes from a compose variable)"}
info = parse_ref(ref)
out = {**info, "ref": ref, "shape": shape(info["tag"]), "status": "ok",
"newest_same_shape": None, "newest_within_major": None, "candidates": 0,
"upgrade_available": False}
tags, err = list_tags(info["registry"], info["repo"])
if err:
out["status"] = f"error: {err}"
return out
out["tags_seen"] = len(set(tags))
# INTEGRITY CHECK: the tag we are currently running MUST appear in the listing. If it does not,
# the listing is incomplete and any "newest" derived from it is a guess — ghcr paginates to tens
# of thousands of tags and a truncated read silently hides whole release lines. immich's
# machine-learning image is pinned v3.1.0, which EXISTS, yet a short read reported v1.134.0 as
# newest; without this check that becomes a confident, wrong answer.
if info["tag"] not in set(tags):
# Docker Hub: the window is recency-ordered, so the pin being outside it just means the pin
# is old — which is fine, because anything NEWER is necessarily inside the window. Confirm
# the pin genuinely exists (so a typo is still caught) and carry on.
if info["registry"] in ("docker.io", "registry-1.docker.io") and _hub_tag_exists(info["repo"], info["tag"]):
out["source"] = f"docker-hub:recent-{HUB_RECENT_PAGES * 100}"
tags = list(tags) + [info["tag"]]
else:
for owner, name in _release_fallback_repos(info["registry"], info["repo"]):
rel = github_release_tags(owner, name)
if info["tag"] in rel:
tags = rel
out["source"] = f"github-releases:{owner}/{name}"
out["tags_seen"] = len(set(rel))
break
else:
out["status"] = ("error: tag listing incomplete — the current pin "
f"{info['tag']!r} is absent from {len(set(tags))} registry tags "
f"and from the project's GitHub releases")
return out
want, cur = out["shape"], vkey(info["tag"])
same = [t for t in set(tags) if shape(t) == want and vkey(t)]
out["candidates"] = len(same)
if not same:
# Not a failure: digest-only pins and `latest`/`stable` have no comparable siblings.
out["status"] = "no comparable tags (shape has no numeric siblings)"
return out
newest = max(same, key=vkey)
out["newest_same_shape"] = newest
if cur:
within = [t for t in same if vkey(t)[:1] == cur[:1]]
if within:
out["newest_within_major"] = max(within, key=vkey)
out["upgrade_available"] = bool(cur and vkey(newest) > cur)
return out
def compose_images_ssh(recipe: str, host: str, recipe_dir: str) -> list[str]:
"""Same as compose_images, but the recipe tree lives on another host (cc-ci has no python3)."""
# NB: no shell-quoting of the directory — it may legitimately start with ~ or $HOME, and
# quoting it stops the remote shell expanding it, which yields an empty (and silent) result.
d = f"{recipe_dir}/{shlex.quote(recipe)}".replace("~", "$HOME")
cmd = (f'for f in {d}/compose*.yml; do case "$f" in *compose.ccci.yml) continue;; esac; '
f'[ -f "$f" ] && {{ cat "$f"; echo; }}; done; exit 0')
out = subprocess.run(["ssh", host, cmd], capture_output=True, text=True, timeout=120)
if out.returncode != 0:
raise RuntimeError(f"ssh {host}: {(out.stderr.strip() or 'no output')[:200]}")
if not out.stdout.strip():
raise RuntimeError(f"ssh {host}: no compose files found under {d}")
refs = []
for m in IMAGE_RE.finditer(out.stdout):
if m.group(1) not in refs:
refs.append(m.group(1))
return refs
def compose_images(recipe: str, recipe_dir: str) -> list[str]:
"""Every `image:` ref in the recipe's own compose files (the cc-ci overlay is NOT the recipe)."""
refs, base = [], os.path.join(recipe_dir, recipe)
for path in sorted(glob.glob(os.path.join(base, "compose*.yml"))):
if os.path.basename(path) == "compose.ccci.yml":
continue
try:
for m in IMAGE_RE.finditer(open(path).read()):
if m.group(1) not in refs:
refs.append(m.group(1))
except OSError:
continue
return refs
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("recipe")
ap.add_argument("--ssh", default=None, metavar="HOST",
help="read the recipe's compose files from HOST over ssh (e.g. --ssh cc-ci); "
"resolving still happens locally")
ap.add_argument("--recipe-dir", default=RECIPE_DIR)
ap.add_argument("--table", action="store_true", help="human-readable table instead of JSON")
ap.add_argument("--only", default=None, help="resolve just the images whose ref contains this")
a = ap.parse_args()
rdir = a.recipe_dir if a.recipe_dir != RECIPE_DIR or not a.ssh else "~/.abra/recipes"
refs = (compose_images_ssh(a.recipe, a.ssh, rdir) if a.ssh
else compose_images(a.recipe, a.recipe_dir))
if a.only:
refs = [r for r in refs if a.only in r]
results = [resolve(r) for r in refs]
report = {
"recipe": a.recipe,
"images": results,
"upgrades_available": [r["ref"] for r in results if r["upgrade_available"]],
"unresolved": [r["ref"] for r in results if r["status"].startswith("error")],
# The whole point: distinguish "checked, current" from "could not check".
"all_resolved": not any(r["status"].startswith("error") for r in results),
}
if not a.table:
print(json.dumps(report, indent=2))
return 0
print(f"{a.recipe}{len(results)} images")
for r in results:
flag = "UPGRADE" if r["upgrade_available"] else ("ERROR" if r["status"].startswith("error") else "current")
print(f" [{flag:7}] {r['repo']}:{r['tag']}" + (" (digest-pinned)" if r["digest"] else ""))
print(f" shape={r['shape']} candidates={r['candidates']}"
f" newest_same_shape={r['newest_same_shape']}"
f" newest_within_major={r['newest_within_major']}")
if r["status"] != "ok":
print(f" status: {r['status']}")
return 0
if __name__ == "__main__":
sys.exit(main())
+843
View File
@@ -0,0 +1,843 @@
#!/usr/bin/env python3
"""Tests for advisory-scan.py.
Two tiers:
OFFLINE (default) — pure logic, fixtures injected in place of the network. Fast, deterministic,
no token, no rate limit. These encode every classification rule and every guarantee the CVE count
makes, including the specific production defects that motivated them.
LIVE (--live) — re-derives the CVE counts published in the week-2026-08-07 report against the real
advisory APIs. Slow, needs network + ideally a GitHub token. Run before changing classification.
Usage:
python3 test-advisory-scan.py # offline only
python3 test-advisory-scan.py --live # offline + historic report regressions
"""
from __future__ import annotations
import importlib.util
import io
import json
import os
import pathlib
import sys
import unittest
import unittest.mock
HERE = pathlib.Path(__file__).resolve().parent
_spec = importlib.util.spec_from_file_location("advisory_scan", HERE / "advisory-scan.py")
A = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(A)
# ── fixture helpers ───────────────────────────────────────────────────────────────────────────────
def adv(cve, patched=None, published=None, severity="high", ghsa=None):
"""One GitHub advisory row as github_advisories() would emit it."""
return {"cve": cve, "ghsa": ghsa or f"GHSA-fake-{cve[-4:]}", "severity": severity,
"summary": f"summary for {cve}", "vulnerable_range": None, "patched": patched,
"url": None, "published_at": published}
def gh(owner_repo, advisories, status="ok"):
return {"source": f"github-advisories:{owner_repo}", "status": status, "advisories": advisories}
def vendor(url, cves=(), status="ok"):
return {"source": url, "status": status, "cves": list(cves),
"context": {c: f"...{c}..." for c in cves}}
def run_scan(gh_entries=(), vendor_entries=(), tag_dates=None, *, v_from=None, v_to=None,
images=None, recipe="fixture", urls=None, releases=None):
"""scan() with every network call replaced by fixtures.
`releases` maps CVE id -> tags whose release notes name it (the third, release-note method)."""
tag_dates = tag_dates or {}
releases = releases or {}
urls = urls if urls is not None else ["https://github.com/app/app"]
with unittest.mock.patch.object(A, "registry_urls", lambda r, d: (list(urls), "/fake/reg.md")), \
unittest.mock.patch.object(A, "github_advisories", lambda u: list(gh_entries)), \
unittest.mock.patch.object(A, "vendor_pages", lambda u: list(vendor_entries)), \
unittest.mock.patch.object(A, "osv", lambda r, v: None), \
unittest.mock.patch.object(A, "_tag_date", lambda o, r, v: tag_dates.get(v)), \
unittest.mock.patch.object(A, "release_fix_versions", lambda src, cve: list(releases.get(cve, []))):
return A.scan(recipe, v_from, v_to, "/fake", images)
def parse_image_args(argv):
"""Drive main()'s --image parsing exactly as the CLI does, returning the tuples scan() receives."""
captured = {}
def fake_scan(recipe, vf, vt, reg, images):
captured["images"] = images
return {"recipe": recipe, "from": vf, "to": vt, "registry": reg, "registry_urls": 0,
"sources": [], "cves": {}, "fixed_by_this_upgrade": [], "unclassified": [],
"sources_failed": [], "sources_benign": [], "count_known": True,
"cve_count_fixed": 0, "windows": {}, "classified_by": {}}
err = io.StringIO()
with unittest.mock.patch.object(A, "scan", fake_scan), \
unittest.mock.patch.object(sys, "argv", ["advisory-scan.py", *argv]), \
unittest.mock.patch.object(sys, "stdout", io.StringIO()), \
unittest.mock.patch.object(sys, "stderr", err):
A.main()
return captured["images"], err.getvalue()
# ── A. version ordering ───────────────────────────────────────────────────────────────────────────
class TestVersionKey(unittest.TestCase):
def test_strips_prefix_and_suffix(self):
self.assertEqual(A._vkey("v1.27.1"), (1, 27, 1))
self.assertEqual(A._vkey("1.27.1-rootless"), (1, 27, 1))
self.assertEqual(A._vkey("2024.06.55"), (2024, 6, 55))
def test_empty_and_none(self):
self.assertEqual(A._vkey(None), ())
self.assertEqual(A._vkey(""), ())
def test_dotted_minor_is_numeric_not_lexical(self):
# The bug this guards: "8.10" must be NEWER than "8.2.3". String compare says otherwise.
self.assertGreater(A._vkey("8.10"), A._vkey("8.2.3"))
self.assertGreater(A._vkey("1.27.10"), A._vkey("1.27.9"))
def test_shorter_prefix_orders_below_its_own_patch(self):
# 7.4 < 7.4.1, so a CVE patched in 7.4.1 IS fixed by moving off a bare 7.4 pin.
self.assertLess(A._vkey("7.4"), A._vkey("7.4.1"))
class TestWindowMembership(unittest.TestCase):
"""(from, to] membership — exclusive lower, inclusive upper, compared zero-padded."""
def _in(self, f, t, c):
return A._within(A._vkey(f), A._vkey(t), A._vkey(c))
def test_bounds(self):
self.assertTrue(self._in("1.27.0", "1.27.1", "1.27.1")) # upper inclusive
self.assertFalse(self._in("1.27.0", "1.27.1", "1.27.0")) # lower exclusive
self.assertFalse(self._in("1.27.0", "1.27.1", "1.26.9"))
self.assertFalse(self._in("1.27.0", "1.27.1", "1.28.0"))
def test_bare_major_upper_bound_includes_its_dot_zero(self):
# Regression: plain tuple order makes (18,) < (18,0), so a fix in 18.0 fell OUTSIDE a
# window ending at 18. Bare major tags are the norm for sidecars (postgres:18, redis:8).
self.assertTrue(self._in("17", "18", "18.0"))
self.assertTrue(self._in("7", "8", "8.0"))
self.assertTrue(self._in("7.4", "8.10", "8.0.4"))
def test_bare_major_upper_bound_excludes_later_patches(self):
# Conservative on the other side: nothing proves which 18.x a floating tag resolved to.
self.assertFalse(self._in("17", "18", "18.5"))
def test_bare_version_is_read_literally_as_dot_zero(self):
# from="8" means 8.0, so a fix in 8.0.4 is inside a window that ends at 9.
self.assertTrue(self._in("8", "9", "8.0.4"))
self.assertFalse(self._in("8", "9", "8.0")) # == the stated lower bound
def test_prefix_lower_bound_still_counts_its_patches(self):
self.assertTrue(self._in("7.4", "8.10", "7.4.1"))
self.assertTrue(self._in("7.4", "8.10", "7.4.6"))
def test_the_false_133_cve_stays_out(self):
self.assertFalse(self._in("7.4", "8.10", "6.0.11"))
# ── B. registry URL extraction ────────────────────────────────────────────────────────────────────
class TestRegistryUrls(unittest.TestCase):
def _write(self, tmp, text):
p = pathlib.Path(tmp) / "r.md"
p.write_text(text)
return A.registry_urls("r", tmp)
def test_strips_trailing_markdown_punctuation(self):
# Production defect: a captured backtick 404'd the fetch and rendered n8n/immich as '?'.
import tempfile
with tempfile.TemporaryDirectory() as tmp:
urls, _ = self._write(tmp, "see `https://docs.n8n.io/release-notes/` and "
"**https://example.com/sec.html**, plus https://a.test/x.")
self.assertIn("https://docs.n8n.io/release-notes/", urls)
self.assertIn("https://example.com/sec.html", urls)
self.assertIn("https://a.test/x", urls)
self.assertFalse([u for u in urls if u.endswith(("`", "*", ".", ","))])
def test_dedupes_and_reports_missing_registry(self):
import tempfile
with tempfile.TemporaryDirectory() as tmp:
urls, path = self._write(tmp, "https://a.test/x https://a.test/x")
self.assertEqual(urls.count("https://a.test/x"), 1)
self.assertTrue(path.endswith("r.md"))
urls, path = A.registry_urls("does-not-exist", "/nonexistent-dir")
self.assertEqual((urls, path), ([], None))
# ── C. --image argument parsing ───────────────────────────────────────────────────────────────────
class TestImageArgParsing(unittest.TestCase):
def test_single_and_repeated(self):
imgs, _ = parse_image_args(["r", "--image", "redis=7.4:8.10"])
self.assertEqual(imgs, [("redis", "7.4", "8.10")])
imgs, _ = parse_image_args(["r", "--image", "redis=7.4:8.10", "--image", "postgres=17:18"])
self.assertEqual(imgs, [("redis", "7.4", "8.10"), ("postgres", "17", "18")])
def test_malformed_is_skipped_with_a_warning_not_a_crash(self):
# It is an ADDITIVE pre-step: one typo must not abort the upgrade's scan step.
for bad in ("redis=7.4", "redis", "=7.4:8.10", "redis=:8.10", "redis=7.4:"):
imgs, err = parse_image_args(["r", "--image", bad])
self.assertEqual(imgs, [], f"{bad!r} should be rejected")
self.assertIn("malformed", err)
def test_good_and_bad_mixed_keeps_the_good(self):
imgs, err = parse_image_args(["r", "--image", "redis=7.4:8.10", "--image", "nope"])
self.assertEqual(imgs, [("redis", "7.4", "8.10")])
self.assertIn("malformed", err)
# ── D. classification ─────────────────────────────────────────────────────────────────────────────
class TestClassificationBoundaries(unittest.TestCase):
def _one(self, patched, v_from="1.27.0", v_to="1.27.1"):
rep = run_scan([gh("app/app", [adv("CVE-2026-0001", patched=patched)])],
v_from=v_from, v_to=v_to)
return rep
def test_patched_at_upper_bound_counts(self):
self.assertEqual(self._one("1.27.1")["fixed_by_this_upgrade"], ["CVE-2026-0001"])
def test_patched_at_lower_bound_does_not_count(self):
# Already fixed in the version we were ON — this upgrade did not fix it.
rep = self._one("1.27.0")
self.assertEqual(rep["fixed_by_this_upgrade"], [])
self.assertEqual(rep["cve_count_fixed"], 0)
def test_patched_below_and_above_window_do_not_count(self):
self.assertEqual(self._one("1.26.0")["fixed_by_this_upgrade"], [])
self.assertEqual(self._one("1.28.0")["fixed_by_this_upgrade"], [])
def test_any_of_several_patched_lines_counts(self):
# n8n regression: one advisory patches several release lines; reading only the first
# dropped the line the deployment was on (CVE-2026-42231/42232 misclassified).
rep = self._one("1.123.32; 2.17.4; 2.18.1", v_from="2.17.0", v_to="2.17.4")
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2026-0001"])
def test_no_patched_data_is_not_counted(self):
self.assertEqual(self._one(None)["fixed_by_this_upgrade"], [])
class TestPerImageWindows(unittest.TestCase):
"""The false-133 family of defects: an image must only ever be judged by its OWN versions."""
APP = gh("discourse/discourse", [adv("CVE-APP-0001", patched="3.5.4", published="2026-03-01T00:00:00Z")])
REDIS = gh("redis/redis", [
adv("CVE-2021-21309", patched="6.0.11", published="2021-02-01T00:00:00Z"),
adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2", published="2025-10-01T00:00:00Z",
severity="critical"),
])
URLS = ["https://github.com/discourse/discourse", "https://github.com/redis/redis"]
def test_sidecar_cve_is_not_judged_by_the_app_window(self):
# redis 6.0.11 sits numerically inside discourse 3.5.3 -> 2026.7.1. It must NOT count.
rep = run_scan([self.APP, self.REDIS], v_from="3.5.3", v_to="2026.7.1",
tag_dates={"3.5.3": "2025-12-30T00:00:00Z", "2026.7.1": "2026-07-31T00:00:00Z"},
urls=self.URLS)
self.assertNotIn("CVE-2021-21309", rep["fixed_by_this_upgrade"])
self.assertIn("CVE-2021-21309", rep["unclassified"])
def test_unwindowed_image_is_unclassified_never_counted(self):
rep = run_scan([self.APP, self.REDIS], v_from="3.5.3", v_to="3.5.4", urls=self.URLS)
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-APP-0001"])
for cve in ("CVE-2021-21309", "CVE-2025-49844"):
self.assertIn(cve, rep["unclassified"])
def test_sidecar_window_counts_only_what_that_bump_fixed(self):
rep = run_scan([self.APP, self.REDIS], v_from="3.5.3", v_to="3.5.4",
images=[("redis", "7.4", "8.10")], urls=self.URLS)
self.assertIn("CVE-2025-49844", rep["fixed_by_this_upgrade"]) # patched 7.4.6, in window
self.assertNotIn("CVE-2021-21309", rep["fixed_by_this_upgrade"]) # patched 6.0.11, below it
self.assertEqual(rep["cve_count_fixed"], 2) # app 1 + redis 1
def test_count_is_the_union_across_images(self):
rep = run_scan([self.APP, self.REDIS], v_from="3.5.3", v_to="3.5.4",
images=[("redis", "7.4", "8.10")], urls=self.URLS)
self.assertEqual(sorted(rep["fixed_by_this_upgrade"]), ["CVE-2025-49844", "CVE-APP-0001"])
def test_each_image_classified_independently(self):
# App crosses a scheme change (date method); redis does not (version method). Both resolve.
rep = run_scan([self.APP, self.REDIS], v_from="3.5.3", v_to="2026.7.1",
images=[("redis", "7.4", "8.10")], urls=self.URLS,
tag_dates={"3.5.3": "2025-12-30T00:00:00Z", "2026.7.1": "2026-07-31T00:00:00Z"})
methods = rep["classified_by"]
self.assertIn("publish date", methods["github-advisories:discourse/discourse"])
self.assertEqual(methods["github-advisories:redis/redis"], "patched version ranges")
self.assertTrue(rep["count_known"])
def test_image_name_matches_as_substring(self):
rep = run_scan([self.APP, gh("discourse/discourse-postgres", [adv("CVE-PG-1", patched="18.0")])],
v_from="3.5.3", v_to="3.5.4", images=[("postgres", "17", "18")],
urls=["https://github.com/discourse/discourse",
"https://github.com/discourse/discourse-postgres"])
self.assertIn("github-advisories:discourse/discourse-postgres", rep["windows"])
self.assertIn("CVE-PG-1", rep["fixed_by_this_upgrade"])
def test_primary_cannot_be_stolen_by_a_loose_image_name(self):
rep = run_scan([self.APP, gh("discourse/discourse-postgres", [adv("CVE-PG-1", patched="18.0")])],
v_from="3.5.3", v_to="3.5.4", images=[("discourse", "1", "2")],
urls=["https://github.com/discourse/discourse",
"https://github.com/discourse/discourse-postgres"])
self.assertEqual(rep["windows"]["github-advisories:discourse/discourse"],
{"from": "3.5.3", "to": "3.5.4"})
def test_unmatched_image_name_is_silently_ignored(self):
# Documents CURRENT behaviour: a typo'd name costs coverage without warning.
rep = run_scan([self.APP], v_from="3.5.3", v_to="3.5.4",
images=[("nosuchimage", "1", "2")], urls=self.URLS[:1])
self.assertEqual(list(rep["windows"]), ["github-advisories:discourse/discourse"])
self.assertTrue(rep["count_known"])
class TestSchemeChangeDateFallback(unittest.TestCase):
DATES = {"3.5.3": "2025-12-30T00:00:00Z", "2026.7.1": "2026-07-31T00:00:00Z"}
def _rep(self, advisories, dates=None):
return run_scan([gh("discourse/discourse", advisories)], v_from="3.5.3", v_to="2026.7.1",
tag_dates=self.DATES if dates is None else dates)
def test_counts_advisories_published_inside_the_date_window(self):
rep = self._rep([adv("CVE-IN-1", published="2026-03-01T00:00:00Z"),
adv("CVE-OUT-1", published="2025-06-01T00:00:00Z"),
adv("CVE-OUT-2", published="2026-09-01T00:00:00Z")])
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-IN-1"])
def test_date_boundaries_match_the_version_rule(self):
# Exclusive lower, inclusive upper — same as 4a, so the two methods agree at the edges.
rep = self._rep([adv("CVE-LOWER", published=self.DATES["3.5.3"]),
adv("CVE-UPPER", published=self.DATES["2026.7.1"])])
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-UPPER"])
def test_advisory_without_a_publish_date_is_not_counted(self):
self.assertEqual(self._rep([adv("CVE-NODATE", published=None)])["fixed_by_this_upgrade"], [])
def test_scheme_change_is_detected_not_version_compared(self):
rep = self._rep([adv("CVE-IN-1", patched="2026.1.0", published="2026-03-01T00:00:00Z")])
self.assertIn("publish date", rep["classified_by"]["github-advisories:discourse/discourse"])
self.assertIn("github-advisories:discourse/discourse", rep["date_window"])
def test_small_major_bump_still_uses_version_ranges(self):
rep = run_scan([gh("app/app", [adv("CVE-X", patched="3.0.0")])], v_from="2.9.0", v_to="3.0.0")
self.assertEqual(rep["classified_by"]["github-advisories:app/app"], "patched version ranges")
# ── E. count guarantees ───────────────────────────────────────────────────────────────────────────
class TestCountGuarantees(unittest.TestCase):
def test_unresolvable_window_yields_unknown_never_zero(self):
# Scheme change AND tag dates unresolvable -> must refuse to emit a number.
rep = run_scan([gh("app/app", [adv("CVE-1", published="2026-01-01T00:00:00Z")])],
v_from="3.5.3", v_to="2026.7.1", tag_dates={})
self.assertIs(rep["cve_count_fixed"], None)
self.assertFalse(rep["count_known"])
md = A.markdown(rep)
self.assertIn("UNKNOWN", md)
self.assertIn("NOT zero", md)
def test_one_unresolvable_image_makes_the_whole_count_unknown(self):
# A partial number would understate a security figure, so it is suppressed entirely.
rep = run_scan([gh("app/app", [adv("CVE-APP", patched="1.1")]),
gh("redis/redis", [adv("CVE-REDIS", patched="8.0")])],
v_from="1.0", v_to="1.1", images=[("redis", "7.4", "9999.1")],
tag_dates={}, urls=["https://github.com/app/app", "https://github.com/redis/redis"])
self.assertIs(rep["cve_count_fixed"], None)
self.assertFalse(rep["count_known"])
def test_genuine_zero_is_reported_as_zero(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="9.9.9")])], v_from="1.0", v_to="1.1")
self.assertEqual(rep["cve_count_fixed"], 0)
self.assertTrue(rep["count_known"])
self.assertIn("0 identified", A.markdown(rep))
def test_404_advisory_feed_is_benign_not_a_failure(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")]),
gh("side/car", [], status="no-advisories-published")],
v_from="1.0", v_to="1.1")
self.assertEqual(rep["sources_failed"], [])
self.assertIn("github-advisories:side/car", rep["sources_benign"])
self.assertEqual(rep["cve_count_fixed"], 1)
def test_template_url_is_benign_not_a_failure(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")])],
[vendor("https://x.test/changelog/v<VERSION>/", status="skipped: template URL")],
v_from="1.0", v_to="1.1")
self.assertEqual(rep["sources_failed"], [])
def test_real_source_failure_is_surfaced(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")], status="error: HTTP 500")],
v_from="1.0", v_to="1.1")
self.assertIn("github-advisories:app/app", rep["sources_failed"])
self.assertIn("FAILED", A.markdown(rep))
def test_no_window_given_classifies_nothing(self):
rep = run_scan([gh("app/app", [adv("CVE-1", patched="1.1")])])
self.assertEqual(rep["fixed_by_this_upgrade"], [])
self.assertIn("CVE-1", rep["unclassified"])
class TestVendorOnlyCves(unittest.TestCase):
"""The gitea case: CVEs named ONLY on a vendor page, absent from the GitHub advisory feed."""
def test_vendor_only_cve_is_recorded_and_surfaced(self):
rep = run_scan([gh("go-gitea/gitea", [])],
[vendor("https://blog.gitea.com/release-1.27.1/", ["CVE-2026-60004"])],
v_from="1.27.0", v_to="1.27.1")
self.assertIn("CVE-2026-60004", rep["cves"])
self.assertIn("CVE-2026-60004", rep["unclassified"])
def test_vendor_only_cve_is_NOT_counted_but_IS_sent_for_judgement(self):
# It carries no version data, so no arithmetic can place it — the deterministic count must
# not include it. It must not be silently dropped either: pass 2 gets it with its evidence.
rep = run_scan([gh("go-gitea/gitea", [])],
[vendor("https://blog.gitea.com/release-1.27.1/", ["CVE-2026-60004"])],
v_from="1.27.0", v_to="1.27.1")
self.assertEqual(rep["fixed_by_this_upgrade"], [])
self.assertEqual(rep["cve_count_fixed"], 0)
self.assertIn("CVE-2026-60004", A.needs_judgement(rep))
def test_cve_in_both_vendor_and_advisory_feed_is_counted_once(self):
rep = run_scan([gh("go-gitea/gitea", [adv("CVE-2026-60004", patched="1.27.1")])],
[vendor("https://blog.gitea.com/x/", ["CVE-2026-60004"])],
v_from="1.27.0", v_to="1.27.1")
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2026-60004"])
self.assertEqual(rep["cve_count_fixed"], 1)
self.assertEqual(len(rep["cves"]["CVE-2026-60004"]["sources"]), 2)
class TestIndeterminateBucket(unittest.TestCase):
"""An advisory with no knowable fix version is neither counted nor dismissed."""
def test_tbd_patched_is_indeterminate_not_excluded(self):
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
self.assertEqual(rep["fixed_by_this_upgrade"], [])
self.assertIn("CVE-TBD", rep["indeterminate"])
self.assertEqual(rep["cve_count_indeterminate"], 1)
def test_placeholder_patched_is_indeterminate(self):
# "7.4.X" could be 7.4.1 — inside the window. Extracting a bare 7.4 and excluding it was
# how CVE-2024-46981 (high) went missing.
rep = run_scan([gh("redis/redis", [adv("CVE-X", patched="6.2.X, 7.2.X, 7.4.X")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
self.assertIn("CVE-X", rep["indeterminate"])
def test_real_versions_outside_the_window_are_decided_not_indeterminate(self):
rep = run_scan([gh("redis/redis", [adv("CVE-OLD", patched="6.0.11")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
self.assertEqual(rep["indeterminate"], [])
self.assertEqual(rep["cve_count_fixed"], 0)
def test_indeterminate_is_surfaced_in_the_markdown_and_not_read_as_zero(self):
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD", severity="critical")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
md = A.markdown(rep)
self.assertIn("could NOT be judged", md)
self.assertIn("must NOT be read as unaffected", md)
class TestReleaseNoteResolution(unittest.TestCase):
"""Third method: a release whose notes NAME the CVE supplies the fix version the advisory lacks."""
def test_release_naming_the_cve_inside_the_window_counts_it(self):
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
releases={"CVE-TBD": ["6.2.19", "7.2.10", "7.4.5", "8.0.3"]})
self.assertIn("CVE-TBD", rep["fixed_by_this_upgrade"])
self.assertEqual(rep["indeterminate"], [])
self.assertEqual(rep["resolved_by_release_notes"]["CVE-TBD"], ["7.4.5", "8.0.3"])
def test_naming_releases_all_below_the_window_means_ALREADY_fixed(self):
# Every known fix predates the version we were already on, so this upgrade did not deliver
# it. That is a DECISION, not an unknown — mailu's redis 8.8.0 → 8.10.0 crosses 12 such
# advisories, and calling them "could not judge" overstates the uncertainty.
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
releases={"CVE-TBD": ["6.2.19"]})
self.assertEqual(rep["fixed_by_this_upgrade"], [])
self.assertEqual(rep["indeterminate"], [])
self.assertIn("CVE-TBD", rep["already_fixed_before_upgrade"])
self.assertIn("outside-window", rep["cves"]["CVE-TBD"]["classification"])
def test_naming_releases_only_ABOVE_the_window_stays_indeterminate(self):
# The fix landed after our target, so we are still exposed. Deliberately NOT decided as a
# tidy "not fixed": it is an open vulnerability and must stay visible to the operator.
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
releases={"CVE-TBD": ["9.0.0"]})
self.assertEqual(rep["fixed_by_this_upgrade"], [])
self.assertIn("CVE-TBD", rep["indeterminate"])
def test_vendor_page_cve_on_the_same_repo_uses_release_notes(self):
# mailu announces its Roundcube CVEs only on github.com/Mailu/Mailu/releases. Requiring an
# advisory feed sent a deterministic case to pass 2; it is now decided in pass 1.
rep = run_scan([gh("Mailu/Mailu", [])],
[vendor("https://github.com/Mailu/Mailu/releases", ["CVE-2026-54432"])],
v_from="2024.06.55", v_to="2024.06.57",
urls=["https://github.com/Mailu/Mailu"],
releases={"CVE-2026-54432": ["2024.06.56"]})
self.assertIn("CVE-2026-54432", rep["fixed_by_this_upgrade"])
self.assertEqual(rep["cve_count_fixed"], 1)
def test_release_evidence_is_recorded_for_audit(self):
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
releases={"CVE-TBD": ["7.4.5"]})
e = rep["cves"]["CVE-TBD"]
self.assertEqual(e["fix_versions_from_release_notes"], ["7.4.5"])
self.assertIn("named in release notes", e["classification"])
def test_it_does_not_override_a_version_range_decision(self):
# A CVE already counted by patched ranges is untouched; the method only rescues undecided.
rep = run_scan([gh("redis/redis", [adv("CVE-OK", patched="7.4.1")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"],
releases={"CVE-OK": ["7.4.1"]})
self.assertNotIn("CVE-OK", rep.get("resolved_by_release_notes") or {})
class TestReleaseLineSemantics(unittest.TestCase):
"""A fix inside the numeric window is not a fix on the branch you actually land on."""
def test_fix_later_on_the_targets_own_line_is_not_counted(self):
# ClickHouse fixed CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20. Landing on 23.10.4.25 crosses
# the 23.9 fix numerically but is BELOW its own line's fix, so it does not have it.
rep = run_scan([gh("ClickHouse/ClickHouse",
[adv("CVE-2023-48704", patched="v23.10.5.20; v23.9.6.20; v23.8.8.20")])],
v_from="23.4.2.11", v_to="23.10.4.25",
urls=["https://github.com/ClickHouse/ClickHouse"])
self.assertEqual(rep["fixed_by_this_upgrade"], [])
def test_fix_earlier_on_the_targets_own_line_is_counted(self):
rep = run_scan([gh("ClickHouse/ClickHouse",
[adv("CVE-2023-47118", patched="v23.10.2.13; v23.8.6.16")])],
v_from="23.4.2.11", v_to="23.10.4.25",
urls=["https://github.com/ClickHouse/ClickHouse"])
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-47118"])
def test_fix_exactly_at_the_target_is_counted(self):
rep = run_scan([gh("ClickHouse/ClickHouse",
[adv("CVE-2023-48298", patched="v23.10.4.25; v23.9.5.29")])],
v_from="23.4.2.11", v_to="23.10.4.25",
urls=["https://github.com/ClickHouse/ClickHouse"])
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2023-48298"])
def test_no_fix_on_the_target_line_falls_back_to_the_window(self):
# redis fixes 7.4.6/8.0.4/8.2.2 with no 8.10.x entry; landing on 8.10 still has them,
# because nothing on the 8.10 line is named as a LATER fix.
rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.0.4; 8.2.2")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2025-49844"])
class TestAlreadyFixedOnFromLine(unittest.TestCase):
"""A fix that landed on the line we upgrade FROM was already ours before the upgrade."""
def test_backport_to_our_own_line_is_not_credited(self):
# mattermost patches every maintained line at once. 10.11.22 -> 10.12.4 crosses 10.12.1, but
# 10.11.22 is already past 10.11.4, so the deployment HAD the fix. Counting it credits the
# upgrade with work it did not do.
rep = run_scan([gh("mattermost/mattermost",
[adv("CVE-1", patched="10.11.4; 10.12.1; 10.5.12")])],
v_from="10.11.22", v_to="10.12.4",
urls=["https://github.com/mattermost/mattermost"])
self.assertEqual(rep["fixed_by_this_upgrade"], [])
def test_a_fix_ABOVE_our_position_on_the_same_line_still_counts(self):
rep = run_scan([gh("mattermost/mattermost", [adv("CVE-2", patched="10.11.30; 10.12.1")])],
v_from="10.11.22", v_to="10.12.4",
urls=["https://github.com/mattermost/mattermost"])
self.assertEqual(rep["fixed_by_this_upgrade"], ["CVE-2"])
def test_placeholders_never_feed_this_rule(self):
# "7.4.X" parses to a bare 7.4, which would read as "already fixed at 7.4" and silently drop
# a real fix — this is exactly how redis CVE-2024-46981 was lost when the rule was added.
rep = run_scan([gh("redis/redis", [adv("CVE-3", patched="6.2.X, 7.2.X, 7.4.X")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
self.assertIn("CVE-3", rep["indeterminate"])
self.assertEqual(rep["fixed_by_this_upgrade"], [])
class TestChangelogAttribution(unittest.TestCase):
"""Projects that publish no advisory feed still say which release fixed what — in their changelog."""
CHANGES = """
Changes with nginx 1.31.3 11 Aug 2026
*) Security: a flaw ... (CVE-2026-60005)
*) Security: another ... (CVE-2026-56434)
Changes with nginx 1.31.2 04 Aug 2026
*) Security: something ... (CVE-2026-48142)
Changes with nginx 1.31.1 21 Jul 2026
*) Security: older ... (CVE-2026-9256)
Changes with nginx 1.20.0 01 Jan 2021
*) Security: ancient ... (CVE-2013-2028)
"""
def test_each_cve_is_attributed_to_the_release_that_fixed_it(self):
got = A._changelog_versions(self.CHANGES)
self.assertEqual(got["CVE-2026-60005"], "1.31.3")
self.assertEqual(got["CVE-2026-48142"], "1.31.2")
self.assertEqual(got["CVE-2026-9256"], "1.31.1")
self.assertEqual(got["CVE-2013-2028"], "1.20.0")
def _scan(self, wfrom, wto):
# nginx publishes NO GitHub advisories — the feed is empty and the changelog is everything.
return run_scan(
[gh("nginx/nginx", [])],
[{"source": "https://nginx.org/en/CHANGES", "status": "ok",
"cves": sorted(A._changelog_versions(self.CHANGES)),
"context": {}, "fixed_in": A._changelog_versions(self.CHANGES)}],
images=[("nginx", wfrom, wto)], urls=["https://github.com/nginx/nginx"])
def test_window_counts_only_the_releases_it_crosses(self):
rep = self._scan("1.31.1", "1.31.3") # 1.31.1 is the FROM, so its CVE is already fixed
self.assertEqual(set(rep["fixed_by_this_upgrade"]),
{"CVE-2026-48142", "CVE-2026-56434", "CVE-2026-60005"})
def test_a_narrower_window_counts_fewer(self):
rep = self._scan("1.31.2", "1.31.3")
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-56434", "CVE-2026-60005"})
def test_ancient_entries_are_not_swept_in(self):
# The changelog lists the project's whole history; only the crossed releases may count.
rep = self._scan("1.31.1", "1.31.3")
self.assertNotIn("CVE-2013-2028", rep["fixed_by_this_upgrade"])
def test_evidence_is_recorded(self):
rep = self._scan("1.31.1", "1.31.3")
self.assertEqual(rep["resolved_by_changelog"]["CVE-2026-60005"], "1.31.3")
class TestComposeDerivedWindows(unittest.TestCase):
"""Windows read off a compose diff, so nobody has to remember which --image args an upgrade needs."""
OLD = """
services:
app:
image: "plausible/analytics:v2.0.0"
db:
image: pgautoupgrade/pgautoupgrade:18-alpine
plausible_events_db:
image: clickhouse/clickhouse-server:23.4.2.11-alpine
volumes:
data:
"""
NEW = """
services:
app:
image: "ghcr.io/plausible/community-edition:v3.2.1"
db:
image: pgautoupgrade/pgautoupgrade:18-alpine
plausible_events_db:
image: clickhouse/clickhouse-server:24.12-alpine
volumes:
data:
"""
def _windows(self, old=None, new=None):
pages = {"to": new if new is not None else self.NEW,
"from": old if old is not None else self.OLD}
with unittest.mock.patch.object(A, "_fetch", lambda u, h=None: pages["to" if "to" in u else "from"]), \
unittest.mock.patch.object(A, "_gitea_auth", lambda u: {}):
return A.windows_from_compose("http://x/to", "http://x/from")[0]
def test_app_service_leads_and_sidecars_follow(self):
w = self._windows()
self.assertEqual(w[0], ("community-edition", "v2.0.0", "v3.2.1"))
self.assertIn(("clickhouse-server", "23.4.2.11-alpine", "24.12-alpine"), w)
def test_unchanged_images_are_not_windows(self):
# pgautoupgrade is identical in both; inventing a window for it would be a false count.
self.assertNotIn("pgautoupgrade", [n for n, _, _ in self._windows()])
def test_a_changed_image_REPO_is_still_the_same_service(self):
# plausible/analytics -> ghcr.io/plausible/community-edition. Keyed by image repo this reads
# as one image vanishing and another appearing, and the app window is lost entirely.
w = self._windows()
self.assertTrue(any(n == "community-edition" and f == "v2.0.0" for n, f, _ in w))
def test_no_change_yields_no_windows(self):
self.assertEqual(self._windows(old=self.NEW, new=self.NEW), [])
def test_templated_tags_are_skipped(self):
new = self.NEW.replace('ghcr.io/plausible/community-edition:v3.2.1', 'ghost:${IMAGE_VERSION}')
self.assertNotIn("ghost", [n for n, _, _ in self._windows(new=new)])
class TestImageNameMatching(unittest.TestCase):
"""An image name and its advisory source rarely spell each other exactly."""
def test_matches_when_the_image_name_is_LONGER_than_the_source(self):
# clickhouse/clickhouse-server vs source ClickHouse/ClickHouse — one-directional matching
# dropped this window silently when the key came from a compose file.
rep = run_scan([gh("ClickHouse/ClickHouse", [adv("CVE-1", patched="23.10.2.13")])],
images=[("clickhouse-server", "23.4.2.11", "24.12")],
urls=["https://github.com/ClickHouse/ClickHouse"])
self.assertIn("github-advisories:ClickHouse/ClickHouse", rep["windows"])
self.assertEqual(rep["cve_count_fixed"], 1)
def test_matches_when_the_image_name_is_SHORTER_than_the_source(self):
rep = run_scan([gh("redis/redis", [adv("CVE-2", patched="7.4.1")])],
images=[("redis", "7.4", "8.10")], urls=["https://github.com/redis/redis"])
self.assertEqual(rep["cve_count_fixed"], 1)
class TestAdjudicationEvidenceAssembly(unittest.TestCase):
"""Pass 2's JUDGEMENT is a model's and not testable; what IS testable is what it gets shown."""
def test_selects_indeterminate_and_vendor_only_cases(self):
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
[vendor("https://blog.test/sec", ["CVE-VENDOR"])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
todo = A.needs_judgement(rep)
self.assertIn("CVE-TBD", todo)
self.assertIn("CVE-VENDOR", todo)
def test_does_not_re_ask_about_cases_pass_1_settled(self):
rep = run_scan([gh("redis/redis", [adv("CVE-OK", patched="7.4.1")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
self.assertNotIn("CVE-OK", A.needs_judgement(rep))
def test_evidence_bundle_carries_the_window_and_published_fields(self):
rep = run_scan([gh("redis/redis", [adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
with unittest.mock.patch.object(A, "advisory_text", lambda g, s=None: {"status": "skipped"}), \
unittest.mock.patch.object(A, "release_fix_versions", lambda s, c: []):
ev = A.evidence_bundle(rep, "CVE-TBD")
self.assertEqual(ev["window"], {"from": "7.4", "to": "8.10"})
self.assertEqual(ev["patched_as_published"], "TBD")
self.assertIn("no fix version", ev["why_undecided"])
def test_pass_1_decisions_are_included_for_review(self):
rep = run_scan([gh("redis/redis", [adv("CVE-OK", patched="7.4.1"),
adv("CVE-OLD", patched="6.0.11"),
adv("CVE-TBD", patched="TBD")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
with unittest.mock.patch.object(A, "advisory_text", lambda g, s=None: {"status": "skipped"}), \
unittest.mock.patch.object(A, "release_fix_versions", lambda s, c: []):
block = A.adjudication_block(rep)
self.assertIn("Pass 1 decisions", block)
self.assertIn("CVE-OK", block) # counted
self.assertIn("CVE-OLD", block) # excluded as outside-window
self.assertIn("CVE-TBD", block) # needs judgement
def test_truncation_is_announced_never_silent(self):
advs = [adv(f"CVE-2026-{1000+i}", patched="TBD") for i in range(30)]
rep = run_scan([gh("redis/redis", advs)], v_from="7.4", v_to="8.10",
urls=["https://github.com/redis/redis"])
with unittest.mock.patch.object(A, "advisory_text", lambda g, s=None: {"status": "skipped"}), \
unittest.mock.patch.object(A, "release_fix_versions", lambda s, c: []), \
unittest.mock.patch.object(A, "MAX_ADJUDICATE", 5):
block = A.adjudication_block(rep)
self.assertIn("not shown", block)
self.assertIn("do not", block.lower())
class TestMarkdownOutput(unittest.TestCase):
def test_lists_every_window_with_its_method(self):
rep = run_scan([gh("discourse/discourse", [adv("CVE-A", patched="3.5.4")]),
gh("redis/redis", [adv("CVE-B", patched="8.0")])],
v_from="3.5.3", v_to="3.5.4", images=[("redis", "7.4", "8.10")],
urls=["https://github.com/discourse/discourse", "https://github.com/redis/redis"])
md = A.markdown(rep)
self.assertIn("discourse/discourse: 3.5.3 → 3.5.4", md)
self.assertIn("redis/redis: 7.4 → 8.10", md)
self.assertIn("**CVEs fixed by this upgrade: 2**", md)
def test_severity_and_fixed_in_are_rendered(self):
rep = run_scan([gh("redis/redis", [adv("CVE-2025-49844", patched="7.4.6; 8.2.2",
severity="critical")])],
v_from="7.4", v_to="8.10", urls=["https://github.com/redis/redis"])
md = A.markdown(rep)
self.assertIn("critical", md)
self.assertIn("7.4.6", md)
# ── F. live regressions against published historic reports ────────────────────────────────────────
class TestHistoricReportNumbers(unittest.TestCase):
"""Re-derive counts published in week-2026-08-07. Network + GitHub token; opt in with --live."""
REGISTRY = str(HERE / "upstream")
@classmethod
def setUpClass(cls):
if not os.environ.get("ADVISORY_SCAN_LIVE"):
raise unittest.SkipTest("live tests: re-run with --live")
def _count(self, recipe, v_from, v_to, images=None):
rep = A.scan(recipe, v_from, v_to, self.REGISTRY, images)
self.assertEqual(rep["sources_failed"], [], f"{recipe}: source failures make the count unsafe")
self.assertTrue(rep["count_known"], f"{recipe}: count came back UNKNOWN")
return rep
def test_gitea_1_27_0_to_1_27_1_is_2(self):
rep = self._count("gitea", "1.27.0", "1.27.1")
self.assertEqual(rep["cve_count_fixed"], 2)
# Both CVSS-9.8 RCEs — the pair whose omission is why this tool exists.
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-59774", "CVE-2026-60004"})
def test_discourse_app_only_is_123(self):
rep = self._count("discourse", "3.5.3", "2026.7.1")
self.assertEqual(rep["cve_count_fixed"], 123)
self.assertIn("publish date", rep["classified_by"]["github-advisories:discourse/discourse"])
def test_discourse_with_redis_sidecar_is_140(self):
# 123 app + 17 redis. Five redis advisories carry a usable patched_versions; the other
# twelve say "TBD" and are resolved from the release notes that name them.
rep = self._count("discourse", "3.5.3", "2026.7.1", [("redis", "7.4", "8.10")])
self.assertEqual(rep["cve_count_fixed"], 140)
self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 12)
self.assertEqual(rep["cve_count_indeterminate"], 0)
# The five redis advisories that a sidecar-blind scan missed, incl. one critical.
for cve in ("CVE-2024-31227", "CVE-2024-31228", "CVE-2024-31449",
"CVE-2025-49844", "CVE-2025-62507"):
self.assertIn(cve, rep["fixed_by_this_upgrade"], f"{cve} missing from discourse+redis")
self.assertEqual(rep["cves"]["CVE-2025-49844"]["severity"], "critical")
def test_discourse_redis_delta_is_exactly_seventeen(self):
app = self._count("discourse", "3.5.3", "2026.7.1")
both = self._count("discourse", "3.5.3", "2026.7.1", [("redis", "7.4", "8.10")])
delta = set(both["fixed_by_this_upgrade"]) - set(app["fixed_by_this_upgrade"])
self.assertEqual(len(delta), 17)
for cve in delta:
self.assertIn("redis", both["cves"][cve]["sources"][0])
def test_mailu_finds_the_roundcube_pair_without_an_agent(self):
# Published as 2 on 2026-08-07, but only because an agent read the release notes; the scan
# itself contributed 0. It now reaches 2 deterministically: the CVEs appear only on
# github.com/Mailu/Mailu/releases, and release 2024.06.56 (inside the window) names them.
rep = self._count("mailu", "2024.06.55", "2024.06.57", [("redis", "8.8.0", "8.10.0")])
self.assertEqual(rep["cve_count_fixed"], 2)
self.assertEqual(set(rep["fixed_by_this_upgrade"]), {"CVE-2026-54432", "CVE-2026-54433"})
# The redis bump fixes nothing new — every advisory it crosses was fixed at or before 8.6.3.
self.assertEqual(rep["cve_count_indeterminate"], 0)
def test_keycloak_26_7_0_to_26_7_1_is_12(self):
# Was 7 while only the GHSA feed was consulted. keycloak lists five more CVEs in the 26.7.1
# release notes' fixed-issues section that it never filed as advisories — the gitea pattern.
rep = self._count("keycloak", "26.7.0", "26.7.1")
self.assertEqual(rep["cve_count_fixed"], 12)
self.assertEqual(len(rep.get("resolved_by_release_notes") or {}), 5)
def _main():
live = "--live" in sys.argv
if live:
sys.argv.remove("--live")
os.environ["ADVISORY_SCAN_LIVE"] = "1"
unittest.main(verbosity=2)
if __name__ == "__main__":
_main()
+27 -3
View File
@@ -18,9 +18,33 @@
support; monthly `release` channel gets ~2 months; `latest` is continuous. The `esr` Docker tag
tracks the current ESR. Target the ESR line for stable deployments.
- discourse/postgres tags are `pg<MAJOR>` (pg13..pg18), non-semver — abra can't parse them. Check Docker
Hub directly: https://hub.docker.com/r/discourse/postgres/tags . pg18 is the newest (2026-07-01); no
pg19 yet. The image auto-upgrades an older cluster in place on boot (pg_upgrade into versioned PGDATA);
Hub directly: https://hub.docker.com/r/discourse/postgres/tags . pg18 is the newest (no pg19 yet;
re-confirmed 2026-08-07: all of pg13..pg18 + latest re-pushed 2026-08-04, still no pg19). The image
auto-upgrades an older cluster in place on boot (pg_upgrade into versioned PGDATA);
no manual dump/restore needed (the old "Welcome to hell" procedure is superseded for this image).
- Redis 8.0 GA supports upgrade from 7.x data files. Discourse uses Redis only as a cache/queue (Sidekiq)
with no persistence modules. Redis 8.0 integrates RediSearch/RedisJSON/etc. as built-ins; discourse
does not use those modules so this is transparent.
does not use those modules so this is transparent. Redis 8.10-alpine is a safe sidecar bump from 8.8.
- **2026.1 ESR → 2026.7 ESR jump (2026-07-28 release; 2026.7.1 = July 31 security intermediate):**
the big ~6-month ESR jump. Behavior changes that arrive on their own (all surfaced via the new
Upcoming Changes opt-out system, so NOT hard breaks for the recipe boot): Discourse Reactions
enabled by default (opt out via `discourse_reactions_enabled`); Uncategorized being removed (opt-out
for now); `rich_editor` site setting gone (rich editor unconditional, per-user `composition_mode`
remains); simpler email subject lines (beta). Web server: **Discourse replaced Unicorn with Pitchfork
(default in 2026.2, Unicorn removed entirely in 2026.4)** — internal, healthcheck `curl /srv/status`
still works. Theme/plugin deprecations: `.hbs` deprecated (2026.7 is last ESR with support, drop in
2026.8.0-latest → 2027.1 first ESR without it; codemod available); `.js.es6` deprecated (rename to
`.js`); legacy widget shims removed (`discourse.breadcrumbs.*`, `add-flag-property`,
`add-header-panel`, `bootbox`). New features: Upcoming Changes config page, Nested replies
(experimental), one-time email code login (alpha), new admin category management, bulk actions
(tag/pin/suspend), livestreams in Discourse Events. Postgres: discourse-health-check flags anything
below pg15; recipe is on pg18 so fine — do NOT bump the pg major in the weekly cron (operator
decision). For a fresh CI deploy (no custom themes/plugins/data) the jump converges cleanly; the
breaking-change surface is operator-facing for existing sites with custom themes. No Ruby version
concern (Ruby ships in the image). Release notes: https://releases.discourse.org/changelog/v2026.7.1/
and https://releases.discourse.org/changelog/v2026.7.0/ ; community jump guide:
https://meta.discourse.org/t/jumping-from-2026-1-esr-to-2026-7-what-i-found/408779 .
- **Recommended release bump for the ESR jump:** `-x` major (it's a 6-month ESR jump with web-server
swap, default-on behavior changes, and theme deprecations — signal operators to review). The recipe
version label is NOT bumped in the upgrade PR; the operator runs `abra recipe release discourse -x`
after the upstream PR merges.
+7
View File
@@ -3,6 +3,12 @@
| service | image | source repo | releases / changelog |
|---------|-------|-------------|----------------------|
| app | gitea/gitea | https://github.com/go-gitea/gitea | https://github.com/go-gitea/gitea/releases |
**Security announcements: https://blog.gitea.com/ — per-release posts (e.g.
https://blog.gitea.com/release-of-1.27.1/) carry the CVE list; the GitHub release notes do NOT.**
This is where CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8, fixed in 1.27.1) were announced,
and why the 2026-08-03/07 reports under-counted gitea's CVEs. advisory-scan.py fetches every URL
in this file, so keep vendor security pages listed here.
| db | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
## Standing notes
@@ -15,3 +21,4 @@
- **1.26.2**: Multiple CVE security fixes — strongly recommended upgrade.
- **1.26.3**: carries a regression (#38177 "context deadline exceeded" opening repo code pages) — upstream says upgrade straight to 1.26.4, skip 1.26.3.
- **1.27.0** (released 2026-07-13, MINOR with BREAKING changes): `Feat(actions)!: improve support for reusable workflows` (#37478) and `Use Content-Security-Policy: script nonce` (#37232, may break custom inline JS / reverse-proxy CSP). Many security + feature additions. A dedicated minor-bump run should evaluate CSP / reusable-workflow impact before adopting.
- **1.27.1** (released 2026-07-27, PATCH): security fix (oauth2 mandatory 2FA enforcement on authorize/grant endpoints, #38606), API swagger alignment, diff-contrast enhancement, and many bugfixes (actions reusable-workflow/job-stranding, OIDC end-session, repo-deletion cleanup, webhook/mail). No breaking changes; safe patch bump from 1.27.0. Release notes: https://github.com/go-gitea/gitea/releases/tag/v1.27.1
+18 -1
View File
@@ -34,12 +34,29 @@
`valkey:9@sha256:8e8d64b405ce18f41b8e5ee20aa4687a8ed0022d1298f2ce31cdcf3a76e09411`
(a NEWER valkey build than v3.0.3's `4963247afc4cd…` — this `8e8d64b4…` is the same digest the live `9`
tag had already moved to at the 2026-07-17 run; immich v3.1.0 now officially ships it, so it IS the
immich-tested combo → re-pin the recipe's `redis` service to `8e8d64b4…` for the v3.1.0 upgrade) and
immich-tested combo → re-pin the recipe's `redis` service to `8e8d64b4…` for the v3.1.0 upgrade) and
`postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191…` (SAME combo as v3.0.3 — immich did NOT
bump vectorchord/pgvectors, so the recipe STAYS AHEAD on `pgvectors0.3.0@sha256:87c050465…`; do NOT
downgrade). Only **breaking change in v3.1.0** is `chore(mobile): drop support for iOS 14` (mobile
client, NOT server-side — no server migration, no DB migration, no operator action for the recipe).
Done in the 2026-07-31 upgrade (v3.0.1→v3.1.0, extending PR #4 again).
- **2026-08-07: upstream main MOVED to v3.1.0 (published `1.10.0+v3.1.0`, commit `794560f`).** The
coopcloud maintainer published v3.1.0 independently of PR #4 (PR #4 tip `77d7937` is NOT in upstream
main, so the reconcile leaves it open — functionally superseded on the version bump). Upstream main
pins: `immich-server/ML v3.1.0`, `postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf633…` (immich's
OFFICIAL v3.1.0 combo — a DOWNGRADE from the recipe's prior `pgvectors0.3.0@87c0…`), and
`valkey:9@sha256:3acc0687f2a2e1091fae6450d7842dd658c941338cf0a873ddd9e14b9e4ea4dd` (DIVERGES from
immich v3.1.0's official valkey pin `8e8d64b4…` — upstream pin hygiene discrepancy, not an upgrade).
Immich is now **up-to-date** (v3.1.0 is latest stable per GitHub releases; no v3.1.x patch / v3.2.x).
PR #4 now diverges from upstream main only on `database` (pgvectors0.3.0 vs 0.2.0) + `redis`
(valkey:9@8e8d64b4 vs @3acc0687) pins — operator decides merge-vs-close.
- **2026-08-07 INFRA note: cc-ci runner's gitea clone-token is STALE (HTTP 401).** `!testme` build
#1210 died at the recipe `git clone` step (~5s, before any deploy) with `could not read Username for
'https://git.autonomic.zone'` — the runner's mounted gitea token `13e299f2…` is rejected (verified
HTTP 401 on the gitea API + git endpoint; `.testenv` basic creds work). PR #4's tree `77d79379` was
GREEN on #1145 (2026-07-31) & #1180 (2026-08-04) — the RED is infra, not the recipe. Fix: rotate the
runner's gitea clone-token (swarm secret), then re-`!testme`. (The 2026-08-04 Drone-direct
`GITEA_TOKEN`-as-build-param workaround is disallowed for normal runs.)
- **Historical (v2.7.5, 2026-04-13):** immich-server v2.7.5 pinned
`postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191…`. PR #2 bumped the recipe to
`pgvectors0.3.0@sha256:87c050465…` (same PG14 + VectorChord 0.4.3, newer pgvectors 0.2.0→0.3.0).
+11 -1
View File
@@ -7,7 +7,7 @@
| celery | lasuite/impress-backend | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
| y-provider | lasuite/impress-y-provider | https://github.com/suitenumerique/docs | https://github.com/suitenumerique/docs/releases |
| docspec | ghcr.io/docspecio/api | https://github.com/docspecIO/api | https://github.com/docspecIO/api/releases |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/pgautoupgrade/releases |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases |
| redis | redis | https://github.com/redis/redis | https://hub.docker.com/_/redis/tags |
| minio | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
| web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
@@ -21,3 +21,13 @@
- v5.2.0 adds two optional new env vars: DOCUMENT_ALL_ENDPOINT_ENABLED and OIDC_OP_USER_ENDPOINT_FORMAT.
Both are backward-compatible (no action required for existing deployments).
- Recipe version label convention: 0.X.Y+vA.B.C where A.B.C is the impress version.
- **v5.4.0 removed Bearer/JWT auth on the API** (upstream PR suitenumerique/docs#2480 dropped
`mozilla_django_oidc.contrib.drf.OIDCAuthentication` from DRF DEFAULT_AUTHENTICATION_CLASSES).
The API now accepts only the app's session cookie from the real OIDC authorization-code flow.
Any test/assertion that sends `Authorization: Bearer <jwt>` to `/api/v1.0/*` will get 401 — this is
the NEW CORRECT behavior, not a regression. The cc-ci lasuite-docs tests were updated for this in
cc-ci PR #12 (merged 2026-08-03): test_oidc_login.py + test_create_doc.py now use session cookies
and assert Bearer is rejected. Don't revert to Bearer assertions.
- redis sidecar (cache/broker for celery+backend): prefer the patch line (8.8.1 over 8.10.0) unless a
feature is needed — 8.8.1 is a security patch (RedisBloom/TDigest RESTORE RCE); 8.10.0 is a minor
with many new features.
+6 -2
View File
@@ -6,12 +6,12 @@
| backend | lasuite/drive-backend | https://github.com/suitenumerique/drive | https://github.com/suitenumerique/drive/releases |
| celery | lasuite/drive-backend | https://github.com/suitenumerique/drive | https://github.com/suitenumerique/drive/releases |
| celery-beat | lasuite/drive-backend | https://github.com/suitenumerique/drive | https://github.com/suitenumerique/drive/releases |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/pgautoupgrade/releases |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases |
| redis | redis | https://github.com/redis/redis | https://github.com/redis/redis/releases |
| mailcatcher | sj26/mailcatcher | https://github.com/sj26/mailcatcher | https://github.com/sj26/mailcatcher/releases |
| minio | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
| minio-createbuckets | minio/minio | https://github.com/minio/minio | https://github.com/minio/minio/releases |
| collabora | collabora/code | https://github.com/CollaboraOnline/online | https://www.collaboraoffice.com/category/release-notes/ |
| collabora | collabora/code | https://github.com/CollaboraOnline/online | https://www.collaboraonline.com/release-notes/ (per-version: .../collabora-online-25-04-release-notes/) |
| onlyoffice | onlyoffice/documentserver-de | https://github.com/ONLYOFFICE/DocumentServer | https://github.com/ONLYOFFICE/DocumentServer/blob/master/CHANGELOG.md |
| web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
@@ -19,5 +19,9 @@
- lasuite/drive-frontend and lasuite/drive-backend share the same version tag (drive monorepo).
- minio and onlyoffice tags use non-semver formats; abra cannot auto-detect upgrades for them.
- collabora/code uses a 5-part version scheme; abra cannot auto-detect upgrades for it.
- minio: Docker Hub `minio/minio` `latest` is frozen at `RELEASE.2025-09-07T16-13-09Z` (the
`RELEASE.2025-10-15T17-29-55Z` GitHub security release is NOT on Docker Hub `minio/minio`).
Newer minio releases moved to the **AIStor** product line (`quay.io/minio/aistor/minio`).
A future minio bump requires a registry+product switch — operator decision, not a routine tag bump.
- nginx 1.31.x (1.31.0, 1.31.1) contains multiple security CVE fixes; upgrade is recommended.
- nginx 1.31.0 breaking change: HTTP/2 and HTTP/3 requests with Connection/Proxy-Connection/Keep-Alive/Transfer-Encoding/Upgrade headers are now rejected (affects proxied HTTP/2 backends).
+5 -17
View File
@@ -5,24 +5,12 @@
| app | lasuite/meet-frontend | https://github.com/suitenumerique/meet | https://github.com/suitenumerique/meet/releases |
| backend | lasuite/meet-backend | https://github.com/suitenumerique/meet | https://github.com/suitenumerique/meet/releases |
| celery | lasuite/meet-backend | https://github.com/suitenumerique/meet | https://github.com/suitenumerique/meet/releases |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/pgautoupgrade/releases |
| redis | redis | https://github.com/redis/redis | https://hub.docker.com/_/redis/tags |
| db | pgautoupgrade/pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade | https://github.com/pgautoupgrade/docker-pgautoupgrade/releases |
| redis | redis | https://github.com/redis/redis | https://github.com/redis/redis/releases |
| livekit | livekit/livekit-server | https://github.com/livekit/livekit | https://github.com/livekit/livekit/releases |
| web | nginx | https://github.com/nginx/nginx | https://nginx.org/en/CHANGES |
## Standing notes
- meet-frontend, meet-backend (used for both backend + celery services) share the same version tag from the suitenumerique/meet monorepo. Upgrade app, backend, and celery in lockstep.
- AUTO_MIGRATIONS=true means DB migrations run automatically on backend startup. No manual step needed.
- v1.17.0v1.19.0: no breaking changes documented; feature additions only (participant muting, PiP, S3 recording, API exposure). Standard rolling upgrade applies.
- v1.19.0: security fix for CVE-2026-45409 (idna ≥3.15) — no operator action needed (baked into image).
- Recipe version label convention: 0.X.Y+vA.B.C where A.B.C is the meet version.
- LiveKit version is decoupled from meet version; only bump if explicitly required.
- v1.13.1: removes backwards compatibility for TURN auth without TTL (deprecated in v1.12.0). Operators who never set TURN TTL must add it before upgrading LiveKit to v1.13.x. No new required env vars for standard deployments.
- v1.13.2: patch (Added/Changed/Fixed — Prometheus metrics for join latency, 512 KiB metadata cap, egress v2 api, etc.); no breaking changes within v1.13.x.
- v1.13.3: patch (mock API server for SDK testing, data track schema metadata, whip ingress bitrates, webrtc interop fix for bundled datachannel, WHIP notifier fix); no breaking changes.
- v1.22.0: feature release (purge deleted/pending files, generalized STT API, LiveKit egress_ended fallback for recordings, PiP tile cap/pagination, reject user access tokens on API, dedicated PostHog feature-flag domain, backend analytics). No breaking changes; no new required env vars for the standalone meet recipe.
- v1.23.0: feature release (migrate visio integration to summary API v2, summary feature flag, Sentry monitoring for agents, MuteEveryoneButton admin/owner gate, dep upgrades). ⚠️ Special mention: removed `api/v1` code from the `summary` sub-project BUT kept meet→summary v1-API compatibility as the DEFAULT — `SUMMARY_SERVICE_VERSION: 2` is only needed if deploying the latest meet AND summary from the monorepo together. The lasuite-meet recipe deploys meet standalone (no summary service, no `SUMMARY_SERVICE_VERSION` env), so NO operator action is required for this recipe.
- v1.24.0: feature release (deprecate `SUMMARY_SERVICE_VERSION=1` — only matters when deploying meet+summary together, NOT the standalone recipe; prioritize screen share in PiP; recording admin search by owner email; participant color gradient when camera off; PostHog external-user identification; new OPTIONAL env `AUTHENTICATED_PARTICIPANTS_CAN_EDIT_DISPLAY_NAME` defaulting `true` — set `false` to force SSO display name; mjml v5; info-panel crash fix for unregistered rooms; Outsource Outlook add-on calendar fix; whisper call error handling). No breaking changes / no required migrations / no new required env for the standalone meet recipe. AUTO_MIGRATIONS=true applies any Django migrations on backend startup. Standard rolling upgrade.
- livekit v1.13.2/v1.13.3/v1.13.4: patch series within v1.13.x (Prometheus metrics for join latency, mock SDK test API, WHIP ingress bitrates, WEBRTC interop fix for bundled datachannel; v1.13.4 adds SIP mocking, IPv6-exclusion option, forward-stats API method, data-track buffering-under-congestion fix, goroutine-leak fix, pion/ice hang-on-close fix). No breaking changes within v1.13.x; the v1.13.1 TURN-TTL note (only relevant if TURN was ever configured) is the only standing operator action in this line.
- nginx 1.31.3: SECURITY patch release (CVE-2026-42533 heap buffer overflow in `map` with regex; CVE-2026-60005 uninitialized memory with `slice`/background cache update; CVE-2026-56434 use-after-free in ssi filter on proxied backend responses). Change: HTTP/2 response header/trailer sizes now bounded by `proxy_buffer_size`/`grpc_buffer_size`; external entities disabled in xslt module by default. Bugfixes in HTTP/2 flow control, proxy_v2/ tunnel modules. No required config change for the recipe (nginx serves static frontend + proxies to backend); standard rolling upgrade.
- Redis 8.8.0: compatible drop-in upgrade; includes ReJSON module with V5V7 API exported. No config or persistence-format changes.
- lasuite/meet-frontend and lasuite/meet-backend share the same version tag (meet monorepo, versioned together); celery uses the SAME image as backend.
- livekit-server must stay compatible with the meet version's expected livekit API — take newest SAME-minor livekit tag, do NOT jump to a newer major/minor line unless meet release notes require it.
- meet-backend runs Django DB migrations on boot (manage.py migrate) — watch backend logs on upgrade.
+37 -1
View File
@@ -2,7 +2,7 @@
| service | image | source repo | releases / changelog |
|----------|-------------------------------------------|---------------------------------------------------|-------------------------------------------------------------------|
| app | mattermost/mattermost-team-edition | https://github.com/mattermost/mattermost | https://docs.mattermost.com/about/mattermost-changelog.html |
| app | mattermost/mattermost-team-edition | https://github.com/mattermost/mattermost | https://docs.mattermost.com/deploy/mattermost-changelog.html |
| postgres | postgres | https://github.com/postgres/postgres | https://www.postgresql.org/docs/release/ |
## Standing notes
@@ -17,6 +17,33 @@
The `release-11.7` Docker Hub floating tag always points to the latest 11.7.x patch.
IMPORTANT: Do NOT use 11.7.011.7.2 — they have a schemeid migration bug in the `roles` table
when upgrading from 10.11.17+; use 11.7.3 or later (current: 11.7.7).
- **2026-08-07 re-check** (endoflife.date/api/mattermost.json; GitHub releases API; Docker Hub):
Operator (weekly /upgrade-all) directed extending PR #2 to **11.10.0** (newest on the 11.x
innovation line, consistent with PR #2's existing line). **11.10.0 is a PRE-RELEASE** — GitHub
`prerelease=True`, published 2026-08-04 (rc1 2026-07-17, rc2 2026-07-30, rc3+v11.10.0 2026-08-04);
Docker Hub `mattermost/mattermost-team-edition:11.10.0` exists (462 MB, pushed 2026-08-04,
actively pulled). No detailed release notes published yet (body = "Mattermost Platform Release
11.10.0"). 11.10 cycle not yet on endoflife.date; monthly cadence → EOL ~2026-11-15. Innovation,
NOT ESR. This run bumps 11.9.0 → 11.10.0 per operator instruction and flags the pre-release +
ESR-vs-innovation choice prominently in the PR body + report. **ESR remains 11.7.8** (stable,
EOL 2027-05-15) — the real LTS/ESR alternative if the operator wants LTS. **Survey-hint
correction (again):** the 2026-08-07 survey claimed "alternative ESR = 10.12.4"; that is STILL
wrong — 10.12 is an EXPIRED innovation release (EOL 2025-12-15, lts:false on endoflife.date);
do NOT switch to 10.12.4. 10.11 ESR EOL 2026-08-15 (8 days); upstream main still pins 10.11.22.
postgres 15-alpine HELD (major DB bump out of scope for weekly cron).
- **2026-08-04 re-check** (endoflife.date/api/mattermost.json; GitHub releases; Docker Hub):
**11.7.8** released 2026-07-31 is the newest 11.7.x ESR patch (ESR supported through
2027-05-15). **11.9.0** (innovation, released 2026-07-08, cycle EOL **2026-10-15** ~10 weeks,
NOT ESR) and **11.8.4** (innovation, EOL 2026-09-15) remain innovation releases — the standing
guidance for AUTONOMOUS runs is still "do NOT target innovation; track 11.7 ESR." HOWEVER, the
2026-08-03 weekly /upgrade-all operator task **explicitly directed a bump to 11.9.0** (the survey
target), so this run targets 11.9.0 per operator instruction with the EOL/innovation status
flagged in the plan + PR body + report for operator reconsideration. 11.9.0 confirmed real:
GitHub tag `v11.9.0` (published 2026-07-08, not prerelease), Docker Hub
`mattermost/mattermost-team-edition:11.9.0` (458 MB, 2026-07-08). Features: Ranked Attributes,
ABAC per-action Permission Rules, Program Masking, Azure Blob Storage. `abra recipe upgrade`
lists 11.10.0 highest and 11.9.0 next. PR #2 extended 11.7.8 → 11.9.0 (fast-forward). 10.11 ESR
security support ENDS 2026-08-15 (~11 days).
- **2026-07-24 re-check** (endoflife.date/api/mattermost.json; GitHub releases): **11.7.7**
released 2026-07-17 is the newest 11.7.x ESR patch — bumped PR #2 `11.7.6 → 11.7.7` (ESR security
patch, "Low to High severity security fixes", no migrations/breaking changes), `!testme` GREEN
@@ -48,3 +75,12 @@
recreate DB, reimport dump. `DROP DATABASE WITH (FORCE)` requires PostgreSQL 13+ — safe on postgres:15-alpine.
The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).
## NVD CPE fallback
This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
CPE-indexed and carries structured version ranges, so it can answer where the vendor
cannot. It LAGS the vendor, so it is a fallback, never the primary source.
- nvd-cpe: mattermost-team-edition = cpe:2.3:a:mattermost:mattermost_server:*:*:*:*:*:*:*:*
- nvd-cpe: postgres = cpe:2.3:a:postgresql:postgresql:*:*:*:*:*:*:*:*
+29
View File
@@ -0,0 +1,29 @@
# Upstream sources — mumble
| service | image | source repo | releases / changelog |
|---------|-------|-------------|----------------------|
| app | mumblevoip/mumble-server | https://github.com/mumble-voip/mumble | https://github.com/mumble-voip/mumble/releases |
| web | rankenstein/mumble-web | https://github.com/rankenstein/mumble-web | https://github.com/rankenstein/mumble-web/releases |
## Standing notes
- This file was **missing entirely** until 2026-08-11. Without it the advisory scan had no source to
query, and still printed "0 identified by the deterministic scan" — which was then published as a
clean `0` in the 2026-08-11 CVE check. The scan now refuses to emit a count when it has no usable
source (it reports UNKNOWN), and `audit-sources.py` flags a missing registry file directly.
- `mumblevoip/mumble-server` tracks the upstream server releases and DOES publish GitHub security
advisories, so it is the recipe's primary CVE source.
- `rankenstein/mumble-web` is a **fork** of the original `Johni0702/mumble-web`, which has been
dormant since 2023-05. The fork itself last pushed 2023-07 and its Docker tag `0.5` was last built
well over five years ago. Neither is archived, but treat the web client as effectively unmaintained:
if a CVE lands there, expect no upstream fix and plan a replacement rather than an upgrade.
- The server image tag is `v<version>-<build>` (e.g. `v1.6.870-4`); the trailing number is the image
build, not an app version, and moves independently of upstream releases — `abra recipe upgrade`
reports "no new versions" for it, so use `resolve-images.py` to see those bumps.
## NVD CPE fallback
This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
CPE-indexed and carries structured version ranges, so it can answer where the vendor
cannot. It LAGS the vendor, so it is a fallback, never the primary source.
- nvd-cpe: mumble-server = cpe:2.3:a:mumble:mumble:*:*:*:*:*:*:*:*

Some files were not shown because too many files have changed in this diff Show More