25 open recipe PRs had accumulated, and the list had stopped being readable: CI
sweep artifacts that were never meant to merge sat next to genuine CVE fixes, and
three PRs the operator had been told were outstanding were in fact already merged
upstream (discourse #6 with 140 CVEs, keycloak #6 with 12, n8n #5) — visible only
once the mirrors were reconciled.
The skill: reconcile every mirror from true upstream FIRST (that step alone closed
those three), survey every open PR deterministically, close the ones that cannot
merge or were never meant to, and report prioritised action items — CVE-carrying
first — for the ones that should land. It never merges a recipe PR.
pr-survey.py gathers the facts and decides nothing: behind_main, mergeable,
diff_files, which images the PR adds vs which are already pinned in main, the
newest !testme verdict, branch kind, age.
One correctness detail worth the extra state: a FAILED diff fetch is reported as
unknown, never as an empty diff. gitea #4 reads that way (force-pushed branch)
while being a verified green fix, and 'empty diff' is a close signal — so the
tool says DIFF-UNREADABLE(do not close on this) instead.
Working against a stale mirror has cost us three different ways:
- mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while
upstream had already merged AND released it (3.1.3+2024.06.57). The work was
done; only our mirror was behind. Reconciling closed the PR automatically.
- a stale mirror makes a survey report 'no upgrades available', so the recipe
silently drops out of the weekly run.
- reading the wrong branch: several coopcloud recipes keep a stale 'main' beside
the real default 'master'. gitea's main is 1.24.2-rootless while master has
1.27.1-rootless and the merged PRs, so reading main manufactures a false
'three releases behind, missing two CVSS-9.8 RCEs' finding.
The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and
already resolves the default branch itself. What was missing was a single obvious
entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or
--all, and is idempotent — recipe work lives in branches, never on mirror main, so
force-syncing main discards nothing.
/ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now
require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now
point at the shared script.
The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.
Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.
Two changes so it does not recur:
- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
layer cache so runs stay fast. 'docker image prune -a' spares anything a container
references, so infra and warm-* canonicals are safe. Volumes are still NOT
pruned — warm-* canonical volumes are data-warm and legitimately dangling.
- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
steady-state measure: the host was at 73% when runs started failing. It also now
checks that recent builds actually produced results.json — an empty run dir is
the fingerprint of a host problem masquerading as a recipe failure — and records
how to read a drone step log out of its sqlite when the API token is unreachable.
/recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author
test changes, and all three had only 'never weaken a test' as guidance. That did
not stop the plausible failure: the fixture INSERTed rows into the app's database,
which was correct for v2 and silently wrong for v3, where a site must belong to a
team. Events were acked 202 and discarded; the recipe sat RED for six weeks.
The rule that would have prevented it — set state up through the app's own
interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and
each of the three paths is told to read it before editing a test.
immich pins two images with BOTH a tag and a digest, which makes abra FATA and
abandon the WHOLE recipe. It therefore contributed no version data at all and
silently dropped out of every survey — indistinguishable from 'up to date'. The
standing answer was prose in three skills telling an agent to check registries by
hand. This replaces it with a tool.
resolve-images.py reads the compose files and queries registries itself:
- Docker Hub, ghcr, and any OCI registry via its own auth challenge (lscr.io
and dock.mau.dev advertise different realms; assuming ghcr's shape 401'd).
- tag SHAPES (digits -> '#') so -alpine stays on -alpine and 'latest' is never
proposed as an upgrade.
- reports newest_within_major AND newest_same_shape, and refuses to choose:
immich's postgres tag encodes the pg major plus the vectorchord/pgvectors
build immich-server expects, so taking the newest breaks the deploy.
- integrity check: if the CURRENT pin is absent from the listing, the listing
was truncated and any 'newest' is a guess. ghcr caps out past 40k tags, so
that falls back to the project's GitHub releases.
- per-repo cache + backoff + Docker Hub auth: a fleet sweep re-reads nginx,
redis and postgres many times and was getting 429s reported as 'unresolved'.
21/21 recipes now resolve. It found upgrades abra missed entirely in five:
mumble (abra said 'no new versions'; four patches behind), plausible's
clickhouse, lasuite-drive's collabora, gitea's mariadb, immich's postgres.
plausible's carried four CVEs, three high.
Also fixes a real over-count found while validating that: a fix inside the
numeric window is not a fix on the branch you land on. ClickHouse patched
CVE-2023-48704 in 23.9.6.20 AND 23.10.5.20 — landing on 23.10.4.25 crosses the
23.9 fix but sits below its own line's, so it does NOT have it. A fix named on
the target's own line and above the target is now proof of absence.
70 tests (64 offline + 6 live). keycloak's live expectation moves 7 -> 12 and
mailu's 0 -> 2: both are the release-note source finding real fixes that were
never filed as advisories.
Skipping the reconcile to keep the sweep 'read-only' was wrong. It researches a
stale checkout, and on the first real run left two recipes with no survey output
at all — indistinguishable from 'no upgrades available' unless you look. The
reconcile is safe precisely because recipe work lives in branches, not on main.
Also documents the trap that produced a false finding in that run: several
coopcloud recipes keep a stale 'main' beside the real default 'master'. gitea's
main is at 1.24.2-rootless while master has 1.27.1-rootless and the merged PRs,
so reading main reports a recipe three releases behind and missing two CVSS-9.8
RCE fixes. Resolve default_branch from the API before reading any file.
And: no output is not 'no upgrade'. It is a third outcome, and only becomes '?'
after the direct registry check has also failed.
/cve-check answers 'what are we exposed to that an upgrade would fix?' without
running an upgrade: per-recipe, resolve the available window for EVERY image
(sidecars included), run the advisory scan over it, adjudicate whatever pass 1
could not decide, publish a report. Read-only — no PRs, no CI, no merges.
/cve-check-and-upgrade does that sweep, then runs /recipe-upgrade only on the
recipes whose upgrade actually closes a CVE, worst severity first, and reports
on both. --min-severity high for just the urgent ones; --dry-run prints the
queue and stops. Never merges.
Deliberate choices, each written into the skills:
- externals are SWEPT but never upgraded here — a security sweep that skipped
deployed software would misreport exposure, but we don't maintain them.
- an unknown count never justifies an upgrade AND is never treated as clean;
it goes to the Addendum.
- no upgrade available means 0 CVEs, not '?'.
- subagents are told which CVEs justify their upgrade, so the PR says why it
exists — a PR naming the RCE it closes gets reviewed sooner.
recipe-report.py grows a page kind: 'cve' files as cve-DATE.html so a sweep
can't overwrite a weekly edition, while BOTH appear in the same archive index,
suffixed 'full report' / 'CVE check'.
/help and /cc-ci-status updated to route to them.
Adds test-advisory-scan.py (58 offline tests on fixtures + 6 live regressions
against the week-2026-08-07 report) and audit-advisory-scan.py, which re-derives
every count with a SEPARATE semver implementation and its own release fetch and
diffs against the scanner. Both found real defects:
1. Window membership was compared on ragged tuples, so (18,) < (18,0) — a CVE
patched in 18.0 fell OUTSIDE a window ending at 18. Bare major tags are the
norm for sidecars (postgres:18, redis:8-alpine). Now zero-padded, which also
keeps the upper bound conservative (18.5 stays out of a window ending at 18).
2. Advisories with no knowable fix version were silently counted as 'not fixed'.
Twelve redis advisories say patched_versions 'TBD' or '7.4.X' with an
open-ended range — six of them high severity. They are now INDETERMINATE:
not counted, not dismissed, and surfaced in the output.
All twelve turned out to be genuinely fixed: redis names each in the release
notes of every branch that got the fix (CVE-2025-32023 -> 6.2.19, 7.2.10,
7.4.5, 8.0.3, 8.2.0). So a third deterministic method resolves them from
release notes, with the naming tags recorded as the citation. discourse's
redis contribution goes 5 -> 17, and its total 128 -> 140.
Pass 2 (--adjudicate) is the model-judged stage for what arithmetic cannot
settle: it hands over each open case's full evidence, plus every verdict pass 1
reached, and takes FIXED/NOT-FIXED/STILL-UNKNOWN with a reason citing that
evidence. It may only raise a count. Vendor-page-only CVEs — the shape of both
gitea CVSS-9.8 RCEs — now reach it instead of being dropped.
Tests cover pass 1 only, by design; pass 2's judgement is a model's. What is
tested there is deterministic: which cases it selects, and that truncation is
announced rather than silent.
SPEC.md rewritten around the two passes.
Restores the single-value form (operator preference) under the --image name.
Repeat the flag per image, all in one call. Malformed values warn on stderr and
are skipped rather than aborting the scan, since it is an additive pre-step.
Counts unchanged: discourse 128 with redis / 123 without, gitea 2.
The image name was packed into the value, so the flag needed a hand-rolled
KEY=FROM:TO parser with its own malformed-input branch, and 'window' named the
wrong thing — the tool has two kinds of window (version ranges and, on the date
fallback, real date windows) and the flag meant only the first.
Now each part is its own argument: --image redis 7.4 8.10, repeatable, all in
one call. argparse enforces the arity, so the string parsing and its error path
are deleted. 'windows' survives internally as the computed-range concept.
Counts unchanged: discourse 128 with redis / 123 without, gitea 2.
The skill still told the reporter to publish '?' whenever the scan hit a
version-scheme change. The scan now resolves those by advisory publish date, so
that instruction would have re-introduced a '?' for a count it can determine.
Also documents that counts are a union across per-image windows, and that a
sidecar-sourced critical must name its image in the bulletin.
A recipe upgrades several images, each through its own version range. The scan
previously classified only the app repo, so sidecar bumps contributed nothing — the
alternative to the earlier bug where sidecars were judged by the APP's window and
produced a false 133.
Now: --window KEY=FROM:TO (repeatable) gives any other source its own range; each
window is classified independently (one may use patched-version ranges while another
falls back to advisory dates) and the count is the UNION. An image with no window is
still not counted — the scan will not guess a range it was not given. If ANY requested
window cannot be ordered, the total is UNKNOWN rather than a partial number.
/recipe-upgrade now instructs passing a --window per bumped sidecar.
Verified on discourse app 3.5.3->2026.7.1 + redis 7.4->8.10: 128 = 123 (app, by
publish date) + 5 (redis, by version range). The redis five are genuine for that bump
(patched 7.4.1 / 7.4.6 / 8.2.3) and include CVE-2025-49844, CRITICAL — previously
invisible. Regressions clean: gitea still 2, discourse without the sidecar window
still 123.
Operator: 'the scanner should not say 0 when it was not able to scan.' Correct — the
previous patch still led with '0 identified' and relegated the caveat to a footnote,
so the headline number was wrong even though the prose was right. A 0 in a security
column is an assertion of safety; it must never be emitted for an undetermined result.
Now: cve_count_fixed is null (not 0) in JSON, a count_known flag distinguishes
'counted zero' from 'could not count', and the markdown headline reads
'CVEs fixed by this upgrade: UNKNOWN — the scan could NOT determine a count' with an
explicit 'This is NOT zero' and instructions to render '?'.
Verified: discourse 3.5.3 -> 2026.7.1 (semver->calver) now reports UNKNOWN; gitea
1.27.0 -> 1.27.1 still reports 2.
The 2026-08-07 regeneration rendered '?' for 5 of 21 recipes. '?' is meant to be a
rare 'we tried and could not tell'; at that rate it is indistinguishable from noise
and hides the real unknowns. Three causes, none of them genuine uncertainty:
1. URL EXTRACTION BUG (mine). The registry is markdown, so urls appear inside
`backticks` and 'quotes'. The extractor captured the trailing punctuation, so
it fetched https://docs.n8n.io/release-notes/` and https://git.autonomic.zone'`
— both 404 on the malformed url, both 200 when clean. Trailing markdown
punctuation is now stripped. Fixed immich + n8n.
2. STALE REGISTRY URL. mattermost-lts pointed at
docs.mattermost.com/about/mattermost-changelog.html, which 404s; the page moved
to /deploy/. Corrected (same class as the pgautoupgrade fix).
3. WRONG SEMANTICS FOR 'NO UPGRADE'. lasuite-docs and custom-html-tiny were
up-to-date this run, so no scan block existed and the report fell back to '?'.
But a recipe with no upgrade has nothing an upgrade could have fixed — that is
0, not unknown. The report skill now says so explicitly, restricts '?' to scans
that RAN and reported genuinely failed sources, states that benign notes
(no-advisories-published / template URL) never trigger '?', and instructs that
many '?' is itself a bug to raise in the Addendum.
Result across all 16 scanned recipes of that run: 0 failed sources (was 5).
Counts also improved with the classifier fix: discourse 130->133, keycloak ->7.
Why: gitea 1.27.1 fixed CVE-2026-60004 + CVE-2026-59774 (both CVSS 9.8). The
2026-08-03 report printed gitea's CVE count as '1', the 2026-08-07 report as
'none'. Cause chain: the upgrade subagent read the GitHub release notes, which
name NEITHER cve (they are announced only in the vendor blog's security section),
so it recorded one unrelated minor item; the report then derived security content
from those notes plus model knowledge, and the model's training predates the CVEs.
Nothing in the pipeline ever queried an advisory source.
cc-ci-plan/advisory-scan.py — deterministic, per recipe, per upgrade window:
1. GitHub Security Advisories API for every source repo in the upstream registry.
PRIMARY: CVE + GHSA + severity + vulnerable/patched ranges, so 'fixed by THIS
upgrade' is computed. Needs no new per-recipe config (134 registry URLs are
already github.com).
2. Vendor release/security pages — every registry URL, fetched + regex-scanned.
This is the source that actually had the gitea CVEs.
3. OSV where a package mapping exists — supplementary.
Each source reports its own status so 'checked, none found' is never confused with
'not checked'. Source selection was measured, not assumed: for these two CVEs OSV
404'd and NVD's API had them by neither CPE, id, nor keyword — advisory DBs lag the
vendor, hence 1+2 lead.
Wiring is strictly ADDITIVE:
- /recipe-upgrade gains step 2a: run the scan, paste the block into the per-recipe
log, and report the UNION of it and the existing release-note reading. The scan
may never lower a count established by reading.
- /recipe-report treats the block as a FURTHER source, prefers its advisory ids /
severities / fixed-in versions for citation, and must render '?' (not 'none')
when a scan is absent or has failed sources — the false-clean 'none' is exactly
what happened on 2026-08-07.
- upstream/gitea.md records blog.gitea.com as the security-announcement URL.
Verified on the real regression: 1.27.0 -> 1.27.1 now yields exactly the 2 missed
criticals with their GHSA ids; the wider 1.26.2 -> 1.27.1 window yields 62.
The extend path grafts HEAD^{tree} WHOLESALE onto the existing upgrade-* branch
(commit-tree -p <branch tip>). Reconcile force-syncs the MIRROR's main to upstream
but never brought the branch — or the local checkout — forward, so each week the
PR base drifted further back and upstream changes made since the branch was cut
were silently absent from the pushed tree. CI then verified a tree that would
never deploy.
Found on gitea PR #5 (2026-08-10): base 0ab323d predated upstream's
'BREAKING CHANGE: remove forgejo' (37ebd22), so the 1.27.1 bump fixing
CVE-2026-60004 + CVE-2026-59774 was !testme-GREEN against a forgejo-bearing tree.
Two changes:
1. Before pushing, if the local work does not contain the freshly-synced upstream
main, merge upstream in — and FAIL LOUDLY (exit 1, naming the checkout) if that
cannot auto-merge, rather than pushing a tree that omits upstream changes.
2. The extend commit now also parents on upstream main when the branch predated it,
so the recorded history matches the pushed tree. Without it the merge-base stays
stale and a later merge can REVERT upstream's changes. Still no force-push.
Verified against the real gitea drift: detection fires, merge is clean, resulting
tree keeps forgejo removed AND the 1.27.1 pin, history contains upstream.
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
screenshot / key / type subcommands; encodes the reset-invalidates-session and
single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.
Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
Three new skills for keeping cc-ci itself current (manual triggers for
what should ideally be automatic):
- /cc-ci-server-update — bump the cc-ci SERVER host's nixpkgs/sops-nix
flake inputs (hard rev pin, moves deliberately), open a cc-ci PR, then
deploy to the live host via nixos-rebuild switch (operator-authorized,
no-CI window) + health gate (no failed units, infra up, endpoints 200).
Manual trigger for periodic nixpkgs bumps.
- /cc-ci-tests-update — sweep all maintained recipes for stale cc-ci
tests (carry-over from /upgrade-all's commented stale tests + a fresh
/ci-test-review sweep), author the minimal test update per stale
recipe, open a cc-ci test PR, verify via the standard PR + !testme ->
CI flow. Closes the loop the weekly /upgrade-all deliberately defers
(it comments-only; this fixes the tests). Never weakens a test.
- /cc-ci-update — umbrella: run server-update then tests-update, in
that order, gated on the server health check (don't sweep tests on a
sick/rolled-back host). Server-first because the harness toolchain
rides on the host nixpkgs.
All three: create + verify, NEVER merge. Reuse the existing ci-test-review
helpers (open-cc-ci-pr.sh, run-all-recipes.sh, verify-pr.sh) and the
recipe-upgrade --with-tests discipline.
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.
- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
(unused AND older than a week, so this week's likely-reused images stay) before
each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
deterministic gate — no-ops (zero tokens) when the run is complete or
progressing; only when a run stalled/died before completing does it launch a
short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Make explicit that ALL formatting/HTML is owned by recipe-report.py render() and
the model's only artifact is the spec JSON — never hand-write/edit HTML. Matters
now that glm-5.2 drives the report. Also fix stale 'default opus' refs (report
now defaults to opencode-go/glm-5.2, overridable via REPORT_BACKEND/REPORT_MODEL).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trust abra fully for any image whose tag it can read — a normal semver/calver
tag with no newer version is genuinely up-to-date. Only cross-check upstream for
images abra physically can't parse (tag+digest pins, digest-only pins), which is
the actual immich blind spot. Avoids redundant upstream checks on every recipe.
abra recipe upgrade is the first approach, but it silently contributes no
candidate for tag+digest pins (FATA: tag and digest not supported), digest-only
pins, and non-semver tags. immich kept getting skipped this way. Before
concluding SKIPPED — up-to-date, do a direct upstream tag check for every image
abra could not cleanly evaluate; only skip when BOTH agree nothing is newer.
§2 no longer bumps the coop-cloud version label in the recipe PR (no
--dry-run compute, no tag). It records the recommended 'abra recipe release
<recipe> -<x|y|z>' (no --dry-run) in the PR body (§3) as the operator's
final publish step — run after the upstream PR merges, it bumps the label +
tags + pushes in one go. Bumps recipe-maintainer submodule to 9daddac (same
change across its /recipe-upstream, -upgrade-apply, -upgrade-plan, -new-tag).
cc-ci recipe-upgrade skill now computes the version via 'abra recipe release --dry-run'
(not a hand-edit) and requires the PR body to link upstream release notes per service.
Bumps the recipe-maintainer submodule pointer to the matching change.
Operator: uptime-kuma is maintained elsewhere — drop it from the weekly upgrade
but keep it in the used-recipes inventory. New cc-ci-plan/used-recipes.md is the
canonical list of every recipe cc-ci deploys/tests, with a weekly|external tier;
upgrade-all §1 now excludes 'external' rows from the candidate list (explicit
--args still override). uptime-kuma = external; all others weekly.
The durable /16 proxy fix landed in phase pvfix (2026-06-13).
Update the guard description from "safety net until that lands"
to "belt-and-suspenders even after the /16 fix" — guard logic
unchanged, description now accurate.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root-caused (empirically, dockerd logs) the discourse/ghost deploy wedges:
the shared proxy overlay (/24=254 VIPs) exhausts as concurrent stack rm leaks
endpoints over many days -> tasks stuck in Swarm 'New'. Add a per-run safety
net to Step 0 (network prune + docker restart when VIP-allocation failures are
logged). Plans + memory for the durable fix (enlarge proxy to /16 in swarm.nix,
maintenance window) and for debugging/fixing the ghost PR afterward.
Per operator: just work through recipes alphabetically keeping CAP (=
DRONE_RUNNER_CAPACITY=2) subagents running at once, starting the next the moment
one finishes (rolling pool via run_in_background). Removes the wave-barrier and
the heavy/light classification entirely — simpler and no slot ever idles.
Per operator: always fill all CAP slots. Heavy/light alternation only spreads
heavies across waves while a light is available; once only heavies remain, run
two-per-wave (capacity is the tuned ceiling) instead of one-per-wave.
Host memory is the binding limit, so never schedule two HEAVY recipes in the
same capacity wave — pair each heavy (discourse/immich/matrix-synapse/
lasuite-drive/mattermost-lts/ghost) with a light one to bound peak memory while
keeping both slots busy. Heaviest-first could co-schedule two heavies and OOM/
wedge the box (the disc-50cc8a 'New'-state wedge). For CAP>2 cap heavies at
~CAP/2; if only heavies remain, run one-per-wave.
Now that the 2026-06-10 concurrency restructure makes concurrent recipe runs
safe (per-run trees, app-domain locks, isolation), default /upgrade-all to run
up to DRONE_RUNNER_CAPACITY (the drone runner's slots, currently 2) recipe
subagents at a time instead of strictly sequential — using all available
concurrency without oversubscribing. Query the live capacity from
'systemctl show drone-runner-exec' (fallback 2); process recipes in waves of
CAP (emit CAP Agent calls per message, await, next wave). Flags: --capacity N,
--sequential (CAP=1, old default — use when the build loops share the box),
--parallel (unbounded). Applies to the NEXT run; the in-flight run is unaffected.
Per operator: drop the hourly cc-ci-reap-dev-deploys systemd timer; instead run the
dev-* reaper at the START (Step 0, alongside the orphan sweep) and END (new step 4b)
of each /upgrade-all run, with THRESHOLD=0 (the run is quiescent then, so clear all
dev-* unconditionally). The reaper keeps its safe default (4h) for ad-hoc use.
Step-2b mandatory teardown is unchanged (primary mechanism); this is the backstop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- /recipe-upgrade step 2b: teardown is now MANDATORY on every exit path (finally),
with a verify-no-leak check; tear down even on failure before reporting.
- reap-dev-deploys.sh: safe, age-gated backstop that removes only idle dev-* stacks
(never CI per-run stacks, warm-*, infra; an active dev loop stays fresh).
- orchestrator: hourly cc-ci-reap-dev-deploys systemd timer runs it against cc-ci,
bounding any leaked dev deploy from a crashed/abandoned loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Absolute, mode-gated rule reinforced in /recipe-upgrade (Guardrails + the new
step-2b direct-deploy loop where the upgrader has cc-ci host access) and noted as
the interim safeguard in IDEAS.md until the deploy loop moves to isolated infra.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The upgrader now deploys the WIP recipe directly on cc-ci (abra app deploy --chaos
under a dev-<recipe> domain on the local swarm) and inspects live logs
(docker service logs) to SEE what the upgrade does, before/alongside the !testme
CI gate. ADDITIONAL to — not a replacement for — the 3-attempt !testme verification;
it front-loads diagnosis so fewer CI attempts are spent on basics. Always torn down
(orphan-sweep is the backstop). /upgrade-all dispatch references the new step 2b.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abra hard-FATAs on image refs with both a tag and a digest (immich:
postgres:14-vectorchord...@sha256:..., valkey:9@sha256:...), aborting the whole
recipe survey so immich was silently dropped. Per operator: don't normalize the
recipe; catch the failure and check the upstream registry directly.
- /upgrade-all box item 4: a tag+digest parse FATA is NOT not-fetchable. Use abra
for the images it parses; for the rest, list upstream tags (Docker Hub / ghcr /
buildx imagetools) and judge availability (match the variant the app supports,
not blindly the max). Upgradeable if abra OR the direct check finds a newer tag.
- /recipe-upgrade implement: hand-bump tag+digest pins (abra can't), and re-resolve
+ re-append the digest for the new tag so the pin is preserved (never drop it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the table's Status column -> TESTS (the CI/test verdict, unchanged
content). Add a new STATUS column showing the PR's LIVE state, fetched
client-side: 'open' vs a ✓ for any not-open state (merged or closed). The cell
is a JS hook (data-repo/data-pr) derived from existing recipe+pr fields; an
inline, dependency-free, CSP-safe script GETs the same-origin /pr/<recipe>/<n>
proxy (cc-ci nix/modules/reports.nix) on load and every 30s, and degrades to a
muted '?' if the proxy/repo is unreachable. Blank cell when a row has no PR.
Doc + SKILL updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents the end-to-end workflow used to land the intentional-skips/4-rung-ladder
feature: explore harness → branch a local cc-ci clone → implement + unit-verify
cold on cc-ci → live full-stage check → open PR (never push main) → independent
adversary verdict → squash-merge on PASS → deploy via /root/builder-clone rebuild.
Includes the adversary-verify-pr6.md plan as a reusable template.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds sweep-orphans.sh (safe-by-allowlist: removes orphan test stacks, standalone
debug containers >30m old, leaked dangling volumes, and reparented docker-run
wrappers; spares infra + warm-* canonicals and their retained volumes) and wires
it as Step 0 of /upgrade-all so a prior run's leaked stack/container/process can't
contend for the shared Swarm or skew the survey. Idempotent; no-op when clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New page order: short lead -> the full wire table (sorted by priority-to-address,
CVE recipes first, new CVEs count column) -> Addendum (bullets of real special
issues, omitted if clean) -> Security Bulletin -> per-recipe "What changed".
- recipe-report.py: _table() gains a CVEs column + recipe-name linking; new
_changes() helper; render() reordered; docstring SPEC SHAPE updated
(cve/addendum/changes added, needs_attention/routine removed).
- recipe-report/SKILL.md + example-spec.json: new procedure, spec shape, and
gold-standard template (2026-06-05, new format).
- launch-report.py: kickoff text reflects the new priority-ordered structure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A stale cc-ci-report session (from a prior week's run, gone idle) caused this week's
launch-report.py 'start' (use-or-create) to leave it and never run a fresh report.
Fix: upgrade-all step 6 now calls 'fresh', and start only leaves a session that's
actively busy producing a report — an idle/leftover session is killed + restarted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reconcile that's supposed to make the mirror main == upstream main was fetching origin/main —
but origin is the cc-ci MIRROR, so it synced the mirror to itself (a no-op) and never pulled real
upstream. Fix: fetch coopcloud explicitly (git.coopcloud.tech/coop-cloud/<recipe>, default branch
main OR master) via an 'upstream' remote and force-sync the mirror main + tags from it. Every recipe
has a coopcloud correspondent; none are forked. Also reorder the skill so the reconcile runs BEFORE
the upgrade check, so the check sees the real current recipe. Verified by divergence test (diverged a
mirror, reconcile snapped it back to coopcloud HEAD).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Close the two gaps vs recipe-maintainer's recipe-upgrade-plan:
- Per-recipe release-notes registry at cc-ci-plan/upstream/<recipe>.md (discover the source repo +
releases/changelog URL for each image once, persist+commit, reuse) — fetch release notes FROM those
URLs instead of rediscovering ad-hoc each run. Format doc + cryptpad seed included.
- Explicitly read the recipe's README for shipped upgrade/migration notes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
autonomic-bot
autonomic-bot