Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.
Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.
The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.
Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.
Two changes so it does not recur:
- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
layer cache so runs stay fast. 'docker image prune -a' spares anything a container
references, so infra and warm-* canonicals are safe. Volumes are still NOT
pruned — warm-* canonical volumes are data-warm and legitimately dangling.
- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
steady-state measure: the host was at 73% when runs started failing. It also now
checks that recent builds actually produced results.json — an empty run dir is
the fingerprint of a host problem masquerading as a recipe failure — and records
how to read a drone step log out of its sqlite when the API token is unreachable.
/recipe-upgrade --with-tests, /ci-test-review and /cc-ci-tests-update all author
test changes, and all three had only 'never weaken a test' as guidance. That did
not stop the plausible failure: the fixture INSERTed rows into the app's database,
which was correct for v2 and silently wrong for v3, where a site must belong to a
team. Events were acked 202 and discarded; the recipe sat RED for six weeks.
The rule that would have prevented it — set state up through the app's own
interface, not its database — now lives in tests/STYLE.md in the cc-ci repo, and
each of the three paths is told to read it before editing a test.
/cve-check answers 'what are we exposed to that an upgrade would fix?' without
running an upgrade: per-recipe, resolve the available window for EVERY image
(sidecars included), run the advisory scan over it, adjudicate whatever pass 1
could not decide, publish a report. Read-only — no PRs, no CI, no merges.
/cve-check-and-upgrade does that sweep, then runs /recipe-upgrade only on the
recipes whose upgrade actually closes a CVE, worst severity first, and reports
on both. --min-severity high for just the urgent ones; --dry-run prints the
queue and stops. Never merges.
Deliberate choices, each written into the skills:
- externals are SWEPT but never upgraded here — a security sweep that skipped
deployed software would misreport exposure, but we don't maintain them.
- an unknown count never justifies an upgrade AND is never treated as clean;
it goes to the Addendum.
- no upgrade available means 0 CVEs, not '?'.
- subagents are told which CVEs justify their upgrade, so the PR says why it
exists — a PR naming the RCE it closes gets reviewed sooner.
recipe-report.py grows a page kind: 'cve' files as cve-DATE.html so a sweep
can't overwrite a weekly edition, while BOTH appear in the same archive index,
suffixed 'full report' / 'CVE check'.
/help and /cc-ci-status updated to route to them.
Operator decision: no policy difference between cc-ci and recipe-maintainer. On
inspection ARM already agrees (recipe-upgrade-cron-all: 'PRs are reviewed and merged
manually by a human afterwards... never merges anything'; 'no human review in the
middle' = skip the mid-run plan confirmation only). Wrappers previously framed this
as a cc-ci override over ARM auto-merge flows — wrong reading; now stated as ONE
unified rule. /help conventions updated to match.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.
Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
autonomic-bot