nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday
03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone
builds → skip), `nix flake update` → build → switch-to-configuration test →
cc-ci health checks (sshd, 0 failed units, core units, every swarm service at
replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401)
→ profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci
fast-forwarded; revert + lock restore on failure; one-line state file for
/cc-ci-status.
Skills (.opencode canonical, .claude pointers' descriptions synced):
- cc-ci-orchestrator-update: THE host update — drives the auto-update unit by
hand; --cc-ci-only for a cc-ci-main-only move; failure playbook.
- cc-ci-server-update: delegates to it and explains why the old procedure
(rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host.
- cc-ci-update: chains orchestrator-update then tests-update.
- cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot
generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator
agent session, secrets inventory; verdict updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
I recorded that recreating the runs directory with a fresh inode preceded
recovery. It recurred afterwards (build 1252), so that was not the fix.
The real signal is that it is INTERMITTENT and tracks concurrent activity: every
failure landed while a second run or manual deploy was in flight, and every build
on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a
failing build. Practical guidance is therefore to wait for the host to go quiet
and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is
the obvious knob to test if it becomes disruptive.
Root cause still not established, and the note now says so plainly rather than
presenting a coincidence as understood.
Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.
Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.
The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.
Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.
Two changes so it does not recur:
- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
layer cache so runs stay fast. 'docker image prune -a' spares anything a container
references, so infra and warm-* canonicals are safe. Volumes are still NOT
pruned — warm-* canonical volumes are data-warm and legitimately dangling.
- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
steady-state measure: the host was at 73% when runs started failing. It also now
checks that recent builds actually produced results.json — an empty run dir is
the fingerprint of a host problem masquerading as a recipe failure — and records
how to read a drone step log out of its sqlite when the API token is unreachable.
/cve-check answers 'what are we exposed to that an upgrade would fix?' without
running an upgrade: per-recipe, resolve the available window for EVERY image
(sidecars included), run the advisory scan over it, adjudicate whatever pass 1
could not decide, publish a report. Read-only — no PRs, no CI, no merges.
/cve-check-and-upgrade does that sweep, then runs /recipe-upgrade only on the
recipes whose upgrade actually closes a CVE, worst severity first, and reports
on both. --min-severity high for just the urgent ones; --dry-run prints the
queue and stops. Never merges.
Deliberate choices, each written into the skills:
- externals are SWEPT but never upgraded here — a security sweep that skipped
deployed software would misreport exposure, but we don't maintain them.
- an unknown count never justifies an upgrade AND is never treated as clean;
it goes to the Addendum.
- no upgrade available means 0 CVEs, not '?'.
- subagents are told which CVEs justify their upgrade, so the PR says why it
exists — a PR naming the RCE it closes gets reviewed sooner.
recipe-report.py grows a page kind: 'cve' files as cve-DATE.html so a sweep
can't overwrite a weekly edition, while BOTH appear in the same archive index,
suffixed 'full report' / 'CVE check'.
/help and /cc-ci-status updated to route to them.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
notplantsandClaude Fable 5.1
autonomic-bot