CI server health: image pruning, disk thresholds, reconcile-upstream #5

2026-08-11T19:03:39Z

autonomic-bot commented

2026-08-11 19:03:39 +00:00

sweep-orphans.sh prunes unused images above 60% disk. The CI server filled up (72GB of images, 63GB unused) and every !testme from build 1236–1242 died at harness startup with ENOSPC; because no results.json was written, the PR badges just read "failure" and looked like recipe regressions. Reclaimed 69.8GB. Volumes are deliberately NOT pruned — warm-* canonical volumes are data-warm and legitimately dangling.
/cc-ci-status now flags server disk at >65% (it was at 73% when runs began failing), checks that recent builds actually produced results.json, and records how to read a drone step log out of its sqlite when the API token is unreachable.
reconcile-upstream.sh — one deterministic entry point, now mandatory before any PR work. Working against a stale mirror cost us three ways, including linking mailu#6 as the fix for two CVEs that upstream had already merged and released.

Retrospective review PR. These commits were pushed straight to main during the 2026-08-11 session; they are already live. This PR exists so the work is reviewable and commentable as a unit — its base is a branch pinned at the pre-session commit, so merging it simply advances that pointer and changes nothing on main. Going forward, orchestrator changes go through a PR before landing.

- **`sweep-orphans.sh` prunes unused images** above 60% disk. The CI server filled up (72GB of images, 63GB unused) and every `!testme` from build 1236–1242 died at harness startup with `ENOSPC`; because no `results.json` was written, the PR badges just read "failure" and looked like recipe regressions. Reclaimed 69.8GB. Volumes are deliberately NOT pruned — warm-* canonical volumes are data-warm and legitimately dangling. - **`/cc-ci-status`** now flags server disk at >65% (it was at 73% when runs began failing), checks that recent builds actually produced `results.json`, and records how to read a drone step log out of its sqlite when the API token is unreachable. - **`reconcile-upstream.sh`** — one deterministic entry point, now mandatory before any PR work. Working against a stale mirror cost us three ways, including linking mailu#6 as the fix for two CVEs that upstream had already merged *and released*. --- *Retrospective review PR.* These commits were pushed straight to `main` during the 2026-08-11 session; they are already live. This PR exists so the work is reviewable and commentable as a unit — its base is a branch pinned at the pre-session commit, so merging it simply advances that pointer and changes nothing on `main`. Going forward, orchestrator changes go through a PR before landing.

autonomic-bot added 4 commits 2026-08-11 19:03:40 +00:00

cc-ci: prune unused images in the sweep; catch a starving host before CI dies ab88e59c21

The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.

Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.

Two changes so it does not recur:

- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
  unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
  layer cache so runs stay fast. 'docker image prune -a' spares anything a container
  references, so infra and warm-* canonicals are safe. Volumes are still NOT
  pruned — warm-* canonical volumes are data-warm and legitimately dangling.

- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
  steady-state measure: the host was at 73% when runs started failing. It also now
  checks that recent builds actually produced results.json — an empty run dir is
  the fingerprint of a host problem masquerading as a recipe failure — and records
  how to read a drone step log out of its sqlite when the API token is unreachable.

cc-ci-status: record the ENOSPC-with-free-disk failure and what recovered it ecf126d98d

Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.

Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.

reconcile-upstream.sh: one deterministic entry point, mandated before PR work bb7ebb4a27

Working against a stale mirror has cost us three different ways:

- mailu #6 was linked as the fix for two internet-facing Roundcube CVEs while
  upstream had already merged AND released it (3.1.3+2024.06.57). The work was
  done; only our mirror was behind. Reconciling closed the PR automatically.
- a stale mirror makes a survey report 'no upgrades available', so the recipe
  silently drops out of the weekly run.
- reading the wrong branch: several coopcloud recipes keep a stale 'main' beside
  the real default 'master'. gitea's main is 1.24.2-rootless while master has
  1.27.1-rootless and the merged PRs, so reading main manufactures a false
  'three releases behind, missing two CVSS-9.8 RCEs' finding.

The reconcile logic already existed inside open-recipe-pr.sh --reconcile-only and
already resolves the default branch itself. What was missing was a single obvious
entry point and a rule saying to run it. reconcile-upstream.sh takes recipes or
--all, and is idempotent — recipe work lives in branches, never on mirror main, so
force-syncing main discards nothing.

/ci-test-review and /cc-ci-tests-update had NO reconcile step at all; both now
require it. /cve-check, /recipe-upgrade and /upgrade-all already reconciled and now
point at the shared script.

cc-ci-status: correct the ENOSPC note — inode recreation was coincidence 94ce5c4de2

I recorded that recreating the runs directory with a fresh inode preceded
recovery. It recurred afterwards (build 1252), so that was not the fix.

The real signal is that it is INTERMITTENT and tracks concurrent activity: every
failure landed while a second run or manual deploy was in flight, and every build
on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a
failing build. Practical guidance is therefore to wait for the host to go quiet
and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is
the obvious knob to test if it becomes disruptive.

Root cause still not established, and the note now says so plainly rather than
presenting a coincidence as understood.

autonomic-bot merged commit a0d1b82869 into review/3-upstream-resolution

2026-08-11 19:03:55 +00:00

autonomic-bot referenced this issue from a commit

2026-08-11 19:17:01 +00:00

add /cc-ci-cleanup — reconcile, close dead PRs, report what actually blocks the rest

autonomic-bot referenced this pull request

2026-08-11 19:17:19 +00:00

add /cc-ci-cleanup — reconcile, close dead PRs, report what blocks the rest #7

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: recipe-maintainers/cc-ci-orchestrator#5