recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cc-ci: prune unused images in the sweep; catch a starving host before CI dies

Browse Source 
The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.

Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.

Two changes so it does not recur:

- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
  unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
  layer cache so runs stay fast. 'docker image prune -a' spares anything a container
  references, so infra and warm-* canonicals are safe. Volumes are still NOT
  pruned — warm-* canonical volumes are data-warm and legitimately dangling.

- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
  steady-state measure: the host was at 73% when runs started failing. It also now
  checks that recent builds actually produced results.json — an empty run dir is
  the fingerprint of a host problem masquerading as a recipe failure — and records
  how to read a drone step log out of its sqlite when the API token is unreachable.
This commit is contained in:
autonomic-bot
2026-08-11 15:41:40 +00:00
parent e89da2d842
commit ab88e59c21
2 changed files with 42 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/upgrade-all/sweep-orphans.sh
+23
View File
@@ -73,6 +73,29 @@ done
# 5) Stray exited containers (debug one-shots) — best-effort prune.
docker container prune -f >/dev/null 2>&1 || true
# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and
# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused),
# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with
# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/<build>'`. Every !testme
# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read
# "failure" and looked like recipe regressions.
#
# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast.
# `image prune -a` removes only images no container references, so anything deployed (infra +
# warm-* canonicals) is untouched; anything else is re-pulled on demand.
#
# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are
# data-warm and their volumes are legitimately dangling between runs.
DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}"
used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')"
if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then
echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images"
freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')"
echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')"
else
echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache"
fi
if [ "$removed" -eq 0 ]; then
echo "== orphan sweep: clean (nothing to remove) =="
else