The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.
Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.
Two changes so it does not recur:
- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
layer cache so runs stay fast. 'docker image prune -a' spares anything a container
references, so infra and warm-* canonicals are safe. Volumes are still NOT
pruned — warm-* canonical volumes are data-warm and legitimately dangling.
- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
steady-state measure: the host was at 73% when runs started failing. It also now
checks that recent builds actually produced results.json — an empty run dir is
the fingerprint of a host problem masquerading as a recipe failure — and records
how to read a drone step log out of its sqlite when the API token is unreachable.
autonomic-bot
ab88e59c21