cc-ci: prune unused images in the sweep; catch a starving host before CI dies

The CI server filled up and every !testme from build 1236 to 1242 died at harness
startup with ENOSPC on /var/lib/cc-ci-runs/<build>. Because the harness never got
far enough to write results.json, the PR badges just said 'failure' — so it read
as recipe regressions, and plausible's genuinely-fixed suite looked still-broken.

Cause: every run pulls each recipe's images and nothing ever removed the old ones.
72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%.

Two changes so it does not recur:

- sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes
  unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the
  layer cache so runs stay fast. 'docker image prune -a' spares anything a container
  references, so infra and warm-* canonicals are safe. Volumes are still NOT
  pruned — warm-* canonical volumes are data-warm and legitimately dangling.

- /cc-ci-status flags server disk at >65% rather than >80%, because this is not a
  steady-state measure: the host was at 73% when runs started failing. It also now
  checks that recent builds actually produced results.json — an empty run dir is
  the fingerprint of a host problem masquerading as a recipe failure — and records
  how to read a drone step log out of its sqlite when the API token is unreachable.
This commit is contained in:
autonomic-bot
2026-08-11 15:41:40 +00:00
parent e89da2d842
commit ab88e59c21
2 changed files with 42 additions and 2 deletions
+19 -2
View File
@@ -78,8 +78,24 @@ ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls
systemctl --failed --no-legend; df -h / | tail -1; tmux ls
```
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable →
note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
HIGH: recommend `hetzner-server-recovery`.
> **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
> pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
> That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
> pruned them), the filesystem filled during a run, and the harness died at startup with
> `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
> spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
> `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
- **!testme actually produces results** (the check that would have caught the above days earlier):
the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
it reads as a recipe regression rather than a sick host. Builds 12361242 all failed that way.
Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk
and the drone step log"*. The step log lives in drone's sqlite
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
(distroless container, swarm secret).
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
@@ -113,7 +129,8 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
`/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".
`ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under
CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
producing results.json, disk under
thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).