From ab88e59c21adc2ea9de1bd1146746ac06326f29c Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 15:41:27 +0000 Subject: [PATCH] cc-ci: prune unused images in the sweep; catch a starving host before CI dies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI server filled up and every !testme from build 1236 to 1242 died at harness startup with ENOSPC on /var/lib/cc-ci-runs/. Because the harness never got far enough to write results.json, the PR badges just said 'failure' — so it read as recipe regressions, and plausible's genuinely-fixed suite looked still-broken. Cause: every run pulls each recipe's images and nothing ever removed the old ones. 72GB of images, 63GB of it unused. Reclaimed 69.8GB; the host went 73% -> 22%. Two changes so it does not recur: - sweep-orphans.sh (runs at the start AND end of every /upgrade-all) now prunes unused images when the disk is >=60% (DISK_PRUNE_PCT). Below that it keeps the layer cache so runs stay fast. 'docker image prune -a' spares anything a container references, so infra and warm-* canonicals are safe. Volumes are still NOT pruned — warm-* canonical volumes are data-warm and legitimately dangling. - /cc-ci-status flags server disk at >65% rather than >80%, because this is not a steady-state measure: the host was at 73% when runs started failing. It also now checks that recent builds actually produced results.json — an empty run dir is the fingerprint of a host problem masquerading as a recipe failure — and records how to read a drone step log out of its sqlite when the API token is unreachable. --- .claude/skills/upgrade-all/sweep-orphans.sh | 23 +++++++++++++++++++++ .opencode/skills/cc-ci-status/SKILL.md | 21 +++++++++++++++++-- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/.claude/skills/upgrade-all/sweep-orphans.sh b/.claude/skills/upgrade-all/sweep-orphans.sh index c6b95e1..442ce56 100755 --- a/.claude/skills/upgrade-all/sweep-orphans.sh +++ b/.claude/skills/upgrade-all/sweep-orphans.sh @@ -73,6 +73,29 @@ done # 5) Stray exited containers (debug one-shots) — best-effort prune. docker container prune -f >/dev/null 2>&1 || true +# 6) Unused IMAGES — the one that actually took CI down. Every run pulls each recipe's images and +# nothing ever removed the old ones: on 2026-08-11 they had grown to 72GB (63GB of it unused), +# the root filesystem hit 100% under two concurrent runs, and the harness died at startup with +# `OSError: [Errno 28] No space left on device: '/var/lib/cc-ci-runs/'`. Every !testme +# from build 1236 to 1242 failed that way — with no results.json, so the PR badges just read +# "failure" and looked like recipe regressions. +# +# Only prune above a threshold, so a healthy host keeps its layer cache and runs stay fast. +# `image prune -a` removes only images no container references, so anything deployed (infra + +# warm-* canonicals) is untouched; anything else is re-pulled on demand. +# +# Volumes are deliberately NOT pruned here — see the KEEP_RE guard in (3): warm-* canonicals are +# data-warm and their volumes are legitimately dangling between runs. +DISK_PRUNE_PCT="${DISK_PRUNE_PCT:-60}" +used_pct="$(df --output=pcent / 2>/dev/null | tail -1 | tr -dc '0-9')" +if [ -n "$used_pct" ] && [ "$used_pct" -ge "$DISK_PRUNE_PCT" ]; then + echo " disk ${used_pct}% >= ${DISK_PRUNE_PCT}% -> pruning unused images" + freed="$(docker image prune -af 2>/dev/null | awk '/Total reclaimed space/ {print $4, $5}')" + echo " reclaimed: ${freed:-0B}; disk now $(df -h / | tail -1 | awk '{print $5" used, "$4" free"}')" +else + echo " disk ${used_pct:-?}% < ${DISK_PRUNE_PCT}% -> keeping image cache" +fi + if [ "$removed" -eq 0 ]; then echo "== orphan sweep: clean (nothing to remove) ==" else diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index 0366ef7..d1c1706 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -78,8 +78,24 @@ ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls systemctl --failed --no-legend; df -h / | tail -1; tmux ls ``` - Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern — - note, don't page), disk >80% (server) / >85% (orchestrator) → findings. Server unreachable → + note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable → HIGH: recommend `hetzner-server-recovery`. + > **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs + > pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run. + > That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever + > pruned them), the filesystem filled during a run, and the harness died at startup with + > `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it + > spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT + > `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling. +- **!testme actually produces results** (the check that would have caught the above days earlier): + the newest few `/var/lib/cc-ci-runs//` dirs must each contain `results.json`. A build that + dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so + it reads as a recipe regression rather than a sick host. Builds 1236–1242 all failed that way. + Finding: *"N recent builds produced no results.json — the harness is dying at startup, check disk + and the drone step log"*. The step log lives in drone's sqlite + (`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and + read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable + (distroless container, swarm secret). - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0). A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) → @@ -113,7 +129,8 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm `/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?". `ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no -CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, disk under +CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all +producing results.json, disk under thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).