From ecf126d98dbda294da485b0fcc0f0883f45d06ab Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 17:39:56 +0000 Subject: [PATCH] cc-ci-status: record the ENOSPC-with-free-disk failure and what recovered it Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes. Ruled out: actual disk (df sampled every 2s across a failing build never moved), inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded), runner sandboxing (namespaces identical to the host), and a wedged runner (restart changed nothing). The same harness with the same numeric run id, run by hand outside drone, worked every time. Recreating the runs directory with a fresh inode preceded recovery; builds have run normally since. The root cause is NOT established, so the note says so rather than presenting a fix that might be coincidence. --- .opencode/skills/cc-ci-status/SKILL.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index d1c1706..b33f962 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -96,6 +96,14 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls (`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable (distroless container, swarm secret). + > **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249 + > died on `mkdir /var/lib/cc-ci-runs/` with **110GB free and 16% inodes**, while the identical + > mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a + > stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting + > `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with + > a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`), + > after which builds ran normally. Root cause was NOT conclusively identified — record the evidence + > if it recurs rather than assuming this is understood. - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0). A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →