cc-ci-status: record the ENOSPC-with-free-disk failure and what recovered it

Builds 1244-1249 died on mkdir of the run dir with 110GB free and 16% inodes.
Ruled out: actual disk (df sampled every 2s across a failing build never moved),
inodes, quotas, a poisoned parent directory (61/61 stress creations succeeded),
runner sandboxing (namespaces identical to the host), and a wedged runner
(restart changed nothing). The same harness with the same numeric run id, run by
hand outside drone, worked every time.

Recreating the runs directory with a fresh inode preceded recovery; builds have
run normally since. The root cause is NOT established, so the note says so rather
than presenting a fix that might be coincidence.
This commit is contained in:
autonomic-bot
2026-08-11 17:39:56 +00:00
parent ab88e59c21
commit ecf126d98d
+8
View File
@@ -96,6 +96,14 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls
(`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
(distroless container, swarm secret).
> **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
> `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with
> a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`),
> after which builds ran normally. Root cause was NOT conclusively identified — record the evidence
> if it recurs rather than assuming this is understood.
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →