2026-08-11 19:03:55 +00:00
1 changed files with 8 additions and 0 deletions
@@ -96,6 +96,14 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls
  (`/var/lib/docker/volumes/drone_ci_commoninternet_net_data/_data/database.sqlite`) — copy it and
  read `logs.log_data` for the failing `steps.step_id`; the bridge's drone token is not extractable
  (distroless container, swarm secret).
+  > **If the error is ENOSPC but the disk is fine**, it is not disk. Seen 2026-08-11: builds 1244-1249
+  > died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
+  > mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
+  > stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
+  > `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with
+  > a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`),
+  > after which builds ran normally. Root cause was NOT conclusively identified — record the evidence
+  > if it recurs rather than assuming this is understood.
 - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
  has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
  A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →