From 94ce5c4de20a92b2b9226da145f62c367432017c Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Tue, 11 Aug 2026 18:54:58 +0000 Subject: [PATCH] =?UTF-8?q?cc-ci-status:=20correct=20the=20ENOSPC=20note?= =?UTF-8?q?=20=E2=80=94=20inode=20recreation=20was=20coincidence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I recorded that recreating the runs directory with a fresh inode preceded recovery. It recurred afterwards (build 1252), so that was not the fix. The real signal is that it is INTERMITTENT and tracks concurrent activity: every failure landed while a second run or manual deploy was in flight, and every build on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a failing build. Practical guidance is therefore to wait for the host to go quiet and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is the obvious knob to test if it becomes disruptive. Root cause still not established, and the note now says so plainly rather than presenting a coincidence as understood. --- .opencode/skills/cc-ci-status/SKILL.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/.opencode/skills/cc-ci-status/SKILL.md b/.opencode/skills/cc-ci-status/SKILL.md index b33f962..6c58cd7 100644 --- a/.opencode/skills/cc-ci-status/SKILL.md +++ b/.opencode/skills/cc-ci-status/SKILL.md @@ -100,10 +100,16 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls > died on `mkdir /var/lib/cc-ci-runs/` with **110GB free and 16% inodes**, while the identical > mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a > stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting - > `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with - > a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`), - > after which builds ran normally. Root cause was NOT conclusively identified — record the evidence - > if it recurs rather than assuming this is understood. + > `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh + > inode (it recurred afterwards — that apparent fix was coincidence). + > + > **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure + > landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was + > still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space + > never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet + > and re-trigger** before treating it as a recipe failure. Root cause is still NOT established; + > `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment + > if it becomes disruptive. - **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0). A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →