cc-ci-status: correct the ENOSPC note — inode recreation was coincidence
I recorded that recreating the runs directory with a fresh inode preceded recovery. It recurred afterwards (build 1252), so that was not the fix. The real signal is that it is INTERMITTENT and tracks concurrent activity: every failure landed while a second run or manual deploy was in flight, and every build on a quiet host passed (1243, 1250, 1251, 1253). Free space never moves during a failing build. Practical guidance is therefore to wait for the host to go quiet and re-trigger before calling it a recipe failure, and DRONE_RUNNER_CAPACITY=2 is the obvious knob to test if it becomes disruptive. Root cause still not established, and the note now says so plainly rather than presenting a coincidence as understood.
This commit is contained in:
@@ -100,10 +100,16 @@ systemctl --failed --no-legend; df -h / | tail -1; tmux ls
|
||||
> died on `mkdir /var/lib/cc-ci-runs/<build>` with **110GB free and 16% inodes**, while the identical
|
||||
> mkdir succeeded as root over ssh, inside the runner's own mount namespace, and 61/61 times in a
|
||||
> stress loop — and the same harness run by hand with a numeric run id worked fine. Restarting
|
||||
> `drone-runner-exec` did NOT help. What preceded recovery was **recreating the runs directory with
|
||||
> a fresh inode** (`mv cc-ci-runs cc-ci-runs.old && mkdir -m 750 cc-ci-runs && mv cc-ci-runs.old/* cc-ci-runs/`),
|
||||
> after which builds ran normally. Root cause was NOT conclusively identified — record the evidence
|
||||
> if it recurs rather than assuming this is understood.
|
||||
> `drone-runner-exec` did NOT help, and neither did recreating the runs directory with a fresh
|
||||
> inode (it recurred afterwards — that apparent fix was coincidence).
|
||||
>
|
||||
> **It is INTERMITTENT and tracks concurrent activity**, which is the useful signal: every failure
|
||||
> landed while a second run or a manual deploy was in flight (1252 was triggered while 1251 was
|
||||
> still finishing), and every build on a quiet host succeeded (1243, 1250, 1251, 1253). Free space
|
||||
> never moved during a failing build. So on ENOSPC-with-free-disk: **wait for the host to go quiet
|
||||
> and re-trigger** before treating it as a recipe failure. Root cause is still NOT established;
|
||||
> `DRONE_RUNNER_CAPACITY=2` allows the overlap, so lowering it to 1 is the obvious next experiment
|
||||
> if it becomes disruptive.
|
||||
- **Bridge / !testme path**: `docker service ls` shows `ccci-bridge_app 1/1` AND the bridge log
|
||||
has no auth errors (`docker service logs --since 24h ccci-bridge_app 2>&1 | grep -ci "401\|user does not exist"` == 0).
|
||||
A silently-401ing bridge drops `!testme` (seen 2026-08-03, stale rotated Gitea secret) →
|
||||
|
||||
Reference in New Issue
Block a user