recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

weekly-run: pre-reclaim stale cc-ci images + hourly glm-5.2 supervisor

Browse Source 
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.

- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
  (unused AND older than a week, so this week's likely-reused images stay) before
  each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
  deterministic gate — no-ops (zero tokens) when the run is complete or
  progressing; only when a run stalled/died before completing does it launch a
  short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
  judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
  a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
This commit is contained in:
autonomic-bot
2026-07-04 04:33:05 +00:00
co-authored by Claude Opus 4.8
parent 52e7c954a3
commit 1bd156e7e6
4 changed files with 238 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/upgrade-all/SKILL.md
+10
View File
@@ -81,6 +81,16 @@ fi
remains as belt-and-suspenders even after the /16 fix: it fires on the exact error signature and restarts
docker to reclaim leaked endpoints if VIP exhaustion ever recurs despite the larger subnet.)
Then **reclaim STALE docker images so the run can't fill the disk mid-flight.** A full run deploys
~16 recipes; their images accumulate week over week and can run the cc-ci root FS to 100% (ENOSPC),
which killed the 2026-07-03 run mid-way (lasuite-drive, Wave 6). Clear only **stale** images —
unused by any container AND older than a week — so this week's likely-reused images are kept:
```
ssh cc-ci 'docker image prune -af --filter until=168h 2>&1 | tail -1; df -h / | tail -1'
```
(When the run is launched via `launch-upgrader.py` this is done automatically as step 0 — the
`prereclaim_cc_ci()` pre-step — so you only run it by hand for a manual `/upgrade-all`.)
## 1. Build the candidate list
Enrolled recipes = the cc-ci `tests/<recipe>/` dirs (same set `ci-test-review` sweeps), **MINUS any
recipe tagged `external` in `cc-ci-plan/used-recipes.md`** — recipes cc-ci uses/tests but does NOT