Root cause chain, all confirmed on the live system:
1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's
kickoff PROMPT is an argv element, and the supervisor's prompt text contains the
literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a
live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>).
2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its
process alive and SPINNING while emitting nothing (verified: 3 days, zero session
output, still burning CPU). Progress now REQUIRES the session tree to have advanced
within STALL_MIN; a live-but-idle proc is explicitly logged as stalled.
Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly
run sat unfinished and unreported.
Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may
resume when the wall lifts and their context is the run's state. New _billing_blocked()
detects the wall from the log tail and the gate surfaces
'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'.
Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while
still matching the real finisher; gate now reasons 'session advanced Nm ago'.
- _archive_stale_titles default label is now 'archive-<title> —' (operator
convention: all archived sessions start with archive- so they sort/filter
together in the web UI).
- launch-report.py start() archives older cc-ci-report sessions before launch,
same invariant as upgrader/supervisor (reuses the launch-upgrader helper).
- 33 sessions restyled/archived live; canonical names now unique:
cc-ci-upgrader (idle finisher), cc-ci-report (generating), cc-ci-supervisor
(none — only exists during a rescue).
_archive_stale_titles() generalized to (title, label); launch-supervisor's
spawn_supervisor() archives every older 'cc-ci-supervisor' session before
launching, so exactly one session carries the name. 11 historical supervisor
sessions archive-renamed live ('supervisor archive — <date> <time>').
Bug (2026-08-04 16:00): _session_id() sorted candidates on (s.time.created) which the
/session API rows DON'T carry — every key was 0, 'newest' degraded to server list
order, and the watchdog resumed the old giant unresumable session, kill_session()ing
the healthy fresh run mid-work.
Fixes:
- Pin the managed session id at launch/resume to LOG_DIR/.{SESSION}-session-id;
_session_id() prefers the pin, validated via direct GET /session/<id> (the LIST is
paginated ~100 rows, membership scans lie). Title lookup is only the fallback and
now sorts on authoritative sqlite time_created.
- _archive_stale_titles() at start: every older top-level session titled
cc-ci-upgrader is renamed 'upgrader archive — weekly <date>', so EXACTLY ONE
session ever carries the canonical name in the opencode web UI (easy to find;
finished runs stay browsable under archive names). 11 historical sessions
renamed live today; the in-flight finisher pinned.
Verified live: _session_id() returns the pinned running session; tree-idle 0.0min
while subagents active. Full synthetic-stall watchdog confirmation queued post-run
(task #13).
Covers the 2.32.4 -> 2.33.3 range for the 2026-08-03 /upgrade-all run:
2.33.0 minor features (admin-managed instance creds, workflow review
requests + publish/unpublish API, API deprecation of activate/deactivate
endpoints, optional N8N_SCHEDULER_MAX_ATTEMPTS env) and the 2.33.1-2.33.3
patches; notes 2.34.0 exists but is not this run's target.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Switch the cc-ci-orchestrator agent (cc-ci-plan/agents.toml) from the claude
backend to the opencode backend so it attaches to the shared opencode web
server (opencode-web.service, 127.0.0.1:4096) and is reachable for Remote
Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project.
Model = opencode/glm-5.2 (same provider family as the weekly upgrader /
supervisor). resume=false — the opencode backend has no --resume equivalent
(a fresh session per launch, matching the existing launch-opencode.sh shape).
Add a cc-ci-orchestrator.service systemd unit so on boot it runs
'agents.py up orchestrator' after opencode-web.service is up — the same
reboot-resilience shape lichen-orchestrator / project-orchestrator already
have. KillMode=process so a rebuild that merely touches this unit does not
tear down the shared tmux server and every agent session with it.
No secrets: the diff is only config + comments (paths, model names, unit
definition).
GO subscription hit a monthly usage limit (resets in 9 days). Add LOOP_TIER
env (default 'zen') to launch-upgrader.py + launch-report.py so the opencode
backend can use either subscription: zen→opencode/glm-5.2, go→opencode-go/glm-5.2.
The tier selects the default model, the usage-limit probe endpoint, and the
auth.json key. The systemd timer reads LOOP_TIER from /srv/cc-ci/upgrader.env.
A run can finish the recipe work + summary + report yet never print a clean
sign-off marker (it may wedge afterwards, as on 2026-07-03). Without a second
completion signal the supervisor would try to 'resume' the already-finished run
every hour until the 96h window closed. Now the gate also treats the run as done
when a weekly summary file (upgrade-all-<date>.md, the Step-5 output) exists with
an mtime after the session began.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
The 2026-07-03 finish run wedged because _completed() returned True while the run
was still mid-work — so the watchdog exited early and nothing recovered the wedge.
Cause: it scanned part.get('text') across ALL message parts, so DONE_MARKER inside
a TOOL part (a subagent `task` prompt / bash command that referenced 'print
UPGRADE RUN COMPLETE') matched. Now: require the marker in the LAST assistant
TEXT (prose) message — the genuine sign-off — ignoring tool-call args and any
mid-run echo of the instruction (work after the echo disqualifies it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
More fixes from live-running the finish of the 2026-07-03 run:
- _session_idle_min(): measure staleness across the whole session TREE (top-level
run + all descendant subagents), via the opencode server time.updated, NOT the
tmux log mtime (which freezes when a headless run doesn't stream to the pane).
A per-recipe subagent deploy runs 20-40min during which the PARENT session's
updated time is stale — reading the parent alone looked 'idle' and would false-
resume, killing the productive run. Renamed from _log_idle_min (kept as alias).
- watchdog(): wrap each poll in try/except so a transient server blip/race can
never kill the watchdog (a dead watchdog silently abandons the run).
- watchdog + supervisor now read the tree-aware session idle instead of log mtime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Live-testing the resume path surfaced two gaps: (1) an `opencode run` proc
EXITS when the model ends its turn, so a long /upgrade-all run's process dies
repeatedly before the whole run completes — and the log mtime freezes on death,
so the watchdog's log-idle>15min signal is both too slow and unreliable. (2) A
resumed run had no watchdog, so nothing re-continued it.
- watchdog(): detect PROC-DEATH (no live `opencode run` proc for the session +
not completed) and resume promptly, in addition to log-idle. Guarded by
MAX_RESUMES (default 20) so a no-progress loop (e.g. disk-full) eventually hands
off to the supervisor/operator instead of spinning forever.
- resume(): auto-spawn a watchdog if none is alive (skips when the watchdog itself
called resume — it lives in {SESSION}-watchdog — so no duplicate).
- launch-supervisor.py gate: defer while the per-run watchdog is alive (it is the
single writer for prompt-recovery). The supervisor only takes over once the
watchdog gives up (MAX_RESUMES) — i.e. a wedge a bare resume can't fix. Removes
the supervisor/watchdog double-resume race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.
- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
(unused AND older than a week, so this week's likely-reused images stay) before
each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
deterministic gate — no-ops (zero tokens) when the run is complete or
progressing; only when a run stalled/died before completing does it launch a
short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Two gaps for the scheduled Thursday glm-5.2 run:
1. Survival: the watchdog was a Popen child of the Type=oneshot service, which
systemd's cgroup cleanup kills on exit. Spawn it under the persistent tmux
server instead (_spawn_watchdog), like the run sessions — survives the oneshot.
2. The report runs on glm-5.2 sharing the same opencode-go budget the upgrade run
drains, so it can 429-stall with no recovery. launch-report.py now spawns the
SAME watchdog pointed at the cc-ci-report session (generic via UPGRADER_SESSION/
_MODEL/_DONE_MARKER/_RESUME_FILE), with a report-specific resume prompt.
Also: _run_pids() is now scoped to the managed session (title or -s <sid>) so the
report watchdog can't kill the idle upgrader process and vice-versa; resume() adds
--dir and honors a custom resume prompt file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_completed() grepped the log for UPGRADE RUN COMPLETE, but the kickoff/resume
PROMPT (a user message) contains that string verbatim, so it false-positived
'done' while the run was still going. Check the model's ASSISTANT message output
via the web server API instead (log grep only as an offline, prompt-excluding
fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Subcommand, function, env (UPGRADER_WATCHDOG), and log file renamed; behavior
unchanged. Only the opencode upgrader 'start' auto-spawns it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The opencode-go subscription's rolling usage-limit (429) ends the 'opencode run'
agent loop mid-run; it does NOT self-resume. Add:
- resume: continue the SAME session (context preserved) via 'opencode run -s <id>
--continue' — finds the session from the web server, kills the idle proc safely
(via /proc scan, never pkill -f self-match), relaunches in the tmux session.
- babysit: poll the session log; on a stall (>15min idle) wait out any 429
retry-after then auto-resume. Spawned automatically by an opencode 'start'.
So a usage-limit pause now self-heals instead of needing a manual nudge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
autonomic-bot
cc-ci-bot
cc-ci
cc-ci-bot
