supervisor: fix 3-day progress-gate deadlock (2026-08-07 run)

Root cause chain, all confirmed on the live system:
1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's
   kickoff PROMPT is an argv element, and the supervisor's prompt text contains the
   literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a
   live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>).
2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its
   process alive and SPINNING while emitting nothing (verified: 3 days, zero session
   output, still burning CPU). Progress now REQUIRES the session tree to have advanced
   within STALL_MIN; a live-but-idle proc is explicitly logged as stalled.
Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly
run sat unfinished and unreported.

Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may
resume when the wall lifts and their context is the run's state. New _billing_blocked()
detects the wall from the log tail and the gate surfaces
'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'.

Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while
still matching the real finisher; gate now reasons 'session advanced Nm ago'.
This commit is contained in:
autonomic-bot
2026-08-10 15:17:02 +00:00
parent d101147b93
commit d441c6caaf
2 changed files with 56 additions and 11 deletions
+18 -3
View File
@@ -143,9 +143,24 @@ def _gate():
# a live `opencode run … -s <sid> --attach` proc, or a log touched within the stall window.
pids = lu._run_pids(sid)
idle = lu._session_idle_min()
if pids or (idle is not None and idle < lu.STALL_MIN):
via = f"{len(pids)} live run proc(s)" if pids else f"log idle {idle:.0f}m < {lu.STALL_MIN:.0f}m"
return False, sid, f"upgrader run progressing ({via}) — leaving it"
advancing = idle is not None and idle < lu.STALL_MIN
# PROGRESS REQUIRES THE SESSION TO BE ADVANCING — a live proc alone is not enough. A run walled
# by the provider keeps its process alive and spinning while emitting nothing (2026-08-07: 3
# days, zero output). Treating "proc exists" as progress is what deadlocked the gate.
if advancing:
return False, sid, f"upgrader run progressing (session advanced {idle:.0f}m ago) — leaving it"
if pids and lu._billing_blocked():
# NEVER kill it: it may resume when the wall lifts, and its context is the run's state.
# Surface it loudly instead — this is an operator-actionable condition, not self-healing.
return False, sid, (
f"run BLOCKED on a provider billing/usage wall ({len(pids)} proc(s) alive, session idle "
f"{idle:.0f}m) — NOT killing (resumable once the wall lifts); operator action required"
)
if pids:
log(
f"note: {len(pids)} live run proc(s) but session idle {idle:.0f}m ≥ "
f"{lu.STALL_MIN:.0f}m — treating as stalled, not progressing"
)
# The per-run watchdog owns PROMPT recovery (resume on proc-death/stall) and is the single writer
# while it lives. Defer to it — it gives up (exits its tmux) only after MAX_RESUMES fail, i.e. the
# run is stuck in a way a bare resume can't fix (e.g. disk-full). THEN the supervisor takes over.