supervisor: fix 3-day progress-gate deadlock (2026-08-07 run)
Root cause chain, all confirmed on the live system: 1. _run_pids() substring-matched the WHOLE cmdline for the session name. An agent's kickoff PROMPT is an argv element, and the supervisor's prompt text contains the literal 'cc-ci-upgrader' — so the supervisor's OWN billing-hung agent matched as a live upgrader run. Now matches FLAG VALUES only (--title <SESSION> / -s <sid>). 2. The gate treated 'a live proc exists' as progress. A provider-walled run keeps its process alive and SPINNING while emitting nothing (verified: 3 days, zero session output, still burning CPU). Progress now REQUIRES the session tree to have advanced within STALL_MIN; a live-but-idle proc is explicitly logged as stalled. Result was ~60 consecutive false 'run progressing — leaving it' no-ops while the weekly run sat unfinished and unreported. Billing-walled runs are REPORTED, never killed (operator policy 2026-08-10): they may resume when the wall lifts and their context is the run's state. New _billing_blocked() detects the wall from the log tail and the gate surfaces 'run BLOCKED on a provider billing/usage wall — NOT killing; operator action required'. Verified live: _run_pids no longer matches the hung Aug-7 supervisor (1497561) while still matching the real finisher; gate now reasons 'session advanced Nm ago'.
This commit is contained in:
@@ -417,25 +417,55 @@ def _limit_retry_after():
|
||||
return 0
|
||||
|
||||
def _run_pids(sid=None):
|
||||
"""PIDs of live `opencode run` procs for THIS session (via /proc scan — never matches self)."""
|
||||
"""PIDs of live `opencode run` procs for THIS session (via /proc scan — never matches self).
|
||||
|
||||
Matches on FLAG VALUES (`--title <SESSION>` / `-s|--session <sid>`), never a substring of the
|
||||
whole cmdline. The substring form caused the 2026-08-07 three-day deadlock: an agent's kickoff
|
||||
PROMPT is passed as an argv element, and the supervisor's prompt text contains the literal
|
||||
"cc-ci-upgrader", so the supervisor's own (billing-hung) agent matched as a live upgrader run —
|
||||
the hourly gate then read its own corpse as "run progressing" and no-opped ~60 times."""
|
||||
me, out = os.getpid(), []
|
||||
for p in os.listdir("/proc"):
|
||||
if not p.isdigit() or int(p) == me:
|
||||
continue
|
||||
try:
|
||||
cl = open(f"/proc/{p}/cmdline", "rb").read().split(b"\0")
|
||||
cl = [c for c in open(f"/proc/{p}/cmdline", "rb").read().split(b"\0") if c]
|
||||
except Exception:
|
||||
continue
|
||||
joined = b" ".join(cl)
|
||||
if not (b"opencode" in joined and b"run" in cl and b"--attach" in cl):
|
||||
if not cl or b"opencode" not in cl[0] or b"run" not in cl or b"--attach" not in cl:
|
||||
continue
|
||||
# Scope to THIS managed session only: a fresh run carries `--title <SESSION>`, a resumed
|
||||
# run carries `-s <sid>`. Without this, the report watchdog would kill the idle upgrader
|
||||
# run (and vice-versa) since both are `opencode run … --attach`.
|
||||
if SESSION.encode() in joined or (sid and sid.encode() in joined):
|
||||
|
||||
def _flag(names):
|
||||
for i, a in enumerate(cl[:-1]):
|
||||
if a in names:
|
||||
return cl[i + 1]
|
||||
return None
|
||||
|
||||
title = _flag((b"--title",))
|
||||
s_val = _flag((b"-s", b"--session"))
|
||||
if title == SESSION.encode() or (sid and s_val == sid.encode()):
|
||||
out.append(int(p))
|
||||
return out
|
||||
|
||||
|
||||
def _billing_blocked(window_bytes=4000):
|
||||
"""True when the tail of this run's log shows a provider billing/usage wall.
|
||||
|
||||
Such a process must NEVER be killed: it may resume when the wall lifts, and killing it can
|
||||
lose in-flight work. It is also NOT progress — it can spin for days emitting nothing (observed
|
||||
2026-08-07..10), so the gate reports it as BLOCKED and leaves recovery to the operator."""
|
||||
try:
|
||||
with open(LOG_FILE, "rb") as f:
|
||||
f.seek(0, os.SEEK_END)
|
||||
f.seek(max(0, f.tell() - window_bytes))
|
||||
tail = f.read().decode(errors="replace").lower()
|
||||
except OSError:
|
||||
return False
|
||||
return any(
|
||||
s in tail
|
||||
for s in ("spending limit", "insufficient balance", "usage limit", "usagelimiterror")
|
||||
)
|
||||
|
||||
def _completed():
|
||||
# Done only when the MODEL signs off with DONE_MARKER as its FINAL word: the marker in the LAST
|
||||
# assistant TEXT (prose) message. This guards THREE false-positives that each abandoned a run:
|
||||
|
||||
Reference in New Issue
Block a user