weekly-run: watchdog resumes on proc-death; supervisor defers to watchdog

Live-testing the resume path surfaced two gaps: (1) an `opencode run` proc
EXITS when the model ends its turn, so a long /upgrade-all run's process dies
repeatedly before the whole run completes — and the log mtime freezes on death,
so the watchdog's log-idle>15min signal is both too slow and unreliable. (2) A
resumed run had no watchdog, so nothing re-continued it.

- watchdog(): detect PROC-DEATH (no live `opencode run` proc for the session +
  not completed) and resume promptly, in addition to log-idle. Guarded by
  MAX_RESUMES (default 20) so a no-progress loop (e.g. disk-full) eventually hands
  off to the supervisor/operator instead of spinning forever.
- resume(): auto-spawn a watchdog if none is alive (skips when the watchdog itself
  called resume — it lives in {SESSION}-watchdog — so no duplicate).
- launch-supervisor.py gate: defer while the per-run watchdog is alive (it is the
  single writer for prompt-recovery). The supervisor only takes over once the
  watchdog gives up (MAX_RESUMES) — i.e. a wedge a bare resume can't fix. Removes
  the supervisor/watchdog double-resume race.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
This commit is contained in:
autonomic-bot
2026-07-04 04:39:40 +00:00
co-authored by Claude Opus 4.8
parent 1bd156e7e6
commit 399e999978
2 changed files with 48 additions and 14 deletions
+5
View File
@@ -127,6 +127,11 @@ def _gate():
if pids or (idle is not None and idle < lu.STALL_MIN):
via = f"{len(pids)} live run proc(s)" if pids else f"log idle {idle:.0f}m < {lu.STALL_MIN:.0f}m"
return False, sid, f"upgrader run progressing ({via}) — leaving it"
# The per-run watchdog owns PROMPT recovery (resume on proc-death/stall) and is the single writer
# while it lives. Defer to it — it gives up (exits its tmux) only after MAX_RESUMES fail, i.e. the
# run is stuck in a way a bare resume can't fix (e.g. disk-full). THEN the supervisor takes over.
if lu._watchdog_alive():
return False, sid, "per-run watchdog alive — it owns recovery; supervisor stays back"
if _sup_alive() and _sup_busy():
return False, sid, "a supervisor agent is already working — skip"
idle_s = f"{idle:.0f}m" if idle is not None else "unknown"