weekly-run: watchdog resumes on proc-death; supervisor defers to watchdog
Live-testing the resume path surfaced two gaps: (1) an `opencode run` proc
EXITS when the model ends its turn, so a long /upgrade-all run's process dies
repeatedly before the whole run completes — and the log mtime freezes on death,
so the watchdog's log-idle>15min signal is both too slow and unreliable. (2) A
resumed run had no watchdog, so nothing re-continued it.
- watchdog(): detect PROC-DEATH (no live `opencode run` proc for the session +
not completed) and resume promptly, in addition to log-idle. Guarded by
MAX_RESUMES (default 20) so a no-progress loop (e.g. disk-full) eventually hands
off to the supervisor/operator instead of spinning forever.
- resume(): auto-spawn a watchdog if none is alive (skips when the watchdog itself
called resume — it lives in {SESSION}-watchdog — so no duplicate).
- launch-supervisor.py gate: defer while the per-run watchdog is alive (it is the
single writer for prompt-recovery). The supervisor only takes over once the
watchdog gives up (MAX_RESUMES) — i.e. a wedge a bare resume can't fix. Removes
the supervisor/watchdog double-resume race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
1bd156e7e6
commit
399e999978
@@ -127,6 +127,11 @@ def _gate():
|
||||
if pids or (idle is not None and idle < lu.STALL_MIN):
|
||||
via = f"{len(pids)} live run proc(s)" if pids else f"log idle {idle:.0f}m < {lu.STALL_MIN:.0f}m"
|
||||
return False, sid, f"upgrader run progressing ({via}) — leaving it"
|
||||
# The per-run watchdog owns PROMPT recovery (resume on proc-death/stall) and is the single writer
|
||||
# while it lives. Defer to it — it gives up (exits its tmux) only after MAX_RESUMES fail, i.e. the
|
||||
# run is stuck in a way a bare resume can't fix (e.g. disk-full). THEN the supervisor takes over.
|
||||
if lu._watchdog_alive():
|
||||
return False, sid, "per-run watchdog alive — it owns recovery; supervisor stays back"
|
||||
if _sup_alive() and _sup_busy():
|
||||
return False, sid, "a supervisor agent is already working — skip"
|
||||
idle_s = f"{idle:.0f}m" if idle is not None else "unknown"
|
||||
|
||||
Reference in New Issue
Block a user