watchdog: a live build is proof of life — don't let waiting_until_max guillotine a long legitimate run

Found in production: rust-mutation emitted WAITING-UNTIL 3h11m out for a cargo-mutants run over the whole
workspace (a genuinely multi-hour job, cargo running the whole time), but the marker branch skipped the
build-aware check entirely and went straight to the cap — so the 7200s cap would have killed a live run and
thrown away hours of work. The cap exists to catch a session that PARKED itself and is stuck; a build still
running under the session is proof it is not. Now the cap only fires when idle exceeds it AND no build is
running. The stated deadline remains the hard bound either way, so a runaway still cannot park forever.
Tests: cap-reboots-when-no-build, cap-yields-to-a-live-build, past-deadline-reboots-even-with-a-build. 68 pass.
This commit is contained in:
2026-07-11 11:50:39 +00:00
parent 582f392ef5
commit b73af35792
2 changed files with 31 additions and 9 deletions
+8 -4
View File
@@ -582,10 +582,14 @@ def stall_check_one(cfg, agent):
# of killing it mid-run. Cap how far out it can push its own reboot, so a runaway can't park
# itself forever ("some max no matter what").
wu_max = int(cfg["watchdog"].get("waiting_until_max", 7200))
if wu_max and idle > wu_max:
# idle here ≈ how long the pane has sat quiet since the agent emitted the marker; once
# that exceeds the cap we reboot no matter how far out the stated deadline is.
reason = f"WAITING-UNTIL exceeded the {wu_max}s cap (idle {int(idle)}s) — rebooting regardless"
# The cap guards against a session that parked itself and is genuinely stuck. A build still
# running under the session (cargo-mutants / a coverage run / a remote ssh) is proof of life —
# those legitimately run for hours — so it defers to the agent's stated deadline instead of
# being guillotined by the cap. Past the deadline (+grace) it reboots either way, so a runaway
# can never park forever.
if wu_max and idle > wu_max and not _build_running(cfg, agent):
reason = (f"WAITING-UNTIL exceeded the {wu_max}s cap (idle {int(idle)}s, no build running) "
f"— rebooting regardless")
elif now <= until + grace:
return
else: