watchdog: a live build is proof of life — don't let waiting_until_max guillotine a long legitimate run
Found in production: rust-mutation emitted WAITING-UNTIL 3h11m out for a cargo-mutants run over the whole workspace (a genuinely multi-hour job, cargo running the whole time), but the marker branch skipped the build-aware check entirely and went straight to the cap — so the 7200s cap would have killed a live run and thrown away hours of work. The cap exists to catch a session that PARKED itself and is stuck; a build still running under the session is proof it is not. Now the cap only fires when idle exceeds it AND no build is running. The stated deadline remains the hard bound either way, so a runaway still cannot park forever. Tests: cap-reboots-when-no-build, cap-yields-to-a-live-build, past-deadline-reboots-even-with-a-build. 68 pass.
This commit is contained in:
@@ -582,10 +582,14 @@ def stall_check_one(cfg, agent):
|
||||
# of killing it mid-run. Cap how far out it can push its own reboot, so a runaway can't park
|
||||
# itself forever ("some max no matter what").
|
||||
wu_max = int(cfg["watchdog"].get("waiting_until_max", 7200))
|
||||
if wu_max and idle > wu_max:
|
||||
# idle here ≈ how long the pane has sat quiet since the agent emitted the marker; once
|
||||
# that exceeds the cap we reboot no matter how far out the stated deadline is.
|
||||
reason = f"WAITING-UNTIL exceeded the {wu_max}s cap (idle {int(idle)}s) — rebooting regardless"
|
||||
# The cap guards against a session that parked itself and is genuinely stuck. A build still
|
||||
# running under the session (cargo-mutants / a coverage run / a remote ssh) is proof of life —
|
||||
# those legitimately run for hours — so it defers to the agent's stated deadline instead of
|
||||
# being guillotined by the cap. Past the deadline (+grace) it reboots either way, so a runaway
|
||||
# can never park forever.
|
||||
if wu_max and idle > wu_max and not _build_running(cfg, agent):
|
||||
reason = (f"WAITING-UNTIL exceeded the {wu_max}s cap (idle {int(idle)}s, no build running) "
|
||||
f"— rebooting regardless")
|
||||
elif now <= until + grace:
|
||||
return
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user