watchdog: proof-of-life outranks the agent's own WAITING-UNTIL deadline
Completes b73af35. That commit stopped the cap from killing a live build, but the OTHER branch still
rebooted past the stated deadline regardless of a running build — and an agent blocked on a shell cannot
re-emit a fresh marker to extend its estimate. So a cargo-mutants run that simply overran its guess would
still be killed at the guess, throwing the work away. The deadline is an ESTIMATE; a running build is a
FACT. Order is now: absolute cap (waiting_until_max) > live build > stated deadline. The cap is the one
bound that reboots even mid-build — which is exactly how a genuinely hung build gets caught, so a runaway
still cannot park forever. Tests: live-build-defers-past-deadline, past-deadline-no-build-reboots,
cap-is-absolute-and-reboots-a-hung-build. 68 pass.
This commit is contained in:
@@ -582,14 +582,19 @@ def stall_check_one(cfg, agent):
|
||||
# of killing it mid-run. Cap how far out it can push its own reboot, so a runaway can't park
|
||||
# itself forever ("some max no matter what").
|
||||
wu_max = int(cfg["watchdog"].get("waiting_until_max", 7200))
|
||||
# The cap guards against a session that parked itself and is genuinely stuck. A build still
|
||||
# running under the session (cargo-mutants / a coverage run / a remote ssh) is proof of life —
|
||||
# those legitimately run for hours — so it defers to the agent's stated deadline instead of
|
||||
# being guillotined by the cap. Past the deadline (+grace) it reboots either way, so a runaway
|
||||
# can never park forever.
|
||||
if wu_max and idle > wu_max and not _build_running(cfg, agent):
|
||||
reason = (f"WAITING-UNTIL exceeded the {wu_max}s cap (idle {int(idle)}s, no build running) "
|
||||
building = _build_running(cfg, agent)
|
||||
# A build still running under the session (cargo-mutants, a coverage run, a remote ssh) is
|
||||
# PROOF OF LIFE, and it outranks the agent's own deadline: the deadline is only an estimate,
|
||||
# and an agent blocked on a shell cannot re-emit a fresh marker to extend it — so killing a
|
||||
# live build at its estimate throws the work away for nothing. `waiting_until_max` is the one
|
||||
# absolute bound: past it we reboot even mid-build, which is what catches a genuinely HUNG
|
||||
# build (the case the cap exists for).
|
||||
if wu_max and idle > wu_max:
|
||||
reason = (f"idle {int(idle)}s past the {wu_max}s WAITING-UNTIL cap "
|
||||
f"({'build still running — treating it as hung' if building else 'no build running'}) "
|
||||
f"— rebooting regardless")
|
||||
elif building:
|
||||
return
|
||||
elif now <= until + grace:
|
||||
return
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user