watchdog: register a wake added to the config mid-run
wake_elapsed was built once at startup and thereafter only ever shrank — the loop already handled an agent whose `wake` was REMOVED mid-run, but a wake ADDED to agents.toml while the watchdog was running was invisible to it, silently and permanently. The config is re-read every tick, which makes the wake look live when it is not. Found on lichen-orchestrator: both flat-file agents were given 30-minute wakes on 2026-08-16 against a watchdog process that had been up since 2026-08-01. Neither wake ever fired — zero "waking rust-flat-file" lines in two days of log — and with watch="heal" (no stall-reboot) each finished turn parked the agent until a human noticed. A supervisor script had to stand in as their wake. New wakes are seeded at 0 so they fire on their own schedule rather than immediately, and the registration is logged so a silent wake is visible next time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E3aQXnUnx9kncNHQi3c92f
This commit is contained in:
@@ -1148,6 +1148,17 @@ def watchdog_loop(cfg_path):
|
||||
if session_alive(a["session"]):
|
||||
limit_tick(cfg, a, capture_pane(a["session"], 40))
|
||||
|
||||
# ...and ADD one that appeared. wake_elapsed was built once at startup, and only ever
|
||||
# shrank (below) — so a `wake` written into agents.toml while the watchdog is running was
|
||||
# invisible to it, silently and forever. Both flat-file agents were given 30-min wakes on
|
||||
# 2026-08-16 against a watchdog up since 2026-08-01: zero wakes fired in the 2 days since,
|
||||
# and with watch="heal" (no stall-reboot) each finished turn parked them until a human
|
||||
# looked. Seed a new wake with a full interval so it fires on its own schedule, not at once.
|
||||
for _n, _a in cfg["agents"].items():
|
||||
if _a.get("wake") and _n not in wake_elapsed:
|
||||
wake_elapsed[_n] = 0
|
||||
log(f"wake registered mid-run for {_n} (interval={_a['wake'].get('interval', 3600)}s)")
|
||||
|
||||
for name, el in list(wake_elapsed.items()):
|
||||
agent = cfg["agents"].get(name)
|
||||
# Config is re-read every tick, but wake_elapsed was built once at startup. If an agent's
|
||||
|
||||
Reference in New Issue
Block a user