watchdog: reboot idle-wedged loops via self-reported WAITING-UNTIL markers

The builder wedged at the context limit (garbled output) — alive but matching
none of heal_session's signals (dead/FATAL/limit), so the watchdog left it
stuck. Fix: loops now declare every wait, and the watchdog reboots a wait that
never resumes.

- plan.md §7 + both prompts: cap every wait at 10 min (chunk longer waits);
  before going idle, the loop's FINAL line must be `WAITING-UNTIL: <ISO8601 UTC>`
  (the resume time, matching its ScheduleWakeup); run /compact proactively at
  ~80% context to avoid wedging near the limit.
- launch.sh: new stall_check (runs every 30s signal tick) — reboots a loop idle
  >= STALL_IDLE (300s) when it has NO current WAITING-UNTIL marker as its last
  message OR is past the time the marker named; a healthy paced wait (marker
  present, before its time) is left alone. Complements heal_session's
  dead/FATAL/limit cases. Reboot is safe — loops re-orient from git + STATUS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-05-29 19:05:29 +01:00
co-authored by Claude Opus 4.8
parent 62b7af7a97
commit e8c4330ce3
4 changed files with 80 additions and 3 deletions
+19 -1
View File
@@ -724,7 +724,9 @@ the *specific* thing. Three cases:
while blocked and trust the ping — but keep a **fallback self-poll on a modest cadence (~24 min)**
in case a ping is missed (a dead session is restarted by the watchdog and re-orients from the repo
anyway). The goal: a pending handoff resolves in well under a minute, not a whole idle interval.
3. **Genuinely idle, nothing pending from either loop** → sleep ~1015 min, then re-orient.
3. **Genuinely idle, nothing pending from either loop** → sleep in chunks of **at most 10 min**, then
re-wake and re-orient; if still nothing, sleep another ≤10 min. **Never a single wait > 10 min**
(600 s) — see the liveness rule below.
Notes: **The Adversary may idle freely when nothing is pending — it should NOT pointlessly re-verify
or busy-poll to look busy.** It gets woken by the watchdog the instant the Builder claims a gate, so
@@ -733,6 +735,22 @@ spinning. **The Builder** should prefer keeping an unblocked backlog item in han
*fully* blocked on a gate; only hit case 2 when everything is genuinely gated behind the pending
verification — and then rely on the watchdog ping (+ fallback poll) rather than a long idle.
**Liveness marker & max-wait (the watchdog ENFORCES this).** Every wait is capped at **10 minutes**;
to wait longer, wake at 10 min, re-check, and wait again. **Immediately before going idle for any
wait, your FINAL output line MUST be exactly:**
WAITING-UNTIL: <ISO-8601 UTC>
— the moment you intend to resume (≤10 min out, matching your `ScheduleWakeup`). Compute it from the
clock, e.g. `date -u -d '+10 min' +%FT%TZ`. The watchdog uses this to tell a healthy wait from a
wedge: if it sees a loop **idle ≥5 min with no current `WAITING-UNTIL` marker as its last message, OR
idle past the time the marker named, it kills + reboots that loop** (which then re-orients from git +
its STATUS/REVIEW files). So always leave a fresh marker before sleeping, and never overrun it.
**Proactive compaction.** If your context usage climbs high (≳80%), run `/compact` *before*
continuing — your state lives in git + the phase STATUS/REVIEW files, so compaction is lossless for
the loop and prevents wedging (garbled output, failed tool calls) near the context limit.
**Anti-drift guards.**
- Cap retries: if an approach fails 3× the same way, stop, write the dead-end in `DECISIONS.md`,
and try a different approach or mark blocked. No thrashing.