Found in production: rust-mutation emitted WAITING-UNTIL 3h11m out for a cargo-mutants run over the whole
workspace (a genuinely multi-hour job, cargo running the whole time), but the marker branch skipped the
build-aware check entirely and went straight to the cap — so the 7200s cap would have killed a live run and
thrown away hours of work. The cap exists to catch a session that PARKED itself and is stuck; a build still
running under the session is proof it is not. Now the cap only fires when idle exceeds it AND no build is
running. The stated deadline remains the hard bound either way, so a runaway still cannot park forever.
Tests: cap-reboots-when-no-build, cap-yields-to-a-live-build, past-deadline-reboots-even-with-a-build. 68 pass.
_parse_waiting_until scanned only the pane's last non-empty line for footer_ui backends (claude/
opencode) — but their input-box footer always renders BELOW the agent's final message, so the marker
was never seen and WAITING-UNTIL was effectively dead for claude agents. It's only consulted once the
pane is already idle, so scan the whole capture and take the most-recent marker (the footer never
contains it). Add waiting_until_max (default 7200s) so an agent can't park its own reboot forever.
Tests: footer-honors-marker-above-footer, takes-most-recent, defer + cap in stall_check_one; make the
stall harness's patch() idempotent so a re-patched name doesn't leak into tearDown. 66 pass.
A silent pane whose claude session has a live descendant compile/coverage/test process (cargo, rustc,
cc1, llvm-cov, lichen-server, chromium, …) is a running build, not a stall. _build_running() inspects
ONLY the descendants of that session's pane_pid (never the claude root, whose args embed the prompt),
matching process comm. Defers the kill+reboot until the build finishes, but never past stall_idle_max
(default 1800s) so a hung build still recovers. Configurable via build_procs_re / stall_idle_max.
The default active_re matched a bare middle-dot+number, which also matches the TUI idle footer's
'· 3 shells' / '· 1 shell still running'. Any background shell then read as ACTIVE, masking a genuine
idle from the stall detector — the agent hung at an empty/stranded prompt and never rebooted. Drop the
timer token; genuine activity is 'esc to interrupt'/'Running tool' plus the log-recently-touched grace.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
pipeline_check wrote PIPELINE-COMPLETE the first tick all-then-configured stage markers existed, and
never cleared it. Appending stages to an already-completed pipeline (as the orchestrator did — adding
rust-linecov-e2e then two -realpds stages after the fork sequence finished) left a stale 'done' sentinel:
the new stages ran, but the file lied that the pipeline was complete, misleading operators reading state.
Now recompute the sentinel every tick from ALL current stages' markers: write it only when every marker
exists, and CLEAR a stale one the moment any stage is incomplete — so appending stages self-corrects.
Gated on the markers directly, not 'active is None' (which is also None when a stage names a missing agent).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The pipeline retires a stage the instant its completion marker appears; if the agent wrote the marker
before its final 'git push' landed, its last commits were stranded locally (observed: a review stage
left 7 unpushed commits). Now, before retiring a COMPLETED stage, the watchdog does a best-effort
'git push origin HEAD:main' in that stage's dir (never raises; 90s timeout). Guarantees the deliverable
reaches the remote.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Adds a [pipeline] block: an ordered sequence of DISTINCT agents (different prompt/model/dir) run one
at a time, advancing when each writes its completion marker — the standalone-agent analog of the
[loop] phase machine (which drives FIXED loop agents via ## DONE). The watchdog reconciles it every
tick, stateless (markers are the source of truth): exactly the first not-yet-complete stage runs,
earlier stages are retired, the active stage is stall/heal-watched. Pipeline agents are enabled=false
(the pipeline owns their lifecycle).
Also fixes a latent watchdog crash: wake_elapsed is built once at startup but config is re-read each
tick, so removing an agent's 'wake' mid-run (e.g. winding down a wake) hit agent['wake'] -> KeyError
and killed the whole watchdog silently (stalling phase advancement + stall recovery). Now skips agents
whose wake was removed.
IDEAS.md: note that [pipeline] and [loop].phases are the same shape and should be unified via an
optional per-phase agent/dir/done (fold pipeline into the phase machine, delete the parallel path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The watchdog gated ALL scheduled wakes behind `if not seq_done`, so once a phase
sequence completed, even a persistent operator-facing supervisor stopped waking.
That breaks follow-on supervision (e.g. a second build started after the first
sequence finishes). Now: loop-tied wakes (on-demand auditor etc.) still quiet after
completion, but persistent agents keep waking — their hourly supervision survives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Recurring stall: a phase is substantively complete (all DoD gates PASS from both
adversaries, no veto) but the builder never writes the done marker, so auto-advance
cannot fire and the loops idle. A blunt stall kill+reboot does not fix it (the
re-kickoffed agent just re-idles).
On a stall, if the agent is a loop agent and the current phase is NOT marked done,
send a one-time DONE-nudge (ping) telling it to write the done marker IF the DoD is
met (both adversaries PASS, no veto), giving a fresh idle window; only escalate to
the kill+reboot if it stays stalled. One nudge per phase (cleared on phase advance).
Gated by [loop].done_nudge (default true); message uses the configured done_marker
and the phase status file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
wake_agent only re-prompted a live persistent/loop session and returned False for a
dead one, so a "task" agent (one-shot, exits after its run) could not be re-run on a
schedule — its wake never fired. Now, for kind=="task", a wake kills+restarts the
task for a clean re-run (skipping only while its previous run is still active). This
makes scheduled work like a coverage audit recur autonomously, no operator trigger.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Stall detection tracked idle time in an in-memory _idle_since map seeded to now()
on first observation, so a freshly-(re)started watchdog reset every agent's stall
clock and had to wait a full stall_idle before it could nudge — an agent idle for
an hour looked freshly-idle after a watchdog restart. Seed from the tmux
window's last-activity timestamp (#{window_activity}) instead, so idle duration
reflects the agent's real last activity regardless of when the watchdog started.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The per-gate functions were inserted immediately after token_phase_flush's log
line, which split the function: its trailing re-baseline block (the
'if next_phase_id is not None: ...' that re-seeds the per-phase baseline for the
next phase, or finalizes when None) was orphaned onto the end of gate_token_check,
where next_phase_id is undefined. The watchdog therefore crashed with NameError on
the first tick of every start. Move that block back into token_phase_flush (where
next_phase_id/cur/sf are in scope) and end gate_token_check at its log line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Two watchdog/metrics improvements to the loop machine:
1) Token-logging granularity is configurable via [watchdog].token_granularity:
'gate' (default) or 'phase'. In 'gate' mode, tokens are attributed to each
claimed gate -- any 'claim(<label>)' commit on the work repo's origin/main
(e.g. claim(D1-D5), claim(feat:multi-file); a leading 'feat:' is stripped) --
in addition to the per-phase rollup, appended to token-log.jsonl tagged
phase_id='<phase>:<label>'. A change in the most-recently-claimed label is the
boundary; the in-flight gate is also flushed when the phase ends. 'phase' mode
keeps the original per-phase-only behaviour.
2) Phase auto-advance is now evaluated on EVERY signal tick instead of only the
heavy tick, so a completed phase advances within signal_interval of its
'## DONE' landing rather than idling up to heavy_interval. Healing stays on the
heavy cadence.
Note: gate-boundary detection assumes the loop's 'claim(<label>)' commit convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Multi-reviewer setups (e.g. a correctness + a readability adversary) can now have
the watchdog ping ALL reviewers on a claim, each in its own session with its own
submit key. A bare string still works (single agent). _ping_agents() helper.
When [watchdog].log_tokens (or [loop].log_tokens) is true, the watchdog records
for each phase how many tokens each agent used (and the total) and how long the
phase took, appended to <log_dir>/token-log.jsonl. Tokens are summed from each
agent's session transcript, attributed by working dir. View with `agents.py
tokens`. Baseline snapshot at phase start + delta at phase advance/complete;
robust across watchdog restarts. Validated: the transcript sum matches an
independent external collector exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
notplants
autonomic-botandClaude Opus 4.8