THE BUG THAT PROMPTED THIS. tangled_pr.py judged success ONLY by an HX-Redirect header on the POST.
A create that SUCCEEDED but answered without that header read as a failure, so the caller retried
and Tangled grew duplicates — that is exactly how #397, #398 and #399 were filed for one branch. A
response header describes what the server meant to say; it is not the artifact.
Now it checks the artifact, in both directions:
* BEFORE posting, refuse if an open pull already exists for this source branch, naming it. A
retry cannot duplicate, whatever the response said. (--allow-duplicate to override.)
* AFTER posting, confirm against the pulls list: a new pull number that did not exist before,
whose page names this source branch, IS the success — with or without a redirect header.
* Failure is reported only when no such pull appeared. A false failure is worse than a loud
error here, because the caller's remedy is to retry.
Verified live: a dry-run against a branch that already has a pull refuses with rc=3, naming #417.
TWO REAL DEFECTS FOUND BY WRITING THE TESTS.
agents.py shelled out to `pgrep -P` and `ps -o comm=`. Neither is on the agent PATH on this host,
and a missing binary under shell=True returns rc=127 with EMPTY stdout — indistinguishable from
"this process has no children" and "no build is running". So _build_running was ALWAYS False and
the stall detector could reboot an agent mid-build. Both now read /proc directly: no PATH
dependency, and it cannot fail silently in that direction.
That shipped because the unit tests MOCKED pgrep and ps. The fakes stood in for the broken
dependency, so the suite passed on a host where neither tool was reachable and never exercised the
real path. The tests now patch _proc_descendants and _comms — the seams this repo owns. A test that
mocks a dependency proves the mock works.
Also fixed a monkeypatch leak those tests had: restoration used a name derivation that silently
matched nothing, so the patch escaped into another test class and failed an unrelated test — only
in a full run, never when that test ran alone. Now addCleanup, which cannot be ordered wrong.
NEW: tests/test_tools.py, 24 tests over tangled_pr, tangled_pr_close and gateway-domain, with every
HTTP boundary injected so they run offline. Mutation-checked: breaking classify(), the pull-number
regex, the branch match, or the scan bound each turns the suite red. Suite is 93 tests, green, and
order-stable across repeated runs.
README: a "PATH on a NixOS host" section. Every one of ps, pgrep, free, cmp, awk, curl, diff,
strings, nm, getent and ping is INSTALLED here and simply not on the agent PATH, so each reports
"command not found" and reads as a missing package. Documents how to check before concluding a tool
is absent, how to add the system profile, `nix shell` for what is genuinely missing, and the rule
that harness code should not shell out for what the kernel already exposes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
wake_elapsed was built once at startup and thereafter only ever shrank — the
loop already handled an agent whose `wake` was REMOVED mid-run, but a wake
ADDED to agents.toml while the watchdog was running was invisible to it,
silently and permanently. The config is re-read every tick, which makes the
wake look live when it is not.
Found on lichen-orchestrator: both flat-file agents were given 30-minute wakes
on 2026-08-16 against a watchdog process that had been up since 2026-08-01.
Neither wake ever fired — zero "waking rust-flat-file" lines in two days of log
— and with watch="heal" (no stall-reboot) each finished turn parked the agent
until a human noticed. A supervisor script had to stand in as their wake.
New wakes are seeded at 0 so they fire on their own schedule rather than
immediately, and the registration is logged so a silent wake is visible next
time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E3aQXnUnx9kncNHQi3c92f
Completes b73af35. That commit stopped the cap from killing a live build, but the OTHER branch still
rebooted past the stated deadline regardless of a running build — and an agent blocked on a shell cannot
re-emit a fresh marker to extend its estimate. So a cargo-mutants run that simply overran its guess would
still be killed at the guess, throwing the work away. The deadline is an ESTIMATE; a running build is a
FACT. Order is now: absolute cap (waiting_until_max) > live build > stated deadline. The cap is the one
bound that reboots even mid-build — which is exactly how a genuinely hung build gets caught, so a runaway
still cannot park forever. Tests: live-build-defers-past-deadline, past-deadline-no-build-reboots,
cap-is-absolute-and-reboots-a-hung-build. 68 pass.
Found in production: rust-mutation emitted WAITING-UNTIL 3h11m out for a cargo-mutants run over the whole
workspace (a genuinely multi-hour job, cargo running the whole time), but the marker branch skipped the
build-aware check entirely and went straight to the cap — so the 7200s cap would have killed a live run and
thrown away hours of work. The cap exists to catch a session that PARKED itself and is stuck; a build still
running under the session is proof it is not. Now the cap only fires when idle exceeds it AND no build is
running. The stated deadline remains the hard bound either way, so a runaway still cannot park forever.
Tests: cap-reboots-when-no-build, cap-yields-to-a-live-build, past-deadline-reboots-even-with-a-build. 68 pass.
_parse_waiting_until scanned only the pane's last non-empty line for footer_ui backends (claude/
opencode) — but their input-box footer always renders BELOW the agent's final message, so the marker
was never seen and WAITING-UNTIL was effectively dead for claude agents. It's only consulted once the
pane is already idle, so scan the whole capture and take the most-recent marker (the footer never
contains it). Add waiting_until_max (default 7200s) so an agent can't park its own reboot forever.
Tests: footer-honors-marker-above-footer, takes-most-recent, defer + cap in stall_check_one; make the
stall harness's patch() idempotent so a re-patched name doesn't leak into tearDown. 66 pass.
A silent pane whose claude session has a live descendant compile/coverage/test process (cargo, rustc,
cc1, llvm-cov, lichen-server, chromium, …) is a running build, not a stall. _build_running() inspects
ONLY the descendants of that session's pane_pid (never the claude root, whose args embed the prompt),
matching process comm. Defers the kill+reboot until the build finishes, but never past stall_idle_max
(default 1800s) so a hung build still recovers. Configurable via build_procs_re / stall_idle_max.
The default active_re matched a bare middle-dot+number, which also matches the TUI idle footer's
'· 3 shells' / '· 1 shell still running'. Any background shell then read as ACTIVE, masking a genuine
idle from the stall detector — the agent hung at an empty/stranded prompt and never rebooted. Drop the
timer token; genuine activity is 'esc to interrupt'/'Running tool' plus the log-recently-touched grace.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
pipeline_check wrote PIPELINE-COMPLETE the first tick all-then-configured stage markers existed, and
never cleared it. Appending stages to an already-completed pipeline (as the orchestrator did — adding
rust-linecov-e2e then two -realpds stages after the fork sequence finished) left a stale 'done' sentinel:
the new stages ran, but the file lied that the pipeline was complete, misleading operators reading state.
Now recompute the sentinel every tick from ALL current stages' markers: write it only when every marker
exists, and CLEAR a stale one the moment any stage is incomplete — so appending stages self-corrects.
Gated on the markers directly, not 'active is None' (which is also None when a stage names a missing agent).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The pipeline retires a stage the instant its completion marker appears; if the agent wrote the marker
before its final 'git push' landed, its last commits were stranded locally (observed: a review stage
left 7 unpushed commits). Now, before retiring a COMPLETED stage, the watchdog does a best-effort
'git push origin HEAD:main' in that stage's dir (never raises; 90s timeout). Guarantees the deliverable
reaches the remote.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Adds a [pipeline] block: an ordered sequence of DISTINCT agents (different prompt/model/dir) run one
at a time, advancing when each writes its completion marker — the standalone-agent analog of the
[loop] phase machine (which drives FIXED loop agents via ## DONE). The watchdog reconciles it every
tick, stateless (markers are the source of truth): exactly the first not-yet-complete stage runs,
earlier stages are retired, the active stage is stall/heal-watched. Pipeline agents are enabled=false
(the pipeline owns their lifecycle).
Also fixes a latent watchdog crash: wake_elapsed is built once at startup but config is re-read each
tick, so removing an agent's 'wake' mid-run (e.g. winding down a wake) hit agent['wake'] -> KeyError
and killed the whole watchdog silently (stalling phase advancement + stall recovery). Now skips agents
whose wake was removed.
IDEAS.md: note that [pipeline] and [loop].phases are the same shape and should be unified via an
optional per-phase agent/dir/done (fold pipeline into the phase machine, delete the parallel path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The watchdog gated ALL scheduled wakes behind `if not seq_done`, so once a phase
sequence completed, even a persistent operator-facing supervisor stopped waking.
That breaks follow-on supervision (e.g. a second build started after the first
sequence finishes). Now: loop-tied wakes (on-demand auditor etc.) still quiet after
completion, but persistent agents keep waking — their hourly supervision survives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Recurring stall: a phase is substantively complete (all DoD gates PASS from both
adversaries, no veto) but the builder never writes the done marker, so auto-advance
cannot fire and the loops idle. A blunt stall kill+reboot does not fix it (the
re-kickoffed agent just re-idles).
On a stall, if the agent is a loop agent and the current phase is NOT marked done,
send a one-time DONE-nudge (ping) telling it to write the done marker IF the DoD is
met (both adversaries PASS, no veto), giving a fresh idle window; only escalate to
the kill+reboot if it stays stalled. One nudge per phase (cleared on phase advance).
Gated by [loop].done_nudge (default true); message uses the configured done_marker
and the phase status file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
wake_agent only re-prompted a live persistent/loop session and returned False for a
dead one, so a "task" agent (one-shot, exits after its run) could not be re-run on a
schedule — its wake never fired. Now, for kind=="task", a wake kills+restarts the
task for a clean re-run (skipping only while its previous run is still active). This
makes scheduled work like a coverage audit recur autonomously, no operator trigger.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Stall detection tracked idle time in an in-memory _idle_since map seeded to now()
on first observation, so a freshly-(re)started watchdog reset every agent's stall
clock and had to wait a full stall_idle before it could nudge — an agent idle for
an hour looked freshly-idle after a watchdog restart. Seed from the tmux
window's last-activity timestamp (#{window_activity}) instead, so idle duration
reflects the agent's real last activity regardless of when the watchdog started.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The per-gate functions were inserted immediately after token_phase_flush's log
line, which split the function: its trailing re-baseline block (the
'if next_phase_id is not None: ...' that re-seeds the per-phase baseline for the
next phase, or finalizes when None) was orphaned onto the end of gate_token_check,
where next_phase_id is undefined. The watchdog therefore crashed with NameError on
the first tick of every start. Move that block back into token_phase_flush (where
next_phase_id/cur/sf are in scope) and end gate_token_check at its log line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Two watchdog/metrics improvements to the loop machine:
1) Token-logging granularity is configurable via [watchdog].token_granularity:
'gate' (default) or 'phase'. In 'gate' mode, tokens are attributed to each
claimed gate -- any 'claim(<label>)' commit on the work repo's origin/main
(e.g. claim(D1-D5), claim(feat:multi-file); a leading 'feat:' is stripped) --
in addition to the per-phase rollup, appended to token-log.jsonl tagged
phase_id='<phase>:<label>'. A change in the most-recently-claimed label is the
boundary; the in-flight gate is also flushed when the phase ends. 'phase' mode
keeps the original per-phase-only behaviour.
2) Phase auto-advance is now evaluated on EVERY signal tick instead of only the
heavy tick, so a completed phase advances within signal_interval of its
'## DONE' landing rather than idling up to heavy_interval. Healing stays on the
heavy cadence.
Note: gate-boundary detection assumes the loop's 'claim(<label>)' commit convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Multi-reviewer setups (e.g. a correctness + a readability adversary) can now have
the watchdog ping ALL reviewers on a claim, each in its own session with its own
submit key. A bare string still works (single agent). _ping_agents() helper.
When [watchdog].log_tokens (or [loop].log_tokens) is true, the watchdog records
for each phase how many tokens each agent used (and the total) and how long the
phase took, appended to <log_dir>/token-log.jsonl. Tokens are summed from each
agent's session transcript, attributed by working dir. View with `agents.py
tokens`. Baseline snapshot at phase start + delta at phase advance/complete;
robust across watchdog restarts. Validated: the transcript sum matches an
independent external collector exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
notplants
autonomic-botandClaude Opus 4.8