A secret a third party reads from a fixed path (ssh key, systemd EnvironmentFile, nix
authKeyFile, TLS keypair) now lives ONCE as a real file in /secrets/files and is symlinked
from where the consumer expects it. The consumer is unchanged and unaware; the file exists
in one directory, at 0600, outside /srv and outside every git tree.
That makes the store and the file directory alternatives, not layers: a secret is a value in
store.yaml OR a file in /secrets/files, never both. The copies of the ssh keys, tailscale
auth key, incus keypair, LE cert and cc-ci testenv have been dropped from store.yaml now that
each has a single home.
Documented exception: an app that rewrites its own credential file (OAuth refresh via
write-temp+rename) replaces the symlink with a regular file and silently re-splits the home.
opencode's auth.json is one, so it stays put and is deliberately not centralised.
Co-Authored-By: Claude <noreply@anthropic.com>
Materializing wrote a second plaintext file per secret, which is the problem the store
was meant to solve: two files drift, and the copy is what ends up committed or grepped.
- tangled_pr / tangled_pr_edit / tangled_repo now read tangled.cookie from the store.
engine/.tangled-session is deleted; --cookie-file remains as a legacy escape hatch.
- materialize() is replaced by run-time injection that leaves nothing at rest:
exec-env <group> -- cmd group as env vars (use this instead of a systemd
EnvironmentFile — same effect, no plaintext on disk)
with-file <key> -- cmd {} 0600 file in a private tmpdir, removed when cmd exits,
for consumers that insist on a path (ssh -i, a TLS key)
Co-Authored-By: Claude <noreply@anthropic.com>
/srv is the projects tree — agents grep, find and `ls -R` it constantly, so a store
under it turns up in ordinary searches and risks being read (or pasted) by accident.
/secrets sits outside that blast radius: nothing routinely walks it, and it is still
0700 loops, still not a repo, still ciphertext at rest.
Override with AO_SECRETS_STORE if a host puts it elsewhere.
Co-Authored-By: Claude <noreply@anthropic.com>
Credentials were scattered in plaintext: a gitea password baked into six git remote
URLs (`git remote -v` prints those), API keys in .env files, an incus client key at
0644. Anything living in a repo is one `git add -A` from being pushed.
So: ONE encrypted file outside every git tree, and a helper each project uses.
/srv/secrets/store.yaml sops+age ciphertext, 0600, not a repo, no remote
~/.config/sops/age/keys.txt the only plaintext secret on disk, 0600
secrets.py is stdlib + the sops binary: get("group.key"), get_group("group"), and
materialize() for consumers that must read a fixed path (systemd EnvironmentFile,
ssh IdentityFile, nix authKeyFile) — those keep their file, but the store is the
source of truth, so a materialized file is never hand-edited.
`list` prints names only, never values, so it is safe in a transcript.
Co-Authored-By: Claude <noreply@anthropic.com>
/pulls/ is 405. The appview's PR form is htmx: it only processes the create when
it sees HX-Request, otherwise it re-renders the page — a 200 that silently creates
nothing. Success comes back as HX-Redirect (…/pulls/<n>), not a 3xx Location, so
the old redirect check never fired. Also send source=branch + the title/body Dirty
flags the form expects, and point the auth-failure hint at the current cookie path.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Sibling of tangled_pr.py. GET/POST /{owner}/{repo}/pulls/{n}/edit (htmx, session
cookie); does not mint a round or touch the patch; re-fetches the form after POST
and verifies the appview holds the new text. Built and proven by the
rust-pr-desc-concise pass (see machine-docs/PR-DESC-CONCISE.md).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BMb31Hx9mnYGRXZ3AL68sk
Completes b73af35. That commit stopped the cap from killing a live build, but the OTHER branch still
rebooted past the stated deadline regardless of a running build — and an agent blocked on a shell cannot
re-emit a fresh marker to extend its estimate. So a cargo-mutants run that simply overran its guess would
still be killed at the guess, throwing the work away. The deadline is an ESTIMATE; a running build is a
FACT. Order is now: absolute cap (waiting_until_max) > live build > stated deadline. The cap is the one
bound that reboots even mid-build — which is exactly how a genuinely hung build gets caught, so a runaway
still cannot park forever. Tests: live-build-defers-past-deadline, past-deadline-no-build-reboots,
cap-is-absolute-and-reboots-a-hung-build. 68 pass.
Found in production: rust-mutation emitted WAITING-UNTIL 3h11m out for a cargo-mutants run over the whole
workspace (a genuinely multi-hour job, cargo running the whole time), but the marker branch skipped the
build-aware check entirely and went straight to the cap — so the 7200s cap would have killed a live run and
thrown away hours of work. The cap exists to catch a session that PARKED itself and is stuck; a build still
running under the session is proof it is not. Now the cap only fires when idle exceeds it AND no build is
running. The stated deadline remains the hard bound either way, so a runaway still cannot park forever.
Tests: cap-reboots-when-no-build, cap-yields-to-a-live-build, past-deadline-reboots-even-with-a-build. 68 pass.
_parse_waiting_until scanned only the pane's last non-empty line for footer_ui backends (claude/
opencode) — but their input-box footer always renders BELOW the agent's final message, so the marker
was never seen and WAITING-UNTIL was effectively dead for claude agents. It's only consulted once the
pane is already idle, so scan the whole capture and take the most-recent marker (the footer never
contains it). Add waiting_until_max (default 7200s) so an agent can't park its own reboot forever.
Tests: footer-honors-marker-above-footer, takes-most-recent, defer + cap in stall_check_one; make the
stall harness's patch() idempotent so a re-patched name doesn't leak into tearDown. 66 pass.
13 tests across 4 classes: the build-proc match set (build tools match; python/node/bash/claude and
substring look-alikes don't), _proc_descendants (real child tree, roots excluded), _build_running
(session-scoped, comm-matched, never the claude root, custom regex override), and stall_check_one's
defer/reboot/hard-cap behavior. Full suite 64 pass.
A silent pane whose claude session has a live descendant compile/coverage/test process (cargo, rustc,
cc1, llvm-cov, lichen-server, chromium, …) is a running build, not a stall. _build_running() inspects
ONLY the descendants of that session's pane_pid (never the claude root, whose args embed the prompt),
matching process comm. Defers the kill+reboot until the build finishes, but never past stall_idle_max
(default 1800s) so a hung build still recovers. Configurable via build_procs_re / stall_idle_max.
Reusable utility (not project-specific): tangled's appview only indexes pulls created through its
OAuth-authenticated web endpoint POST /{owner}/{repo}/pulls/ (it fetches the knot patch + inserts into
its DB directly); raw com.atproto.repo.createRecord does NOT index. This tool reuses a browser session
cookie (gitignored .tangled-session) to POST target/source branch names. No blob upload, no CSRF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The default active_re matched a bare middle-dot+number, which also matches the TUI idle footer's
'· 3 shells' / '· 1 shell still running'. Any background shell then read as ACTIVE, masking a genuine
idle from the stall detector — the agent hung at an empty/stranded prompt and never rebooted. Drop the
timer token; genuine activity is 'esc to interrupt'/'Running tool' plus the log-recently-touched grace.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
pipeline_check wrote PIPELINE-COMPLETE the first tick all-then-configured stage markers existed, and
never cleared it. Appending stages to an already-completed pipeline (as the orchestrator did — adding
rust-linecov-e2e then two -realpds stages after the fork sequence finished) left a stale 'done' sentinel:
the new stages ran, but the file lied that the pipeline was complete, misleading operators reading state.
Now recompute the sentinel every tick from ALL current stages' markers: write it only when every marker
exists, and CLEAR a stale one the moment any stage is incomplete — so appending stages self-corrects.
Gated on the markers directly, not 'active is None' (which is also None when a stage names a missing agent).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The pipeline retires a stage the instant its completion marker appears; if the agent wrote the marker
before its final 'git push' landed, its last commits were stranded locally (observed: a review stage
left 7 unpushed commits). Now, before retiring a COMPLETED stage, the watchdog does a best-effort
'git push origin HEAD:main' in that stage's dir (never raises; 90s timeout). Guarantees the deliverable
reaches the remote.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Adds a [pipeline] block: an ordered sequence of DISTINCT agents (different prompt/model/dir) run one
at a time, advancing when each writes its completion marker — the standalone-agent analog of the
[loop] phase machine (which drives FIXED loop agents via ## DONE). The watchdog reconciles it every
tick, stateless (markers are the source of truth): exactly the first not-yet-complete stage runs,
earlier stages are retired, the active stage is stall/heal-watched. Pipeline agents are enabled=false
(the pipeline owns their lifecycle).
Also fixes a latent watchdog crash: wake_elapsed is built once at startup but config is re-read each
tick, so removing an agent's 'wake' mid-run (e.g. winding down a wake) hit agent['wake'] -> KeyError
and killed the whole watchdog silently (stalling phase advancement + stall recovery). Now skips agents
whose wake was removed.
IDEAS.md: note that [pipeline] and [loop].phases are the same shape and should be unified via an
optional per-phase agent/dir/done (fold pipeline into the phase machine, delete the parallel path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The watchdog gated ALL scheduled wakes behind `if not seq_done`, so once a phase
sequence completed, even a persistent operator-facing supervisor stopped waking.
That breaks follow-on supervision (e.g. a second build started after the first
sequence finishes). Now: loop-tied wakes (on-demand auditor etc.) still quiet after
completion, but persistent agents keep waking — their hourly supervision survives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Recurring stall: a phase is substantively complete (all DoD gates PASS from both
adversaries, no veto) but the builder never writes the done marker, so auto-advance
cannot fire and the loops idle. A blunt stall kill+reboot does not fix it (the
re-kickoffed agent just re-idles).
On a stall, if the agent is a loop agent and the current phase is NOT marked done,
send a one-time DONE-nudge (ping) telling it to write the done marker IF the DoD is
met (both adversaries PASS, no veto), giving a fresh idle window; only escalate to
the kill+reboot if it stays stalled. One nudge per phase (cleared on phase advance).
Gated by [loop].done_nudge (default true); message uses the configured done_marker
and the phase status file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
wake_agent only re-prompted a live persistent/loop session and returned False for a
dead one, so a "task" agent (one-shot, exits after its run) could not be re-run on a
schedule — its wake never fired. Now, for kind=="task", a wake kills+restarts the
task for a clean re-run (skipping only while its previous run is still active). This
makes scheduled work like a coverage audit recur autonomously, no operator trigger.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Stall detection tracked idle time in an in-memory _idle_since map seeded to now()
on first observation, so a freshly-(re)started watchdog reset every agent's stall
clock and had to wait a full stall_idle before it could nudge — an agent idle for
an hour looked freshly-idle after a watchdog restart. Seed from the tmux
window's last-activity timestamp (#{window_activity}) instead, so idle duration
reflects the agent's real last activity regardless of when the watchdog started.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
The per-gate functions were inserted immediately after token_phase_flush's log
line, which split the function: its trailing re-baseline block (the
'if next_phase_id is not None: ...' that re-seeds the per-phase baseline for the
next phase, or finalizes when None) was orphaned onto the end of gate_token_check,
where next_phase_id is undefined. The watchdog therefore crashed with NameError on
the first tick of every start. Move that block back into token_phase_flush (where
next_phase_id/cur/sf are in scope) and end gate_token_check at its log line.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Two watchdog/metrics improvements to the loop machine:
1) Token-logging granularity is configurable via [watchdog].token_granularity:
'gate' (default) or 'phase'. In 'gate' mode, tokens are attributed to each
claimed gate -- any 'claim(<label>)' commit on the work repo's origin/main
(e.g. claim(D1-D5), claim(feat:multi-file); a leading 'feat:' is stripped) --
in addition to the per-phase rollup, appended to token-log.jsonl tagged
phase_id='<phase>:<label>'. A change in the most-recently-claimed label is the
boundary; the in-flight gate is also flushed when the phase ends. 'phase' mode
keeps the original per-phase-only behaviour.
2) Phase auto-advance is now evaluated on EVERY signal tick instead of only the
heavy tick, so a completed phase advances within signal_interval of its
'## DONE' landing rather than idling up to heavy_interval. Healing stays on the
heavy cadence.
Note: gate-boundary detection assumes the loop's 'claim(<label>)' commit convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
Multi-reviewer setups (e.g. a correctness + a readability adversary) can now have
the watchdog ping ALL reviewers on a claim, each in its own session with its own
submit key. A bare string still works (single agent). _ping_agents() helper.
Coarsest review cadence: the Builder self-certifies the build phases and the
Adversary does ONE comprehensive cold-verification of the whole accumulated build
in a final `review` phase (vs orig per-phase, lean per-gate). Full original
prompts + a DEFERRED REVIEW CADENCE override, so it isolates verification cadence.
Cheapest coordination; the trade-off is the independent check arrives late (late
rework risk + self-certification drift on build phases). README spells it out.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
So that "stateless vs builder-adversary" and "lean vs stateless" isolate context
hygiene / review granularity WITHOUT the confound of the minimal prompts' reduced
testing pressure (which we found cuts ~25% of test methods). stateless = orig +
context hygiene; lean = orig + context hygiene + per-gate review. min stays the
pure minimal-prompt variant (isolates verbosity vs orig).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single Builder that builds AND self-verifies (same DoD rigor), with NO
independent Adversary and no claim/review handoff. The control for measuring
what the AI adversary costs (its tokens, ~half of a loop-pair run) and buys
(independent cold verification vs self-certification).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When [watchdog].log_tokens (or [loop].log_tokens) is true, the watchdog records
for each phase how many tokens each agent used (and the total) and how long the
phase took, appended to <log_dir>/token-log.jsonl. Tokens are summed from each
agent's session transcript, attributed by working dir. View with `agents.py
tokens`. Baseline snapshot at phase start + delta at phase advance/complete;
robust across watchdog restarts. Validated: the transcript sum matches an
independent external collector exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Isolates the two effects conflated in builder-adversary-stateless: keeps all the
CONTEXT HYGIENE (compact/diffs/lean loads) but ENFORCES full per-gate review
granularity (one claim per gate, one independent verdict per gate, no batching).
Tests whether the token saving is real efficiency vs reduced scrutiny.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same pattern + AI-as-adversary verification as builder-adversary-min, but the
role prompts add CONTEXT HYGIENE: /compact at every checkpoint (lossless — state
is on disk), read diffs not trees, spill bulk output to files, adversary loads
only {plan, STATUS, diff}. Loop agents non-resumed → fresh session per phase.
Targets cache-read (the dominant cost in a long loop) without changing what the
agents do or how they verify.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same topology/behaviour as builder-adversary (loop pair, phase machine,
claim()/review() handoff, machine-docs coordination, cold verification) but the
role + kickoff prompts are compressed to minimal tokens, keeping every
load-bearing rule. Config and plans are unchanged. The separate
agent-orchestrator-benchmark repo runs a head-to-head token comparison.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A sketch backlog of further examples, each teaching a distinct orchestration
topology (anthill/stigmergy, kitchen line/pipeline, incident room/blackboard,
senate/debate, baton/mutex+failover, immune system/reactive, evolution chamber,
plus ATC and day-night extras). Not implemented — ideas only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Based on @ponder.ooo's "snake pit agent orchestrator" idea (bsky 2026-05-28) and
Claude's metaphor-mapping elaboration: agents are snakes, tasks are food tossed
into a shared pit; snakes devour/digest/regurgitate/excrete.
A worker-pool-over-a-shared-queue topology (contrast the builder-adversary phase
machine):
- pit/ is a filesystem queue; snakes claim by atomic mv (no two eat the same food)
- species = specialized agents: keeper (zookeeper), planner (regurgitation IS
task decomposition), snake-1..3 (worker pool), cleanup (scavenger + coprophagy)
- no [loop] phase machine; persistent agents self-pace via /loop
- README carries the full bio→compute mapping table from the thread image
Verified: `agents.py status --config agents.toml` lists all 6 agents + service.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A self-contained examples/builder-adversary/ that distills the cc-ci production
loop pair into a tiny, fully-local task (build a `wc` CLI in two phases):
- agents.toml: builder + adversary loops, persistent orchestrator, on_complete
reporter, cleanlogs service; phase machine with a per-phase model override
- prompts/: kickoff template + builder/adversary roles carrying the load-bearing
protocol (claim()/review() handoff, machine-docs file-location rule,
WHAT+HOW+EXPECTED+WHERE=STATUS / WHY=JOURNAL anti-anchoring, WAITING-UNTIL liveness)
- plans/: two phase plans (wc, json) each with a cold-verifiable Definition of Done
- README: how to run, the work-repo two-clone isolation model, how to adapt
Verified: `agents.py status --config agents.toml` parses and lists all agents.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unit tests (no agents/tmux): config load + defaults merge, kickoff-template
assembly, phase machine (advance/idempotent-complete/append-resumes), limit
reset-banner parsing, WAITING-UNTIL/stall parsing, claude+opencode activity
detectors. Live smokes bring a throwaway project up THROUGH agents.py on each
real backend in an isolated sandbox (unique prefix, opencode on a non-4096
port), verify attach + status + down, and clean up. tests/run.sh runs unit
always + smokes when backends present; README documents it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
notplantsandClaude
autonomic-botandClaude Opus 4.8