Compare commits

..
56 Commits
Author SHA1 Message Date
notplants f0c5c4b3cc Merge Codex Remote Control support 2026-08-23 03:47:37 +00:00
notplants 9afd9e0a99 Preserve agent cwd through Codex app server 2026-08-23 02:44:06 +00:00
notplants a7d61d812e Fix Codex Remote thread writer conflicts 2026-08-23 02:39:00 +00:00
notplants 927c90de43 Add Codex remote-control backend 2026-08-23 02:04:54 +00:00
notplantsandClaude Opus 5 31839b6236 make the read tool refuse a pull that does not exist
A Tangled 404 renders 200 and carries no pull record, so tangled_comments.py's
"0 comment(s)" was indistinguishable from "no such pull". On 2026-08-22 that
produced a false report that a pull was filed and unreviewed, a retracted claim
that the queue tool was broken, and a reviewer's time spent disproving it.

tangled_comment_post.py never had the defect, because it MUST resolve a
subject-uri to function. So the read tool now requires the same identifier and
exits non-zero when the page does not yield one. Proven: #99999 and a
non-existent #428 exit 1 naming the cause, while a real pull still serves its
comments.

This is the structural fix over the vigilance fix, and the argument for it is
that both of us knew the countermeasure and neither applied it — the trap was
already written in one set of notes and in three of the reviewer's own reviews.

And the post tool's error named only the cookie, so anyone hitting it on a
phantom pull would re-auth, succeed, and still fail. It now names both causes in
likelihood order, the pull first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
2026-08-22 08:00:16 +00:00
notplantsandClaude Fable 5 fe9d98fad3 README: drop the PATH note — the fix belongs in the host config, not the docs
Operator: rather than documenting the workaround, notplants-nix should put the system profile on
the agent PATH. It already does (two orchestrator units export it); the gap is that the change is
committed and not yet deployed. The /proc rewrite in agents.py stands on its own — harness code
should not depend on an external tool for something the kernel exposes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
2026-08-21 03:52:39 +00:00
notplantsandClaude Fable 5 e185cec88c tangled_pr: verify against the pulls list, not a response header; and a tools test suite
THE BUG THAT PROMPTED THIS. tangled_pr.py judged success ONLY by an HX-Redirect header on the POST.
A create that SUCCEEDED but answered without that header read as a failure, so the caller retried
and Tangled grew duplicates — that is exactly how #397, #398 and #399 were filed for one branch. A
response header describes what the server meant to say; it is not the artifact.

Now it checks the artifact, in both directions:
  * BEFORE posting, refuse if an open pull already exists for this source branch, naming it. A
    retry cannot duplicate, whatever the response said. (--allow-duplicate to override.)
  * AFTER posting, confirm against the pulls list: a new pull number that did not exist before,
    whose page names this source branch, IS the success — with or without a redirect header.
  * Failure is reported only when no such pull appeared. A false failure is worse than a loud
    error here, because the caller's remedy is to retry.
Verified live: a dry-run against a branch that already has a pull refuses with rc=3, naming #417.

TWO REAL DEFECTS FOUND BY WRITING THE TESTS.

agents.py shelled out to `pgrep -P` and `ps -o comm=`. Neither is on the agent PATH on this host,
and a missing binary under shell=True returns rc=127 with EMPTY stdout — indistinguishable from
"this process has no children" and "no build is running". So _build_running was ALWAYS False and
the stall detector could reboot an agent mid-build. Both now read /proc directly: no PATH
dependency, and it cannot fail silently in that direction.

That shipped because the unit tests MOCKED pgrep and ps. The fakes stood in for the broken
dependency, so the suite passed on a host where neither tool was reachable and never exercised the
real path. The tests now patch _proc_descendants and _comms — the seams this repo owns. A test that
mocks a dependency proves the mock works.

Also fixed a monkeypatch leak those tests had: restoration used a name derivation that silently
matched nothing, so the patch escaped into another test class and failed an unrelated test — only
in a full run, never when that test ran alone. Now addCleanup, which cannot be ordered wrong.

NEW: tests/test_tools.py, 24 tests over tangled_pr, tangled_pr_close and gateway-domain, with every
HTTP boundary injected so they run offline. Mutation-checked: breaking classify(), the pull-number
regex, the branch match, or the scan bound each turns the suite red. Suite is 93 tests, green, and
order-stable across repeated runs.

README: a "PATH on a NixOS host" section. Every one of ps, pgrep, free, cmp, awk, curl, diff,
strings, nm, getent and ping is INSTALLED here and simply not on the agent PATH, so each reports
"command not found" and reads as a missing package. Documents how to check before concluding a tool
is absent, how to add the system profile, `nix shell` for what is genuinely missing, and the rule
that harness code should not shell out for what the kernel already exposes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
2026-08-21 03:50:11 +00:00
notplantsandClaude Fable 5 e1ba9b39be secrets: project-scoped secrets go in /secrets/<project>/
Operator convention, 2026-08-20. If a secret belongs to one project it lives in that project's
directory rather than in files/ with the project name baked into the filename:
/secrets/lichen/test-pds.env, not /secrets/files/lichen-test-pds.env. files/ is reserved for
things genuinely shared across projects.

A flat directory forces every name to carry its own scope, which nobody does consistently, and
then 'what does this project hold' and 'what do I revoke if it is compromised' both need a grep.
A directory answers both by listing. The convention already existed in practice (b1,
notplants-orchestrator, emily-sandbox) and was simply never written down.

The symlink rule is unchanged: a consumer insisting on a fixed path gets a symlink into
/secrets/<project>/, so the file still exists exactly once.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3LdmEL7CvCYTNpoBq1kce
2026-08-20 22:08:17 +00:00
notplants 23391cef2b gateway-domain: tag:orchestrator is now allowed by the ACL too
A rule was added so the testing gateway may also reach nodes tagged
tag:orchestrator, not only tag:notplants-test-server. Both tags now appear in
the skill's table, and the tool's warning fires only when a node carries
neither.

Verified end to end from an orchestrator box (tag:orchestrator, tag:server)
rather than assumed:

  - the gateway can open a TCP connection to it over the tailnet
  - a request to https://acltest.gtest.commoninternet.net/ returns the box's
    own self-signed certificate, subject and issuer both CN=acltest...,
    which is only possible if the gateway proxied the stream instead of
    terminating it
  - the payload came back and the box's listener logged the request

The troubleshooting section gains the one-liner that isolates this from the
gateway itself: open a TCP connection to your backend from the gateway.
2026-08-20 18:22:58 +00:00
notplants c7cbac6fb2 gateway-domain: the backend needs tag:notplants-test-server
The tailnet ACL only permits the gateway to open connections to nodes tagged
tag:notplants-test-server. Map an untagged node and the gateway accepts the
mapping and then never connects — no error anywhere, and it presents as a
broken gateway rather than as a missing tag on your own box.

Nothing said so, and it is not discoverable from the failure. The skill now
leads with the requirement and the one-liner to check it, and names it as the
first thing to look at when traffic does not flow.

The tool also warns when the node it is about to map does not carry the tag.
It cannot check a backend given explicitly on the command line — it only sees
its own tags — so that case stays documented rather than enforced.

Found by mapping this orchestrator box (tag:orchestrator, tag:server) as a
smoke test: the mapping was written and looked entirely healthy.
2026-08-20 18:01:41 +00:00
notplants 22bd897a86 gateway-domain: give a tailnet box a real public domain
An agent on a box with no public IP frequently needs a reachable HTTPS URL —
an OAuth callback, a webhook receiver, a demo link. The testing gateway
already holds a wildcard record for *.gtest.commoninternet.net and forwards
by SNI, but nothing here knew that, so every agent had to be told by hand.

    tools/gateway-domain.py add myapp
    #   myapp.gtest.commoninternet.net  ->  100.84.190.30

The backend defaults to the running box's own tailscale IP, which is the case
that comes up almost every time.

The admin password comes from gateway.admin_password in the secret store; the
tool reads it itself, so no caller handles the value and there is no second
copy to drift or get committed.

Two things the tool refuses to do, both learned by doing them:

Backends must be a literal IPv4 address. The gateway's validate_ip accepts a
hostname, but put_domain/remove_domain only match lines whose backend is
numeric ([\d.:]+). A hostname mapping can therefore be written once and never
updated or removed through the admin UI — it becomes an orphan that only a
hand-edit of tunnel_map.conf clears. One got created while testing this.

Verification re-reads the mapping table instead of trusting the POST body.
The admin app mutates its in-memory dict and renders that, so a delete that
silently failed still renders as success. Checking the response alone
reported "removed" for an entry that was still on disk.

skills/gateway-domain/ carries the rest: that the gateway does NOT terminate
TLS (your box serves the cert for that name), how ACME still works through
it, that only 22/80/443 are open at the edge, and how to recover if an
interrupted e2e run leaves the admin password reseeded.
2026-08-20 17:10:28 +00:00
notplantsandClaude Opus 5 bb03bed218 tangled_pr_close.py: close a pull as the bot, verified against the page
engine/ could create, edit, merge, resubmit and comment on a pull but not close
one. The appview's Close button is an htmx POST to /pulls/{n}/close. This tool
posts it with the session cookie, optionally posts one comment first (the rule
for the site-concurrency series' phase 8: every closed pull names its
successor), refuses to close a MERGED pull, and trusts nothing — it re-fetches
the pull page and requires the badge to read Closed.

First use: lichen.page.review #397-#399, three duplicates of #396.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E3aQXnUnx9kncNHQi3c92f
2026-08-18 23:36:04 +00:00
notplantsandClaude Opus 5 64d49b4402 watchdog: register a wake added to the config mid-run
wake_elapsed was built once at startup and thereafter only ever shrank — the
loop already handled an agent whose `wake` was REMOVED mid-run, but a wake
ADDED to agents.toml while the watchdog was running was invisible to it,
silently and permanently. The config is re-read every tick, which makes the
wake look live when it is not.

Found on lichen-orchestrator: both flat-file agents were given 30-minute wakes
on 2026-08-16 against a watchdog process that had been up since 2026-08-01.
Neither wake ever fired — zero "waking rust-flat-file" lines in two days of log
— and with watch="heal" (no stall-reboot) each finished turn parked the agent
until a human noticed. A supervisor script had to stand in as their wake.

New wakes are seeded at 0 so they fire on their own schedule rather than
immediately, and the registration is logged so a silent wake is visible next
time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E3aQXnUnx9kncNHQi3c92f
2026-08-17 21:52:38 +00:00
notplants-bot 5227cd7f6e tools/secrets.sh: EXIT trap must return 0
scrub() ended in `[ -n "$WD" ] && { ... }`. As an EXIT trap, its status becomes the
script's status, so with WD unset every read-only command exited 1 while printing the correct
answer. Callers saw a silent false failure — tests/run.sh reported 'could not read
tailscale_authkey' having successfully read it.
2026-08-17 00:12:15 +00:00
notplants-bot ec9f592e46 tools/secrets.sh — safe sops wrapper, so this cannot happen again
On 2026-08-16 a hand-rolled sops pipeline wrote plaintext to the real secrets path and only
THEN tried to encrypt it. The encrypt failed (creation_rules keyed on a filename the temp path
did not match), sops exited non-zero after the clobber had already happened, and the plaintext
file was copied to the host world-readable before anyone noticed.

The invariant here is that plaintext never exists at the destination path: every mutation
happens on a temp file in a 0700 dir, and install_encrypted() refuses to move anything into
place that is not verified ciphertext AND does not round-trip through a decrypt. A failure at
any step leaves the original untouched — verified by reproducing the incident (break the
creation rule, attempt a set, confirm the file's hash is unchanged).

Two other traps are handled because they already bit us: --config is passed explicitly, since
sops discovers .sops.yaml from the CWD and the manual runs only worked by accident of being in
the right directory; and PATH is hardened, because a tool 'not found' merely because
/run/current-system/sw/bin was absent reads as 'decryption failed', which is the wrong
conclusion entirely.

  SECRETS_HOST=b1 engine/tools/secrets.sh verify|list|get|set|unset|edit|deploy
2026-08-17 00:06:26 +00:00
notplants 6ee9197fce tangled tools: tangled_pr_resubmit.py — a push does not advance a PR round
Pushing a fixup to the source branch does NOT update the pull: the appview serves the patch
it fetched when the pull was opened or last resubmitted, so a reviewer keeps reading the
pre-fixup code and no `/round/<r>/interdiff` exists. Nothing warns you — the push succeeded,
the branch is right, the pull silently lags. It cost a review cycle on 2026-08-16 (three
flat-file PRs reviewed against stale trees because "pushed" was read as "resubmitted").

The appview's action is a bare htmx POST with no body,
`POST /{owner}/{repo}/pulls/{n}/resubmit`, which re-fetches the branch patch from the knot
and opens the next round. The tool wraps it and:

- takes several `--pull` numbers at once (a stack-wide rebase touches every rung);
- prints the new round and its interdiff URL — what a reply to a reviewer leads with;
- says plainly when the round did NOT move (the branch already matched the pull) instead of
  reporting a silent success;
- `--check` reports the current round without posting.

It reads the round from the round selector rather than the page text, because a comment that
quotes a `/round/2/interdiff` URL would otherwise be counted as a round (that fooled me
first).

README gains a table of the Tangled tools, which did not exist, and states the trap.
2026-08-16 21:40:26 +00:00
notplantsandClaude Opus 5 0dfd491ed9 tangled tools: comment-post + merge/close with mergeability gate
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ycNmtHB3N9WZyHcZ6T6Xe
2026-08-12 04:57:17 +00:00
notplants 50fc8fd89c tangled_comments: read a pull's review comments; secrets: resolve sops off-PATH 2026-08-12 04:21:58 +00:00
notplantsandClaude Opus 4.8 52b59bbe00 docs(skills): add codeberg-pages site-publishing skill
Skill covering publishing a static site to Codeberg Pages, including
custom domains on the new git-pages server: pages branch, A/AAAA + CNAME
DNS, the _git-pages-repository TXT authorization record, per-domain
deploy webhooks (http:// for first deploy), Let's Encrypt issuance, and
the obsolete .domains file. Verified against docs.codeberg.org.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017k4W786WWd8yN1m4Ypuxzx
2026-08-01 21:58:47 +00:00
notplantsandClaude ef85d40a63 feat(secrets): path-bound secrets are symlinks into /secrets/files
A secret a third party reads from a fixed path (ssh key, systemd EnvironmentFile, nix
authKeyFile, TLS keypair) now lives ONCE as a real file in /secrets/files and is symlinked
from where the consumer expects it. The consumer is unchanged and unaware; the file exists
in one directory, at 0600, outside /srv and outside every git tree.

That makes the store and the file directory alternatives, not layers: a secret is a value in
store.yaml OR a file in /secrets/files, never both. The copies of the ssh keys, tailscale
auth key, incus keypair, LE cert and cc-ci testenv have been dropped from store.yaml now that
each has a single home.

Documented exception: an app that rewrites its own credential file (OAuth refresh via
write-temp+rename) replaces the symlink with a regular file and silently re-splits the home.
opencode's auth.json is one, so it stays put and is deliberately not centralised.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 17:06:18 +00:00
notplantsandClaude 300e69d3b3 refactor(secrets): consumers read the store; drop materialized copies
Materializing wrote a second plaintext file per secret, which is the problem the store
was meant to solve: two files drift, and the copy is what ends up committed or grepped.

- tangled_pr / tangled_pr_edit / tangled_repo now read tangled.cookie from the store.
  engine/.tangled-session is deleted; --cookie-file remains as a legacy escape hatch.
- materialize() is replaced by run-time injection that leaves nothing at rest:
    exec-env <group> -- cmd      group as env vars (use this instead of a systemd
                                 EnvironmentFile — same effect, no plaintext on disk)
    with-file <key> -- cmd {}    0600 file in a private tmpdir, removed when cmd exits,
                                 for consumers that insist on a path (ssh -i, a TLS key)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:58:26 +00:00
notplantsandClaude 6c56c1953e refactor(secrets): store lives at /secrets, not /srv/secrets
/srv is the projects tree — agents grep, find and `ls -R` it constantly, so a store
under it turns up in ordinary searches and risks being read (or pasted) by accident.
/secrets sits outside that blast radius: nothing routinely walks it, and it is still
0700 loops, still not a repo, still ciphertext at rest.

Override with AO_SECRETS_STORE if a host puts it elsewhere.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:54:03 +00:00
notplantsandClaude 7605efe624 feat(secrets): one sops+age store for the host, documented for every project
Credentials were scattered in plaintext: a gitea password baked into six git remote
URLs (`git remote -v` prints those), API keys in .env files, an incus client key at
0644. Anything living in a repo is one `git add -A` from being pushed.

So: ONE encrypted file outside every git tree, and a helper each project uses.

  /srv/secrets/store.yaml       sops+age ciphertext, 0600, not a repo, no remote
  ~/.config/sops/age/keys.txt   the only plaintext secret on disk, 0600

secrets.py is stdlib + the sops binary: get("group.key"), get_group("group"), and
materialize() for consumers that must read a fixed path (systemd EnvironmentFile,
ssh IdentityFile, nix authKeyFile) — those keep their file, but the store is the
source of truth, so a materialized file is never hand-edited.

`list` prints names only, never values, so it is safe in a transcript.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-01 16:51:36 +00:00
notplantsandClaude 26d93c1eda fix(tangled_pr): POST the htmx /pulls/new form, not /pulls/
/pulls/ is 405. The appview's PR form is htmx: it only processes the create when
it sees HX-Request, otherwise it re-renders the page — a 200 that silently creates
nothing. Success comes back as HX-Redirect (…/pulls/<n>), not a 3xx Location, so
the old redirect check never fired. Also send source=branch + the title/body Dirty
flags the form expects, and point the auth-failure hint at the current cookie path.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-08-01 16:36:09 +00:00
notplantsandClaude Fable 5 5a993aacba tangled_pr_edit.py: edit an existing pull's title/body via the appview edit form
Sibling of tangled_pr.py. GET/POST /{owner}/{repo}/pulls/{n}/edit (htmx, session
cookie); does not mint a round or touch the patch; re-fetches the form after POST
and verifies the appview holds the new text. Built and proven by the
rust-pr-desc-concise pass (see machine-docs/PR-DESC-CONCISE.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BMb31Hx9mnYGRXZ3AL68sk
2026-07-31 15:02:27 +00:00
notplantsandClaude Opus 4.8 2af97160c4 tangled_repo.py: create a Tangled repo as the bot (POST /repo/new); sibling of tangled_pr.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 09:36:54 +00:00
notplants aa1b625732 watchdog: proof-of-life outranks the agent's own WAITING-UNTIL deadline
Completes b73af35. That commit stopped the cap from killing a live build, but the OTHER branch still
rebooted past the stated deadline regardless of a running build — and an agent blocked on a shell cannot
re-emit a fresh marker to extend its estimate. So a cargo-mutants run that simply overran its guess would
still be killed at the guess, throwing the work away. The deadline is an ESTIMATE; a running build is a
FACT. Order is now: absolute cap (waiting_until_max) > live build > stated deadline. The cap is the one
bound that reboots even mid-build — which is exactly how a genuinely hung build gets caught, so a runaway
still cannot park forever. Tests: live-build-defers-past-deadline, past-deadline-no-build-reboots,
cap-is-absolute-and-reboots-a-hung-build. 68 pass.
2026-07-11 13:53:39 +00:00
notplants b73af35792 watchdog: a live build is proof of life — don't let waiting_until_max guillotine a long legitimate run
Found in production: rust-mutation emitted WAITING-UNTIL 3h11m out for a cargo-mutants run over the whole
workspace (a genuinely multi-hour job, cargo running the whole time), but the marker branch skipped the
build-aware check entirely and went straight to the cap — so the 7200s cap would have killed a live run and
thrown away hours of work. The cap exists to catch a session that PARKED itself and is stuck; a build still
running under the session is proof it is not. Now the cap only fires when idle exceeds it AND no build is
running. The stated deadline remains the hard bound either way, so a runaway still cannot park forever.
Tests: cap-reboots-when-no-build, cap-yields-to-a-live-build, past-deadline-reboots-even-with-a-build. 68 pass.
2026-07-11 11:50:39 +00:00
notplants 582f392ef5 watchdog: make WAITING-UNTIL work for footer_ui backends + cap runaway defers
_parse_waiting_until scanned only the pane's last non-empty line for footer_ui backends (claude/
opencode) — but their input-box footer always renders BELOW the agent's final message, so the marker
was never seen and WAITING-UNTIL was effectively dead for claude agents. It's only consulted once the
pane is already idle, so scan the whole capture and take the most-recent marker (the footer never
contains it). Add waiting_until_max (default 7200s) so an agent can't park its own reboot forever.
Tests: footer-honors-marker-above-footer, takes-most-recent, defer + cap in stall_check_one; make the
stall harness's patch() idempotent so a re-patched name doesn't leak into tearDown. 66 pass.
2026-07-10 20:39:51 +00:00
notplants c591f5430d tests: cover build-aware stall detection
13 tests across 4 classes: the build-proc match set (build tools match; python/node/bash/claude and
substring look-alikes don't), _proc_descendants (real child tree, roots excluded), _build_running
(session-scoped, comm-matched, never the claude root, custom regex override), and stall_check_one's
defer/reboot/hard-cap behavior. Full suite 64 pass.
2026-07-08 16:53:08 +00:00
notplants ce66948245 watchdog: build-aware stall detection — defer reboot while a real build/test runs (scoped to the watched session's child procs), with stall_idle_max hard cap
A silent pane whose claude session has a live descendant compile/coverage/test process (cargo, rustc,
cc1, llvm-cov, lichen-server, chromium, …) is a running build, not a stall. _build_running() inspects
ONLY the descendants of that session's pane_pid (never the claude root, whose args embed the prompt),
matching process comm. Defers the kill+reboot until the build finishes, but never past stall_idle_max
(default 1800s) so a hung build still recovers. Configurable via build_procs_re / stall_idle_max.
2026-07-08 16:42:57 +00:00
notplantsandClaude Opus 4.8 071d74f21f feat(tools): tangled_pr.py — file a Tangled PR via the appview OAuth web endpoint
Reusable utility (not project-specific): tangled's appview only indexes pulls created through its
OAuth-authenticated web endpoint POST /{owner}/{repo}/pulls/ (it fetches the knot patch + inserts into
its DB directly); raw com.atproto.repo.createRecord does NOT index. This tool reuses a browser session
cookie (gitignored .tangled-session) to POST target/source branch names. No blob upload, no CSRF.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-07-08 05:10:38 +00:00
notplantsandClaude Opus 4.8 997f73af1f fix(watchdog): don't treat idle footer's '· N shells' as active (default active_re)
The default active_re matched a bare middle-dot+number, which also matches the TUI idle footer's
'· 3 shells' / '· 1 shell still running'. Any background shell then read as ACTIVE, masking a genuine
idle from the stall detector — the agent hung at an empty/stranded prompt and never rebooted. Drop the
timer token; genuine activity is 'esc to interrupt'/'Running tool' plus the log-recently-touched grace.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-07-07 20:02:05 +00:00
notplantsandClaude Opus 4.8 234d6a054e fix(pipeline): PIPELINE-COMPLETE mirrors live stage list, not a write-once latch
pipeline_check wrote PIPELINE-COMPLETE the first tick all-then-configured stage markers existed, and
never cleared it. Appending stages to an already-completed pipeline (as the orchestrator did — adding
rust-linecov-e2e then two -realpds stages after the fork sequence finished) left a stale 'done' sentinel:
the new stages ran, but the file lied that the pipeline was complete, misleading operators reading state.

Now recompute the sentinel every tick from ALL current stages' markers: write it only when every marker
exists, and CLEAR a stale one the moment any stage is incomplete — so appending stages self-corrects.
Gated on the markers directly, not 'active is None' (which is also None when a stage names a missing agent).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-07-07 19:25:12 +00:00
notplantsandClaude Opus 4.8 08fd58ccc0 feat(pipeline): push-on-retire — no stranded commits when a stage completes
The pipeline retires a stage the instant its completion marker appears; if the agent wrote the marker
before its final 'git push' landed, its last commits were stranded locally (observed: a review stage
left 7 unpushed commits). Now, before retiring a COMPLETED stage, the watchdog does a best-effort
'git push origin HEAD:main' in that stage's dir (never raises; 90s timeout). Guarantees the deliverable
reaches the remote.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-07-04 17:47:23 +00:00
notplantsandClaude Opus 4.8 b739504e1f feat(watchdog): [pipeline] — sequential standalone-agent phases via completion markers
Adds a [pipeline] block: an ordered sequence of DISTINCT agents (different prompt/model/dir) run one
at a time, advancing when each writes its completion marker — the standalone-agent analog of the
[loop] phase machine (which drives FIXED loop agents via ## DONE). The watchdog reconciles it every
tick, stateless (markers are the source of truth): exactly the first not-yet-complete stage runs,
earlier stages are retired, the active stage is stall/heal-watched. Pipeline agents are enabled=false
(the pipeline owns their lifecycle).

Also fixes a latent watchdog crash: wake_elapsed is built once at startup but config is re-read each
tick, so removing an agent's 'wake' mid-run (e.g. winding down a wake) hit agent['wake'] -> KeyError
and killed the whole watchdog silently (stalling phase advancement + stall recovery). Now skips agents
whose wake was removed.

IDEAS.md: note that [pipeline] and [loop].phases are the same shape and should be unified via an
optional per-phase agent/dir/done (fold pipeline into the phase machine, delete the parallel path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-07-04 13:45:47 +00:00
notplantsandClaude Opus 4.8 08bbb60343 fix(watchdog): stop phase-machine handoff/gate-token work after SEQUENCE-COMPLETE
Gate-token tracking + handoff pings kept running on the completed phase machine,
churning 0-token gate records every tick. Gate them on `not seq_done`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-24 15:38:02 +00:00
notplantsandClaude Opus 4.8 164df87e98 fix(wake): persistent-agent wakes survive SEQUENCE-COMPLETE
The watchdog gated ALL scheduled wakes behind `if not seq_done`, so once a phase
sequence completed, even a persistent operator-facing supervisor stopped waking.
That breaks follow-on supervision (e.g. a second build started after the first
sequence finishes). Now: loop-tied wakes (on-demand auditor etc.) still quiet after
completion, but persistent agents keep waking — their hourly supervision survives.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-24 15:36:02 +00:00
notplantsandClaude Opus 4.8 44bb1da1be feat(watchdog): DONE-nudge for ceremony-lag (built-but-unmarked phase) before kill+reboot
Recurring stall: a phase is substantively complete (all DoD gates PASS from both
adversaries, no veto) but the builder never writes the done marker, so auto-advance
cannot fire and the loops idle. A blunt stall kill+reboot does not fix it (the
re-kickoffed agent just re-idles).

On a stall, if the agent is a loop agent and the current phase is NOT marked done,
send a one-time DONE-nudge (ping) telling it to write the done marker IF the DoD is
met (both adversaries PASS, no veto), giving a fresh idle window; only escalate to
the kill+reboot if it stays stalled. One nudge per phase (cleared on phase advance).
Gated by [loop].done_nudge (default true); message uses the configured done_marker
and the phase status file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-24 02:40:14 +00:00
notplantsandClaude Opus 4.8 e6b53513d4 feat(wake): re-run one-shot task agents on their wake interval (autonomous cadence)
wake_agent only re-prompted a live persistent/loop session and returned False for a
dead one, so a "task" agent (one-shot, exits after its run) could not be re-run on a
schedule — its wake never fired. Now, for kind=="task", a wake kills+restarts the
task for a clean re-run (skipping only while its previous run is still active). This
makes scheduled work like a coverage audit recur autonomously, no operator trigger.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-23 05:17:07 +00:00
notplantsandClaude Opus 4.8 65ceeb3a7b fix(watchdog): seed stall clock from pane's real last-activity, not watchdog start
Stall detection tracked idle time in an in-memory _idle_since map seeded to now()
on first observation, so a freshly-(re)started watchdog reset every agent's stall
clock and had to wait a full stall_idle before it could nudge — an agent idle for
an hour looked freshly-idle after a watchdog restart. Seed  from the tmux
window's last-activity timestamp (#{window_activity}) instead, so idle duration
reflects the agent's real last activity regardless of when the watchdog started.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-23 04:40:34 +00:00
notplantsandClaude Opus 4.8 57082acc05 fix(tokens): restore token_phase_flush re-baseline; drop stray block from gate_token_check
The per-gate functions were inserted immediately after token_phase_flush's log
line, which split the function: its trailing re-baseline block (the
'if next_phase_id is not None: ...' that re-seeds the per-phase baseline for the
next phase, or finalizes when None) was orphaned onto the end of gate_token_check,
where next_phase_id is undefined. The watchdog therefore crashed with NameError on
the first tick of every start. Move that block back into token_phase_flush (where
next_phase_id/cur/sf are in scope) and end gate_token_check at its log line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-22 07:27:50 +00:00
notplantsandClaude Opus 4.8 188c12ad9e feat: configurable per-gate token logging + responsive phase auto-advance
Two watchdog/metrics improvements to the loop machine:

1) Token-logging granularity is configurable via [watchdog].token_granularity:
   'gate' (default) or 'phase'. In 'gate' mode, tokens are attributed to each
   claimed gate -- any 'claim(<label>)' commit on the work repo's origin/main
   (e.g. claim(D1-D5), claim(feat:multi-file); a leading 'feat:' is stripped) --
   in addition to the per-phase rollup, appended to token-log.jsonl tagged
   phase_id='<phase>:<label>'. A change in the most-recently-claimed label is the
   boundary; the in-flight gate is also flushed when the phase ends. 'phase' mode
   keeps the original per-phase-only behaviour.

2) Phase auto-advance is now evaluated on EVERY signal tick instead of only the
   heavy tick, so a completed phase advances within signal_interval of its
   '## DONE' landing rather than idling up to heavy_interval. Healing stays on the
   heavy cadence.

Note: gate-boundary detection assumes the loop's 'claim(<label>)' commit convention.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UWTdUq2bsic7JZGqJp3nD6
2026-06-22 05:15:08 +00:00
notplants 98d198baa9 feat(handoff): claim_pings/review_pings accept a list — ping every reviewer
Multi-reviewer setups (e.g. a correctness + a readability adversary) can now have
the watchdog ping ALL reviewers on a claim, each in its own session with its own
submit key. A bare string still works (single agent). _ping_agents() helper.
2026-06-22 00:24:41 +00:00
notplants 781db071dd docs(readme): add Examples section (Builder/Adversary variants, snakepit) + benchmark note 2026-06-16 02:35:40 +00:00
notplantsandClaude Opus 4.8 90375f004e docs(examples): add builder-adversary-deferred — verify after a long segment
Coarsest review cadence: the Builder self-certifies the build phases and the
Adversary does ONE comprehensive cold-verification of the whole accumulated build
in a final `review` phase (vs orig per-phase, lean per-gate). Full original
prompts + a DEFERRED REVIEW CADENCE override, so it isolates verification cadence.
Cheapest coordination; the trade-off is the independent check arrives late (late
rework risk + self-certification drift on build phases). README spells it out.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:02:44 +00:00
notplantsandClaude Opus 4.8 c6c7ce8640 change: base stateless + lean on the FULL original prompts (not minimal)
So that "stateless vs builder-adversary" and "lean vs stateless" isolate context
hygiene / review granularity WITHOUT the confound of the minimal prompts' reduced
testing pressure (which we found cuts ~25% of test methods). stateless = orig +
context hygiene; lean = orig + context hygiene + per-gate review. min stays the
pure minimal-prompt variant (isolates verbosity vs orig).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 03:17:47 +00:00
notplantsandClaude Opus 4.8 a0f7652e9e docs(examples): add builder-solo — single builder, no adversary (control)
A single Builder that builds AND self-verifies (same DoD rigor), with NO
independent Adversary and no claim/review handoff. The control for measuring
what the AI adversary costs (its tokens, ~half of a loop-pair run) and buys
(independent cold verification vs self-certification).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:34:50 +00:00
notplantsandClaude Opus 4.8 924874aafa feat: optional log_tokens — per-phase token + time accounting
When [watchdog].log_tokens (or [loop].log_tokens) is true, the watchdog records
for each phase how many tokens each agent used (and the total) and how long the
phase took, appended to <log_dir>/token-log.jsonl. Tokens are summed from each
agent's session transcript, attributed by working dir. View with `agents.py
tokens`. Baseline snapshot at phase start + delta at phase advance/complete;
robust across watchdog restarts. Validated: the transcript sum matches an
independent external collector exactly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:48:17 +00:00
notplantsandClaude Opus 4.8 e0425e6108 docs(examples): add builder-adversary-lean — context hygiene + per-gate review
Isolates the two effects conflated in builder-adversary-stateless: keeps all the
CONTEXT HYGIENE (compact/diffs/lean loads) but ENFORCES full per-gate review
granularity (one claim per gate, one independent verdict per gate, no batching).
Tests whether the token saving is real efficiency vs reduced scrutiny.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:42:12 +00:00
notplantsandClaude Opus 4.8 985d33dd51 docs(examples): add builder-adversary-stateless — context-lean variant
Same pattern + AI-as-adversary verification as builder-adversary-min, but the
role prompts add CONTEXT HYGIENE: /compact at every checkpoint (lossless — state
is on disk), read diffs not trees, spill bulk output to files, adversary loads
only {plan, STATUS, diff}. Loop agents non-resumed → fresh session per phase.
Targets cache-read (the dominant cost in a long loop) without changing what the
agents do or how they verify.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:47:58 +00:00
notplantsandClaude Opus 4.8 737ef81066 docs(examples): add builder-adversary-min — minimal-prompt variant
Same topology/behaviour as builder-adversary (loop pair, phase machine,
claim()/review() handoff, machine-docs coordination, cold verification) but the
role + kickoff prompts are compressed to minimal tokens, keeping every
load-bearing rule. Config and plans are unchanged. The separate
agent-orchestrator-benchmark repo runs a head-to-head token comparison.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:18:33 +00:00
notplantsandClaude Opus 4.8 11843f41a4 docs(examples): add IDEAS.md — backlog of creative example topologies
A sketch backlog of further examples, each teaching a distinct orchestration
topology (anthill/stigmergy, kitchen line/pipeline, incident room/blackboard,
senate/debate, baton/mutex+failover, immune system/reactive, evolution chamber,
plus ATC and day-night extras). Not implemented — ideas only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 18:13:48 +00:00
notplantsandClaude Opus 4.8 e4453dcfdd docs(examples): add the "snake pit" worker-pool example
Based on @ponder.ooo's "snake pit agent orchestrator" idea (bsky 2026-05-28) and
Claude's metaphor-mapping elaboration: agents are snakes, tasks are food tossed
into a shared pit; snakes devour/digest/regurgitate/excrete.

A worker-pool-over-a-shared-queue topology (contrast the builder-adversary phase
machine):
- pit/ is a filesystem queue; snakes claim by atomic mv (no two eat the same food)
- species = specialized agents: keeper (zookeeper), planner (regurgitation IS
  task decomposition), snake-1..3 (worker pool), cleanup (scavenger + coprophagy)
- no [loop] phase machine; persistent agents self-pace via /loop
- README carries the full bio→compute mapping table from the thread image

Verified: `agents.py status --config agents.toml` lists all 6 agents + service.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:50:42 +00:00
notplantsandClaude Opus 4.8 7f237a522c docs(examples): add a Builder/Adversary loop-pair example (the cc-ci pattern)
A self-contained examples/builder-adversary/ that distills the cc-ci production
loop pair into a tiny, fully-local task (build a `wc` CLI in two phases):

- agents.toml: builder + adversary loops, persistent orchestrator, on_complete
  reporter, cleanlogs service; phase machine with a per-phase model override
- prompts/: kickoff template + builder/adversary roles carrying the load-bearing
  protocol (claim()/review() handoff, machine-docs file-location rule,
  WHAT+HOW+EXPECTED+WHERE=STATUS / WHY=JOURNAL anti-anchoring, WAITING-UNTIL liveness)
- plans/: two phase plans (wc, json) each with a cold-verifiable Definition of Done
- README: how to run, the work-repo two-clone isolation model, how to adapt

Verified: `agents.py status --config agents.toml` parses and lists all agents.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:50:42 +00:00
autonomic-botandClaude Opus 4.8 cdcece9a9a test: add tests/ — unit suite + isolated live claude/opencode smokes + runner
Unit tests (no agents/tmux): config load + defaults merge, kickoff-template
assembly, phase machine (advance/idempotent-complete/append-resumes), limit
reset-banner parsing, WAITING-UNTIL/stall parsing, claude+opencode activity
detectors. Live smokes bring a throwaway project up THROUGH agents.py on each
real backend in an isolated sandbox (unique prefix, opencode on a non-4096
port), verify attach + status + down, and clean up. tests/run.sh runs unit
always + smokes when backends present; README documents it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 18:55:34 +00:00
84 changed files with 6262 additions and 39 deletions
+1
View File
@@ -4,3 +4,4 @@
__pycache__/ __pycache__/
*.pyc *.pyc
result result
.tangled-session
+246 -2
View File
@@ -16,7 +16,9 @@ agents.py the driver + watchdog (pure Python stdlib; needs python >=
agent-log.py render claude JSONL transcripts into clean, greppable logs agent-log.py render claude JSONL transcripts into clean, greppable logs
agents.example.toml a self-contained 2-agent example project agents.example.toml a self-contained 2-agent example project
prompts/ generic role + kickoff templates (builder / adversary / kickoff) prompts/ generic role + kickoff templates (builder / adversary / kickoff)
examples/ runnable example projects — the Builder/Adversary variant family, snakepit, …
smoke.sh bring the example up + tear it down in an isolated sandbox, then clean up smoke.sh bring the example up + tear it down in an isolated sandbox, then clean up
tests/ the test suite — unit tests + isolated live backend smokes + a runner
flake.nix/.lock a Nix devShell with the runtime deps (python311, tmux, git) flake.nix/.lock a Nix devShell with the runtime deps (python311, tmux, git)
``` ```
@@ -48,6 +50,42 @@ python3 agents.py --config agents.toml phase show # where the loop phase mach
--- ---
## Examples
`examples/` holds runnable example projects — copy one, point `agents.py` at its `agents.toml`, and
go. The headline set is a family of **Builder/Adversary** variants that build the *same* task but each
differ in one dimension — useful both as templates and as a study of the pattern:
- **`builder-adversary`** — the canonical loop pair: a Builder that builds and an Adversary that
cold-verifies every claim, coordinating only through git (`claim(`/`review(` commits + the watchdog
handoff). **Start here.**
- **`builder-adversary-min`** — the same pattern with the prompts compressed to minimal tokens.
- **`builder-adversary-stateless`** — `builder-adversary` + **context hygiene** (compact at each
checkpoint, read diffs not trees, lean loads) to minimise carried/reloaded context.
- **`builder-adversary-lean`** — context hygiene + **per-gate** review (one claim/verdict per gate).
- **`builder-adversary-deferred`** — the Adversary verifies **once**, after the whole build, in a
final comprehensive `review` phase (vs per-phase / per-gate).
- **`builder-solo`** — a single Builder that self-certifies, with **no Adversary** (the control).
- **`snakepit`** — a different topology entirely: a pool of identical worker "snakes" pulling tasks
from a shared filesystem queue, plus cleanup specialists. (`examples/IDEAS.md` sketches more.)
Each example has its own `README.md`. Run one by hand:
```bash
cd examples/builder-adversary
python3 ../../agents.py status --config agents.toml # read-only
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
```
**Benchmark.** The separate
[`agent-orchestrator-benchmark`](https://git.autonomic.zone/recipe-maintainers/agent-orchestrator-benchmark)
repo runs these Builder/Adversary variants head-to-head (N=5, real `agents.py up` runs) to measure
what drives token cost. Short version: an independent adversary costs **~4.7×** a solo builder, but
the review *cadence* (per-gate / per-phase / deferred) is **nearly token-neutral**, and **context
hygiene** is the one clean **~22%** win. See that repo's `FINDINGS.md`.
---
## The config: `agents.toml` ## The config: `agents.toml`
Five section types: `[watchdog]`, `[backend.<name>]`, `[defaults]`, `[[agent]]` / `[[service]]`, Five section types: `[watchdog]`, `[backend.<name>]`, `[defaults]`, `[[agent]]` / `[[service]]`,
@@ -62,6 +100,23 @@ heavy_interval = 300 # seconds between heal + phase-advance checks
limit_probe_fallback = 300 # re-probe cadence for a usage-limited agent when reset time is unparsable limit_probe_fallback = 300 # re-probe cadence for a usage-limited agent when reset time is unparsable
limit_reset_slack = 45 # seconds to wait past a parsed reset before probing limit_reset_slack = 45 # seconds to wait past a parsed reset before probing
stall_grace = 180 # seconds of slack past a WAITING-UNTIL marker before a stall reboot stall_grace = 180 # seconds of slack past a WAITING-UNTIL marker before a stall reboot
log_tokens = false # opt-in: record per-phase token + time usage (see below)
```
**Per-phase token + time logging (`log_tokens`).** Set `log_tokens = true` (under `[watchdog]` or
`[loop]`) and the watchdog records, for **each phase**, how many tokens **each agent** used and how
long the phase took — appended as one JSON object per phase to `<log_dir>/token-log.jsonl`. Tokens
are summed from each agent's Claude Code session transcript and attributed **by working dir**, so
give each agent its own `dir` (the Builder/Adversary loop pair already uses separate clones) for
accurate per-agent numbers. The watchdog snapshots a baseline when a phase starts and writes the
delta (per agent, and the total) when the phase advances or the sequence completes — robust across
watchdog restarts. Pretty-print it with `agents.py tokens`:
```
phase dur(s) builder adversary TOTAL
-----------------------------------------------------
lex 372.0 3,910,118 3,221,447 7,131,565
parse 410.5 ...
``` ```
### `[defaults]` — inherited by every agent ### `[defaults]` — inherited by every agent
@@ -123,6 +178,25 @@ preamble = "set -a; . ./.env; set +a" # shell run before launch (e.g. l
active_re = "esc interrupt|thinking|running tool|preparing patch" active_re = "esc interrupt|thinking|running tool|preparing patch"
limit_re = "usage limit|limit reached" limit_re = "usage limit|limit reached"
[backend.codex] # Codex TUI, exposed through Codex Remote Control
bin = "codex"
preamble = "{bin} remote-control start --json >/dev/null"
remote_addr = "unix://" # connect the TUI to that daemon (one shared thread writer)
flags = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
supports_resume = false # thread-ID persistence is not implemented by this harness
prompt_delivery = "arg"
process_name = "codex"
footer_ui = true
log_grace = 180
submit_key = "Enter"
startup_prompt_re = "Trusting the directory|Do you trust"
startup_prompt_response = "1" # explicitly trust the agent's configured working directory
startup_prompt_delay = 2
stall_idle = 300
active_re = "esc to interrupt|working|thinking|running|searching|exploring|implementing"
limit_re = "usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "connection is errored|not logged in|authentication required"
[backend.demo] # a dependency-free backend for testing the harness mechanics [backend.demo] # a dependency-free backend for testing the harness mechanics
bin = "echo '[demo] {session} up'; exec sleep 1000000" bin = "echo '[demo] {session} up'; exec sleep 1000000"
prompt_delivery = "exec" # {kickoff}=prompt file, {session}=session name, {model}=model prompt_delivery = "exec" # {kickoff}=prompt file, {session}=session name, {model}=model
@@ -130,8 +204,20 @@ prompt_delivery = "exec" # {kickoff}=prompt file, {session}=session name,
For an `"arg"` backend the flag *templates* are configurable (so you can point at a non-claude For an `"arg"` backend the flag *templates* are configurable (so you can point at a non-claude
CLI): `resume_flag` (default `--resume '{id}'`), `model_flag` (default `--model '{model}'`), CLI): `resume_flag` (default `--resume '{id}'`), `model_flag` (default `--model '{model}'`),
`remote_control_flag` (default `--remote-control '{session}'`). A backend that sets `process_name` `remote_control_flag` (default `--remote-control '{session}'`), `remote_addr_flag` (default
`--remote '{addr}'`), and `remote_cwd_flag` (default `-C {dir}` whenever `remote_addr` is set).
A backend that sets `process_name`
participates in backend-mismatch healing; one that doesn't (e.g. `demo`) never does. participates in backend-mismatch healing; one that doesn't (e.g. `demo`) never does.
An optional `preamble` runs first and must succeed before the TUI starts; it supports `{bin}`,
`{session}`, `{model}`, and `{dir}` templates. The Codex backend uses it to idempotently start the
shared Remote Control daemon, then uses `remote_addr = "unix://"` to connect each tmux-hosted Codex
TUI to that daemon. The accompanying `-C` is required because the daemon's process directory is
independent of the agent's. This keeps the app-server as the sole thread-store writer while local
and Remote clients share the session in the configured project directory.
For an unattended TUI with a deterministic first-run gate, `startup_prompt_re` opts into one
screen check after `startup_prompt_delay` seconds; on a match, the harness types the explicitly
configured `startup_prompt_response` and `submit_key`. Codex uses this to trust the agent's
configured project directory so its argv kickoff can proceed.
### `[[agent]]` — one block per agent ### `[[agent]]` — one block per agent
@@ -212,6 +298,9 @@ phases = [
subject matches `claim_pattern` / `review_pattern`, and watches the two `inboxes` files. When a subject matches `claim_pattern` / `review_pattern`, and watches the two `inboxes` files. When a
claim lands it pings the `claim_pings` agent; a review pings `review_pings`; an inbox change claim lands it pings the `claim_pings` agent; a review pings `review_pings`; an inbox change
pings the relevant side. This is how the Builder and Adversary coordinate purely through git. pings the relevant side. This is how the Builder and Adversary coordinate purely through git.
`claim_pings` / `review_pings` may be a single agent name **or a list** — e.g.
`claim_pings = ["correctness-adversary", "readability-adversary"]` pings every reviewer on a claim
(each in its own session), for multi-reviewer setups.
--- ---
@@ -239,6 +328,7 @@ agents.py status table of every agent: kind, backend, model, w
agents.py watchdog the supervisor loop (what the <prefix>watchdog session runs) agents.py watchdog the supervisor loop (what the <prefix>watchdog session runs)
agents.py logs <name> tail that session's log agents.py logs <name> tail that session's log
agents.py phase [show|next|set N] inspect / move the loop phase index agents.py phase [show|next|set N] inspect / move the loop phase index
agents.py tokens per-phase token + time report (when [watchdog].log_tokens = true)
agents.py selftest regression-test the backend activity detector (needs no config) agents.py selftest regression-test the backend activity detector (needs no config)
agents.py init [dir] scaffold a starter agents.toml + prompts/ in a project dir agents.py init [dir] scaffold a starter agents.toml + prompts/ in a project dir
--config PATH use a specific config (default: ./agents.toml) --config PATH use a specific config (default: ./agents.toml)
@@ -298,6 +388,114 @@ Run it by hand with `engine/agents.py up --config agents.toml`.
--- ---
## Tangled (the atproto forge) — the PR tools
One tool per action, all reading `tangled.cookie` from the secret store (`--cookie-file` is a
legacy fallback). They exist because Tangled has no client CLI: each action is an htmx POST to
the appview that only a logged-in session can make.
| tool | action |
|---|---|
| `tangled_pr.py` | open a pull (branch-based; `--target` is the branch it merges into) |
| `tangled_pr_edit.py` | edit a pull's title/body, or `--show` the current one |
| `tangled_pr_resubmit.py` | **advance a pull to a new round after you pushed a fixup**, and print the interdiff URL |
| `tangled_pr_merge.py` | `--check` mergeability, then `--merge` or `--close` |
| `tangled_comments.py` | read a pull's review comments (`--json` for machine use) |
| `tangled_comment_post.py` | post a comment on a pull |
| `tangled_repo.py` | create a repo as the bot (see also the `tangled-repo` skill) |
**The trap `tangled_pr_resubmit.py` exists for:** pushing the branch does NOT update the pull.
The appview keeps serving the patch it fetched when the pull was opened or last resubmitted,
so reviewers read the pre-fixup code and no interdiff exists — with no warning, because the
push itself succeeded. Always: push, resubmit, then reply with the printed interdiff URL. See
`machine-docs/PR-WORKFLOW.md`, "A push does NOT advance the round".
## The testing gateway — public domains for tailnet boxes
A box on the tailnet with no public IP can still have a real HTTPS domain: the shared testing
gateway holds a wildcard record for `*.gtest.commoninternet.net` and forwards by SNI.
```bash
python3 engine/tools/gateway-domain.py add myapp # -> myapp.gtest.commoninternet.net -> this box
python3 engine/tools/gateway-domain.py list
python3 engine/tools/gateway-domain.py remove myapp
```
The admin password is `gateway.admin_password` in the secret store below; the tool reads it
itself. **The gateway does not terminate TLS** — it proxies the encrypted stream, so your box
serves the certificate for that name. Full detail, including how to get a cert and why hostname
backends are refused, is in the `gateway-domain` skill (`skills/gateway-domain/SKILL.md`).
## Secrets — one encrypted store, never in git
**Every credential on an orchestrator host lives in one sops+age encrypted file. Do not put a
secret anywhere else** — not in a git remote URL, not in a project `.env`, not in a prompt.
```
/secrets/store.yaml the store: sops+age ciphertext, mode 0600
~/.config/sops/age/keys.txt the age private key — the ONE plaintext secret, mode 0600
```
`/secrets/` is deliberately **not a git repo and has no remote**, so there is no path by
which a `git add`/`git push` can leak it; the store is ciphertext at rest anyway.
Read it with `engine/secrets.py` (stdlib + the `sops` binary, no Python deps):
```python
from secrets import get, get_group
cookie = get("tangled.cookie") # a single value
env = get_group("cc_ci_testenv") # a whole group as a dict
```
```sh
python3 engine/secrets.py list # group/key NAMES only — never prints values
python3 engine/secrets.py get tangled.cookie # one value on stdout
python3 engine/secrets.py materialize tangled-session # write a runtime file from the store
sops /secrets/store.yaml # add/edit: decrypts to $EDITOR, re-encrypts on save
```
**One home per secret — two shapes.**
*Values our code reads* live **in the store**; import this module and ask for them. Nothing is
written to disk (`engine/.tangled-session` is gone — the tangled tools read `tangled.cookie`).
*Secrets a third party reads from a fixed path* (ssh keys, a systemd `EnvironmentFile`, nix's
`authKeyFile`, a TLS keypair) live as **real files in `/secrets/files/`, symlinked from the path
the consumer expects**:
```
~/.ssh/tangled-ed25519 -> /secrets/files/tangled-ed25519
/etc/ts-auth-key -> /secrets/files/ts-auth-key
/srv/cc-ci/.testenv -> /secrets/files/cc-ci.testenv
```
The consumer is unchanged and unaware; the file exists once, in one directory, at 0600. Do **not**
also copy such a secret into `store.yaml` — that is two sources of truth again.
For a one-off where neither shape fits, inject at run time and leave nothing behind:
```sh
python3 engine/secrets.py exec-env <group> -- some-command # group as env vars
python3 engine/secrets.py with-file <group.key> -- cmd -i {} # 0600 file in a private
# tmpdir, deleted on exit
```
**The symlink exception: apps that rewrite their own credential file.** An app that refreshes an
OAuth token by writing `auth.json` atomically (write-temp + rename) **replaces the symlink with a
regular file**, silently splitting the home again. `~/.local/share/opencode/auth.json` is such a
file, so it stays where it is and is deliberately *not* centralised. Before symlinking a secret,
ask whether its owner ever writes it back.
**Rules of thumb**
- Prefer ssh remotes over `https://user:pass@host/...`. A password in a remote URL is printed by
`git remote -v`, copied into every clone, and survives in `.git/config` where nobody looks.
- A private key is `chmod 600`. Check with
`find . -name '*.key' -o -name 'id_*' ! -name '*.pub' -perm /044`.
- Anything a project must keep on disk goes in `.gitignore` **and** gets its real home in the store.
---
## Nix ## Nix
A `flake.nix` provides a reproducible devShell with the runtime deps (`python311` for stdlib A `flake.nix` provides a reproducible devShell with the runtime deps (`python311` for stdlib
@@ -309,12 +507,58 @@ nix develop -c python3 agents.py selftest # or run one command in it
nix flake check # evaluate + build the devShell nix flake check # evaluate + build the devShell
``` ```
The agent CLIs themselves (`claude`, `opencode`) are **external, non-Nix tools** — install them The agent CLIs themselves (`claude`, `codex`, `opencode`) are **external, non-Nix tools** — install them
per their own docs and make sure they are on `PATH` before launching live agents. The devShell per their own docs and make sure they are on `PATH` before launching live agents. The devShell
documents this in its banner. documents this in its banner.
--- ---
## Testing
The `tests/` directory holds the harness's own test suite. One runner drives everything:
- `tests/test_unit.py` — the harness: config load, kickoff, the phase machine, limit parsing,
waiting-until, the build-process detector.
- `tests/test_tools.py` — the standalone tools (`tangled_pr.py`, `tangled_pr_close.py`,
`tools/gateway-domain.py`). **No network**: every HTTP boundary is injected as a fake `_fetch`.
**Mock the seam you own, not the tool you depend on.** The build-detector tests used to fake
`pgrep` and `ps` subprocess calls, so they passed on a host where neither was reachable and the
real defect — empty output read as "no build running" — was invisible to every test. They now
patch `_proc_descendants` and `_comms`, the functions this repo owns. A test that mocks a
dependency proves the mock works.
```bash
nix develop -c ./tests/run.sh # unit tests always; live backend smokes when available
# or just: ./tests/run.sh # (python3 + tmux must be on PATH)
```
What it runs:
- **Unit tests** (`tests/test_unit.py`) — pure logic, **no agents spawned, no live tmux sessions**.
Cover config load + defaults merge, kickoff-template assembly, the phase machine (advance on the
done marker, idempotent sequence-complete, append-a-phase resumes), usage-limit reset-banner
parsing, `WAITING-UNTIL` / stall parsing, and the per-backend activity detectors (claude +
opencode footers). Always run; a failure fails the suite. Run them alone with
`python3 -m unittest discover -s tests` (or `python3 tests/test_unit.py`).
- **Live backend smokes** (`tests/smoke_claude.sh`, `tests/smoke_codex.sh`,
`tests/smoke_opencode.sh`) — each brings a
throwaway scratch project up **through `agents.py`** on a real backend, in a fully isolated
sandbox (its own unique `session_prefix`, a temp `log_dir`, and — for opencode — a dedicated
server on a non-default port `AOTEST_OC_PORT`, default `4097`), confirms the session attaches and
`status` reports it RUNNING, then `down`s it and cleans up (no leftover sessions, port freed).
Each **SKIPs gracefully** (exit 0) when its backend's binary or creds are unavailable. Useful env:
`CLAUDE_BIN` / `CODEX_BIN` / `OPENCODE_BIN`, `AOTEST_MODEL`, `AOTEST_OC_PORT`,
`AOTEST_OC_CREDS`.
- **Isolation sanity** — after the live runs, the runner asserts no `aotest-*` tmux sessions leaked
and reports that any live sessions are untouched.
The smokes are safe by construction: a unique per-run session prefix (never `cc-ci-` or any real
project's), a dedicated opencode port (never `4096`), and a cleanup trap that fires on success,
failure, and Ctrl+C.
---
## Adding things ## Adding things
- **Add an agent** — add an `[[agent]]` block; `agents.py up <name>`. No code change. - **Add an agent** — add an `[[agent]]` block; `agents.py up <name>`. No code change.
+20 -1
View File
@@ -6,7 +6,7 @@
# #
# This example is self-contained: its agents use a dependency-free `demo` backend (a shell that # This example is self-contained: its agents use a dependency-free `demo` backend (a shell that
# just idles), so the whole project can be brought up and torn down with no external agent CLI — # just idles), so the whole project can be brought up and torn down with no external agent CLI —
# see ./smoke.sh. The `claude` and `opencode` backends below are the real ones; point an agent at # see ./smoke.sh. The `claude`, `codex`, and `opencode` backends below are real; point an agent at
# them with `backend = "claude"` for a live run. # them with `backend = "claude"` for a live run.
# ─────────────────────────── global watchdog cadence ─────────────────────────── # ─────────────────────────── global watchdog cadence ───────────────────────────
@@ -42,11 +42,30 @@ supports_resume = true
prompt_delivery = "arg" prompt_delivery = "arg"
process_name = "claude" # used for backend-mismatch healing process_name = "claude" # used for backend-mismatch healing
submit_key = "Enter" submit_key = "Enter"
startup_prompt_re = "Trusting the directory|Do you trust"
startup_prompt_response = "1"
startup_prompt_delay = 2
stall_idle = 300 stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+" active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)" limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified" fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.codex] # Codex TUI + the shared Remote Control app-server daemon
bin = "codex"
preamble = "{bin} remote-control start --json >/dev/null"
remote_addr = "unix://" # make the TUI a client of the daemon; it must not own a second writer
flags = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
supports_resume = false # the harness does not yet persist Codex thread IDs
prompt_delivery = "arg"
process_name = "codex"
footer_ui = true
log_grace = 180
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|working|thinking|running|searching|exploring|implementing"
limit_re = "usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "connection is errored|not logged in|authentication required"
[backend.opencode] # the real opencode backend (a TUI; prompt typed after connect) [backend.opencode] # the real opencode backend (a TUI; prompt typed after connect)
bin = "opencode" bin = "opencode"
attach = "{bin} attach {server} --dir {dir}" attach = "{bin} attach {server} --dir {dir}"
+568 -31
View File
@@ -14,6 +14,7 @@ Usage:
agents.py watchdog the supervisor loop (reads the config every tick) agents.py watchdog the supervisor loop (reads the config every tick)
agents.py logs <name> tail an agent's session log agents.py logs <name> tail an agent's session log
agents.py phase [set N|next|show] inspect / move the loop phase agents.py phase [set N|next|show] inspect / move the loop phase
agents.py tokens per-phase token + time report (needs [watchdog].log_tokens = true)
agents.py selftest backend activity-detector regression checks (no config needed) agents.py selftest backend activity-detector regression checks (no config needed)
agents.py init [dir] scaffold a starter agents.toml + prompts/ in a project dir agents.py init [dir] scaffold a starter agents.toml + prompts/ in a project dir
@@ -65,6 +66,7 @@ def load_config(path):
"backends": raw.get("backend", {}), "backends": raw.get("backend", {}),
"defaults": defaults, "defaults": defaults,
"loop": raw.get("loop", {}), "loop": raw.get("loop", {}),
"pipeline": raw.get("pipeline", {}),
"project_dir": str(project_dir), "project_dir": str(project_dir),
"log_dir": str(_resolve(project_dir, log_dir_raw)), "log_dir": str(_resolve(project_dir, log_dir_raw)),
"session_prefix": session_prefix, "session_prefix": session_prefix,
@@ -347,15 +349,45 @@ def start_agent(cfg, agent, *, force=False):
if backend.get("remote_control"): if backend.get("remote_control"):
parts.append(_render_template(backend.get("remote_control_flag", parts.append(_render_template(backend.get("remote_control_flag",
"--remote-control '{session}'"), {"session": session})) "--remote-control '{session}'"), {"session": session}))
if backend.get("remote_addr"):
remote_values = {
"addr": backend["remote_addr"], "session": session, "model": model,
"dir": shlex.quote(cwd),
}
parts.append(_render_template(
backend.get("remote_addr_flag", "--remote '{addr}'"), remote_values))
remote_cwd_flag = backend.get("remote_cwd_flag", "-C {dir}")
if remote_cwd_flag:
parts.append(_render_template(remote_cwd_flag, remote_values))
if model: if model:
parts.append(_render_template(backend.get("model_flag", "--model '{model}'"), {"model": model})) parts.append(_render_template(backend.get("model_flag", "--model '{model}'"), {"model": model}))
if backend.get("flags"): if backend.get("flags"):
parts.append(backend["flags"]) parts.append(backend["flags"])
parts.append(f"\"$(cat '{kf}')\"") parts.append(f"\"$(cat '{kf}')\"")
cmd = " ".join(p for p in parts if p) cmd = " ".join(p for p in parts if p)
# Some interactive CLIs need an idempotent service prepared before their TUI starts.
# The Codex backend starts the shared app-server here, while remote_addr above makes the
# pane's TUI a client of that daemon so both local and Remote UIs share one thread writer.
if backend.get("preamble"):
preamble = _render_template(backend["preamble"], {
"bin": backend["bin"], "session": session, "model": model,
"dir": shlex.quote(cwd),
})
cmd = f"{preamble} && {cmd}"
log(f"starting {session} ({agent['backend']}, kind={agent['kind']}, phase={pid}, " log(f"starting {session} ({agent['backend']}, kind={agent['kind']}, phase={pid}, "
f"model={model or 'default'}{', resume' if rid else ''})") f"model={model or 'default'}{', resume' if rid else ''})")
new_session(session, cwd, cmd, log_path) new_session(session, cwd, cmd, log_path)
# An unattended TUI may present a deterministic first-run gate before it processes the
# prompt already supplied on argv (Codex asks whether to trust a new project directory).
# Backends opt in with both the screen regex and response; nothing is accepted implicitly.
startup_re = backend.get("startup_prompt_re")
if startup_re:
time.sleep(int(backend.get("startup_prompt_delay", 2)))
if re.search(startup_re, capture_pane(session, 40), re.I):
_run(["tmux", "send-keys", "-t", TP(session), "-l", "--",
str(backend.get("startup_prompt_response", ""))])
_run(["tmux", "send-keys", "-t", TP(session),
backend.get("submit_key", "Enter")])
def start_service(cfg, svc): def start_service(cfg, svc):
session = svc["session"] session = svc["session"]
@@ -465,6 +497,109 @@ def limit_tick(cfg, agent, pane):
# ── stall detection ────────────────────────────────────────────────────────────── # ── stall detection ──────────────────────────────────────────────────────────────
_idle_since: dict[str, float] = {} _idle_since: dict[str, float] = {}
_done_nudged: dict[str, bool] = {} # per-session: sent the one-time "write the done marker" nudge this phase
_build_deferred: set = set() # per-session: currently deferring a stall because a real build is running
# Default set of process names (comm) that mean "genuine long-running work is happening" — a compile,
# coverage run, mutation run, or a real e2e test driving the server/browser. If one of these is a
# descendant of the agent's tmux pane, a silent pane is a running build, NOT a stall. High-signal names
# only (no bare python/node/bash — those match the orchestrator's own engine + would false-positive).
DEFAULT_BUILD_PROCS_RE = (r"^(cargo|cargo-llvm-cov|cargo-mutants|cargo-nextest|nextest|rustc|rustdoc|"
r"cc1|cc1plus|collect2|lld|ld\.lld|llvm-cov|llvm-profdata|"
r"lichen-server|lichen-cms|lichen-shell|lichen-cli|chromium|chrome|playwright)$")
def _proc_descendants(roots):
"""All descendant PIDs of the given root PIDs (BFS via pgrep -P), EXCLUDING the roots themselves —
the pane root is the claude process whose args embed the prompt (which mentions cargo/rustc/…), so
matching it would false-positive; we only inspect its real child processes.
Reads /proc directly rather than shelling out to `pgrep -P`. WHY (2026-08-21): pgrep is not
installed on every host we run on, and a missing binary makes `subprocess.run(shell=True)`
return rc=127 with EMPTY stdout — indistinguishable from "this process has no children". That
made this function return an empty set on such a host, which made `_build_running` always False,
which let the stall detector reboot an agent in the middle of a build. Reading /proc removes the
dependency and cannot fail silently in that direction.
(Note the repo already bans `pgrep -f` for a DIFFERENT reason — it self-matches. Two distinct
traps, same tool.)"""
roots = [str(p) for p in roots if str(p).isdigit()]
children = {}
for entry in os.listdir("/proc"):
if not entry.isdigit():
continue
try:
with open(f"/proc/{entry}/stat") as fh:
fields = fh.read().rsplit(") ", 1)[-1].split() # rsplit: comm may contain spaces
ppid = fields[1]
except (OSError, IndexError):
continue # process exited mid-scan
children.setdefault(ppid, []).append(entry)
seen, stack = set(), list(roots)
while stack:
pid = stack.pop()
if pid in seen:
continue
seen.add(pid)
stack += children.get(pid, [])
return seen - set(roots)
def _build_running(cfg, agent):
"""True if a real build/coverage/test process is running under the agent's tmux session, so a long
silent pane isn't mistaken for a stall. Matches process comm (never the claude root's args). Bounded
by stall_idle_max upstream so a genuinely hung build still gets rebooted eventually."""
session = agent["session"]
r = subprocess.run(f"tmux list-panes -t {session!r} -F '#{{pane_pid}}'",
shell=True, capture_output=True, text=True)
kids = _proc_descendants(r.stdout.split())
if not kids:
return False
try:
rx = re.compile(cfg["watchdog"].get("build_procs_re", DEFAULT_BUILD_PROCS_RE))
except re.error:
rx = re.compile(DEFAULT_BUILD_PROCS_RE)
# Read /proc/<pid>/comm rather than `ps -o comm=`: ps is absent on some hosts we run on, and
# an absent binary yields EMPTY stdout, which this function cannot tell from "no build is
# running" — so the stall detector would reboot an agent mid-build. Same class as the pgrep
# dependency removed from _proc_descendants above; both were invisible because the unit tests
# MOCKED these subprocess calls, so the suite exercised the fake and never the dependency.
return any(rx.match(c) for c in _comms(kids))
def _comms(pids):
"""The comm (process name) of each pid, skipping any that exited mid-scan. No external tools."""
out = []
for pid in sorted(pids):
try:
with open(f"/proc/{pid}/comm") as fh:
c = fh.read().strip()
except OSError:
continue
if c:
out.append(c)
return out
def _done_nudge_msg(cfg, ph):
"""The DONE-nudge: prompts a stalled loop agent to finalize a built-but-unmarked phase."""
dm = cfg["loop"].get("done_marker", "## DONE")
pid = ph.get("id", "")
status = ph.get("status", f"STATUS-{pid}.md")
sub = _state_subdir(cfg)
return (f"watchdog nudge: you've stalled in phase '{pid}', which is NOT yet marked '{dm}'. Resume now "
f"— pull any pending review/inbox and continue. If (and ONLY if) every DoD item has a fresh "
f"PASS from BOTH adversaries with no standing veto, write '{dm}' to {sub}/{status} and push, "
f"so the phase settles and auto-advances. Do not stay idle.")
def _pane_last_active(session):
"""Unix timestamp of the tmux window's last activity (last output change), or None.
Seeds idle-duration from the agent's REAL last activity rather than `now`, so stalls are
detected regardless of when the watchdog process started — restarting the watchdog no longer
resets every agent's stall clock."""
r = subprocess.run(f"tmux display-message -p -t {session!r} '#{{window_activity}}'",
shell=True, capture_output=True, text=True)
try:
return float(r.stdout.strip())
except (ValueError, AttributeError):
return None
def _last_nonempty_line(text): def _last_nonempty_line(text):
for line in reversed(text.splitlines()): for line in reversed(text.splitlines()):
@@ -473,17 +608,16 @@ def _last_nonempty_line(text):
return "" return ""
def _parse_waiting_until(cfg, agent, pane): def _parse_waiting_until(cfg, agent, pane):
if backend_of(cfg, agent).get("footer_ui"): # Only consulted once the pane is already idle (see stall_check_one), so scanning the whole
line = _last_nonempty_line(pane) # capture and taking the MOST-RECENT marker is safe — and it's the only thing that works for a
if not line.startswith("WAITING-UNTIL:"): # footer_ui backend (claude/opencode), whose input-box footer always renders BELOW the agent's
return None # final message. The footer never contains the marker, so the last match is the agent's own
m = re.search(r"WAITING-UNTIL:\s*(\S+)", line) # signal, whether or not a status footer follows it.
else: matches = re.findall(r"WAITING-UNTIL:\s*(\S+)", pane)
m = re.search(r"WAITING-UNTIL:\s*(\S+)", pane) if not matches:
if not m:
return None return None
try: try:
return datetime.fromisoformat(m.group(1).replace("Z", "+00:00")).timestamp() return datetime.fromisoformat(matches[-1].replace("Z", "+00:00")).timestamp()
except Exception: except Exception:
return None return None
@@ -500,24 +634,69 @@ def stall_check_one(cfg, agent):
return return
if pane_active(cfg, agent, pane): if pane_active(cfg, agent, pane):
_idle_since[session] = 0.0 _idle_since[session] = 0.0
_build_deferred.discard(session)
return return
since = _idle_since.get(session) or now # Seed from the pane's real last-activity (not `now`), so a watchdog that just (re)started still
# sees an already-idle pane as idle-for-its-true-duration instead of resetting the clock.
since = _idle_since.get(session) or _pane_last_active(session) or now
_idle_since[session] = since _idle_since[session] = since
idle = now - since idle = now - since
grace = int(cfg["watchdog"].get("stall_grace", 180)) grace = int(cfg["watchdog"].get("stall_grace", 180))
until = _parse_waiting_until(cfg, agent, pane) until = _parse_waiting_until(cfg, agent, pane)
if until is not None: if until is not None:
if now <= until + grace: # An agent that starts a long remote/async run (remote cargo build, terraform apply, VM
# provision, long ssh) prints `WAITING-UNTIL: <ISO8601>` so the watchdog holds off instead
# of killing it mid-run. Cap how far out it can push its own reboot, so a runaway can't park
# itself forever ("some max no matter what").
wu_max = int(cfg["watchdog"].get("waiting_until_max", 7200))
building = _build_running(cfg, agent)
# A build still running under the session (cargo-mutants, a coverage run, a remote ssh) is
# PROOF OF LIFE, and it outranks the agent's own deadline: the deadline is only an estimate,
# and an agent blocked on a shell cannot re-emit a fresh marker to extend it — so killing a
# live build at its estimate throws the work away for nothing. `waiting_until_max` is the one
# absolute bound: past it we reboot even mid-build, which is what catches a genuinely HUNG
# build (the case the cap exists for).
if wu_max and idle > wu_max:
reason = (f"idle {int(idle)}s past the {wu_max}s WAITING-UNTIL cap "
f"({'build still running — treating it as hung' if building else 'no build running'}) "
f"— rebooting regardless")
elif building:
return return
elif now <= until + grace:
return
else:
reason = f"past its WAITING-UNTIL by {int(now-until)}s — self-wake did not fire" reason = f"past its WAITING-UNTIL by {int(now-until)}s — self-wake did not fire"
else: else:
stall_idle = int(backend_of(cfg, agent).get("stall_idle", 300)) stall_idle = int(backend_of(cfg, agent).get("stall_idle", 300))
if idle < stall_idle: if idle < stall_idle:
return return
reason = f"idle {int(idle)}s with no WAITING-UNTIL marker" # Build-aware: a silent pane with a real compile/coverage/test process running is NOT a stall —
# defer the reboot until it finishes, but never past stall_idle_max (hard cap for a hung build).
stall_idle_max = int(backend_of(cfg, agent).get("stall_idle_max", 1800))
if idle < stall_idle_max and _build_running(cfg, agent):
if session not in _build_deferred:
log(f"stall-defer: {agent['name']} ({session}) idle {int(idle)}s but a build/test is "
f"running — waiting (hard cap {stall_idle_max}s)")
_build_deferred.add(session)
return
reason = (f"idle {int(idle)}s past build-aware hard cap {stall_idle_max}s — rebooting regardless"
if idle >= stall_idle_max else
f"idle {int(idle)}s with no WAITING-UNTIL marker and no build running")
# Ceremony-lag guard: a loop agent idling in a phase that's built but NOT marked done won't let the
# phase advance (the recurring "all gates PASS but no ## DONE written" stall). Nudge it ONCE per phase
# to finalize (write the done marker if the DoD is met) before falling back to the blunt kill+reboot.
if (cfg["loop"].get("done_nudge", True) and agent.get("kind") == "loop" and phases(cfg)
and not phase_done(cfg, cur_phase(cfg).get("status", "")) and not _done_nudged.get(session)):
log(f"stall: {agent['name']} ({session}) {reason} — DONE-nudge (phase built but not marked done)")
ping_session(session, _done_nudge_msg(cfg, cur_phase(cfg)),
submit_key=backend_of(cfg, agent).get("submit_key", "Enter"))
_done_nudged[session] = True
_idle_since[session] = now # fresh idle window to act on the nudge before reboot escalates
return
log(f"stall: {agent['name']} ({session}) {reason} — kill + reboot") log(f"stall: {agent['name']} ({session}) {reason} — kill + reboot")
start_agent(cfg, agent, force=True) start_agent(cfg, agent, force=True)
_idle_since[session] = 0.0 _idle_since[session] = 0.0
_build_deferred.discard(session)
# ── healing ────────────────────────────────────────────────────────────────────── # ── healing ──────────────────────────────────────────────────────────────────────
@@ -563,6 +742,15 @@ def wake_agent(cfg, agent):
if not wake: if not wake:
return True return True
session = agent["session"] session = agent["session"]
# A one-shot `task` is "woken" by RE-RUNNING it fresh — it has no persistent REPL to re-prompt — so
# scheduled work (e.g. a coverage audit) recurs autonomously on its interval, no operator needed.
# Skip only while its previous run is still going; otherwise kill + restart for a clean re-run.
if agent.get("kind") == "task":
if session_alive(session) and pane_active(cfg, agent, capture_pane(session, 25)):
return False
log(f"wake: re-running task {agent['name']} ({session})")
start_agent(cfg, agent, force=True)
return True
if not session_alive(session): if not session_alive(session):
return False return False
backend = backend_of(cfg, agent) backend = backend_of(cfg, agent)
@@ -598,15 +786,22 @@ def _show_pushed(cfg, repo, path):
return r.stdout return r.stdout
return "" return ""
def _ping_agents(cfg, value, default, msg):
"""Ping one or more agents. `value` is an agent name, a LIST of names, or falsy (→ default).
Each target is pinged in its own session with its own backend's submit key — so a handoff can
notify multiple reviewers (e.g. claim_pings = ["correctness-adversary", "readability-adversary"])."""
names = value if isinstance(value, list) else [value or default]
for name in names:
agent = cfg["agents"].get(name)
session = agent["session"] if agent and agent.get("session") else (cfg["session_prefix"] + str(name))
submit = backend_of(cfg, agent).get("submit_key", "Enter") if agent else "Enter"
ping_session(session, msg, submit_key=submit)
def handoff_check(cfg): def handoff_check(cfg):
h = cfg["loop"].get("handoff") h = cfg["loop"].get("handoff")
if not h: if not h:
return return
repo = handoff_repo(cfg) repo = handoff_repo(cfg)
sub = lambda name: cfg["agents"].get(name, {}).get("session", cfg["session_prefix"] + name)
builder_name = h.get("review_pings", "builder")
submit = (backend_of(cfg, cfg["agents"][builder_name]).get("submit_key", "Enter")
if builder_name in cfg["agents"] else "Enter")
claim_pat = h.get("claim_pattern", "^claim") claim_pat = h.get("claim_pattern", "^claim")
review_pat = h.get("review_pattern", "^review") review_pat = h.get("review_pattern", "^review")
_git(repo, "fetch -q origin") _git(repo, "fetch -q origin")
@@ -617,15 +812,15 @@ def handoff_check(cfg):
elif head != _hand["sha"]: elif head != _hand["sha"]:
subjects = _git(repo, f"log --format=%s {_hand['sha']}..origin/main").stdout subjects = _git(repo, f"log --format=%s {_hand['sha']}..origin/main").stdout
if re.search(claim_pat, subjects, re.M | re.I): if re.search(claim_pat, subjects, re.M | re.I):
log("handoff: claim commit → pinging reviewer") log("handoff: claim commit → pinging reviewer(s)")
ping_session(sub(h.get("claim_pings", "adversary")), _ping_agents(cfg, h.get("claim_pings", "adversary"), "adversary",
"watchdog ping: the other loop pushed a gate CLAIM commit. " "watchdog ping: the other loop pushed a gate CLAIM commit. "
"Pull and verify the claimed gate now.", submit_key=submit) "Pull and verify the claimed gate now.")
if re.search(review_pat, subjects, re.M | re.I): if re.search(review_pat, subjects, re.M | re.I):
log("handoff: review commit → pinging builder") log("handoff: review commit → pinging builder")
ping_session(sub(h.get("review_pings", "builder")), _ping_agents(cfg, h.get("review_pings", "builder"), "builder",
"watchdog ping: the other loop pushed a verdict/finding commit. " "watchdog ping: the other loop pushed a verdict/finding commit. "
"Pull the review file and act.", submit_key=submit) "Pull the review file and act.")
_hand["sha"] = head _hand["sha"] = head
inboxes = h.get("inboxes", []) inboxes = h.get("inboxes", [])
md5 = lambda s: hashlib.md5(s.encode()).hexdigest() md5 = lambda s: hashlib.md5(s.encode()).hexdigest()
@@ -641,9 +836,9 @@ def handoff_check(cfg):
hh = md5(content) hh = md5(content)
if hh != _hand[key]: if hh != _hand[key]:
log(f"handoff: {fname} changed → pinging {target}") log(f"handoff: {fname} changed → pinging {target}")
ping_session(sub(target), _ping_agents(cfg, target, target,
f"watchdog ping: the other loop pushed {sub_dir}/{fname} — pull, read it, " f"watchdog ping: the other loop pushed {sub_dir}/{fname} — pull, read it, "
f"act, then delete the file (commit + push) to mark it consumed.", submit_key=submit) f"act, then delete the file (commit + push) to mark it consumed.")
_hand[key] = hh _hand[key] = hh
else: else:
_hand[key] = "" _hand[key] = ""
@@ -663,6 +858,186 @@ def start_loops(cfg):
for a in loop_agents(cfg): for a in loop_agents(cfg):
start_agent(cfg, a) start_agent(cfg, a)
# ── optional per-phase token + time logging (log_tokens) ──────────────────────────
# When [watchdog].log_tokens (or [loop].log_tokens) is true, the watchdog records, for each phase,
# how many tokens each agent used and how long the phase took, appended to <log_dir>/token-log.jsonl.
# Tokens are summed from each agent's Claude Code session transcript, attributed by working dir — so
# give each agent its OWN dir for accurate per-agent numbers (the Builder/Adversary loop pair already
# uses separate clones). View with: agents.py tokens.
def log_tokens_enabled(cfg):
return bool(cfg.get("watchdog", {}).get("log_tokens") or cfg.get("loop", {}).get("log_tokens"))
def _transcript_dir(workdir):
name = str(workdir).rstrip("/").replace("/", "-").replace(".", "-")
return Path(os.path.expanduser("~/.claude/projects")) / name
def _sum_tokens(workdir):
t = {"input": 0, "output": 0, "cache_create": 0, "cache_read": 0}
d = _transcript_dir(workdir)
if d.is_dir():
for f in d.glob("*.jsonl"):
try:
for line in f.open(errors="ignore"):
try:
o = json.loads(line)
except Exception:
continue
if o.get("type") == "assistant":
u = (o.get("message", {}) or {}).get("usage", {}) or {}
t["input"] += u.get("input_tokens", 0) or 0
t["output"] += u.get("output_tokens", 0) or 0
t["cache_create"] += u.get("cache_creation_input_tokens", 0) or 0
t["cache_read"] += u.get("cache_read_input_tokens", 0) or 0
except OSError:
continue
t["total"] = t["input"] + t["output"] + t["cache_create"] + t["cache_read"]
return t
def _token_cumulative(cfg):
"""Cumulative tokens per agent so far, summed from each agent's transcript dir."""
return {a["name"]: _sum_tokens(a["dir"]) for a in cfg["agents"].values()}
_TOKEN_KEYS = ("input", "output", "cache_create", "cache_read", "total")
def _token_state_path(cfg): return Path(cfg["state_dir"]) / "token-phase.json"
def _token_log_path(cfg): return Path(cfg["log_dir"]) / "token-log.jsonl"
def _tok_delta(cur, base): return {k: cur.get(k, 0) - base.get(k, 0) for k in _TOKEN_KEYS}
def token_phase_begin(cfg, phase_id):
"""Set the baseline (cumulative tokens + start time) for the phase now starting. Idempotent
across watchdog restarts: keeps the original baseline if already tracking this phase."""
if not log_tokens_enabled(cfg):
return
sf = _token_state_path(cfg)
try:
if json.loads(sf.read_text()).get("phase_id") == phase_id:
return
except Exception:
pass
sf.write_text(json.dumps({"phase_id": phase_id,
"started": datetime.now().isoformat(timespec="seconds"),
"baseline": _token_cumulative(cfg)}))
def token_phase_flush(cfg, next_phase_id):
"""Close the current phase: append its per-agent + total token deltas and duration to the
token-log, then re-baseline for next_phase_id (or finalize tracking if None)."""
if not log_tokens_enabled(cfg):
return
sf = _token_state_path(cfg)
try:
st = json.loads(sf.read_text())
except Exception:
return
cur = _token_cumulative(cfg)
base = st.get("baseline", {})
started = st.get("started")
try:
dur = round((datetime.now() - datetime.fromisoformat(started)).total_seconds(), 1)
except Exception:
dur = None
per_agent = {n: _tok_delta(cur.get(n, {}), base.get(n, {})) for n in cur}
total = {k: sum(per_agent[n][k] for n in per_agent) for k in _TOKEN_KEYS}
rec = {"phase_id": st.get("phase_id"), "started": started,
"ended": datetime.now().isoformat(timespec="seconds"), "duration_s": dur,
"agents": per_agent, "total": total}
with _token_log_path(cfg).open("a") as fh:
fh.write(json.dumps(rec) + "\n")
parts = ", ".join(f"{n}={per_agent[n]['total']:,}" for n in per_agent)
log(f"[log_tokens] phase {rec['phase_id']}: {total['total']:,} tok in {dur}s ({parts})")
if next_phase_id is not None:
sf.write_text(json.dumps({"phase_id": next_phase_id,
"started": datetime.now().isoformat(timespec="seconds"),
"baseline": cur}))
else:
sf.unlink(missing_ok=True)
# ── token logging granularity: per phase, or also per GATE (log_tokens) ────────────
# Phases are tracked per phase (token_phase_begin/flush). With token_granularity="gate" (the default)
# tokens are ALSO attributed to each gate. A "gate" is a claimed unit — any `claim(<label>)` commit on
# the work repo's origin/main (e.g. claim(D1-D5), claim(feat:multi-file)); a leading "feat:" is
# stripped for readability. A change in the most-recently-claimed label is a boundary; on each
# boundary the previous gate's per-agent token delta + duration is appended to token-log.jsonl tagged
# phase_id="<phase>:<label>", so `agents.py tokens` lists it as its own row. The per-phase rollup
# record is written either way; "phase" granularity logs only that.
_gate_claim_re = re.compile(r"^claim\(\s*([^)]+?)\s*\)", re.I)
def token_granularity(cfg):
"""'gate' (default; per claimed gate, plus the per-phase rollup) or 'phase' (per phase only)."""
g = (cfg.get("watchdog", {}).get("token_granularity")
or cfg.get("loop", {}).get("token_granularity") or "gate")
return g if g in ("gate", "phase") else "gate"
def _token_gate_state_path(cfg):
return Path(cfg["state_dir"]) / "token-gate.json"
def _latest_claimed_gate(cfg):
"""Label of the most-recent `claim(<label>)` subject on the work repo's origin/main, or None.
A leading 'feat:' is stripped so feature gates read as their bare name."""
r = _git(handoff_repo(cfg), "log -1 --format=%s --grep 'claim(' origin/main")
m = _gate_claim_re.match((r.stdout or "").strip())
if not m:
return None
label = m.group(1).strip()
return label[5:].strip() if label.lower().startswith("feat:") else label
def _write_token_delta(cfg, phase_id, st):
"""Append one per-agent token delta record (vs the baseline in st) to token-log.jsonl."""
cur = _token_cumulative(cfg)
base = st.get("baseline", {})
started = st.get("started")
try:
dur = round((datetime.now() - datetime.fromisoformat(started)).total_seconds(), 1)
except Exception:
dur = None
per_agent = {n: _tok_delta(cur.get(n, {}), base.get(n, {})) for n in cur}
total = {k: sum(per_agent[n][k] for n in per_agent) for k in _TOKEN_KEYS}
rec = {"phase_id": phase_id, "started": started,
"ended": datetime.now().isoformat(timespec="seconds"), "duration_s": dur,
"agents": per_agent, "total": total}
with _token_log_path(cfg).open("a") as fh:
fh.write(json.dumps(rec) + "\n")
return rec, per_agent, total
def gate_token_flush(cfg):
"""Close out the currently-tracked gate (if any): write its token delta, then drop the state."""
sf = _token_gate_state_path(cfg)
try:
st = json.loads(sf.read_text())
except Exception:
return
gate = st.get("gate")
if not gate:
return
phase_id = f"{st['phase']}:{gate}" if st.get("phase") else gate
rec, per_agent, total = _write_token_delta(cfg, phase_id, st)
parts = ", ".join(f"{n}={per_agent[n]['total']:,}" for n in per_agent)
log(f"[log_tokens] gate {phase_id}: {total['total']:,} tok in {rec['duration_s']}s ({parts})")
try:
sf.unlink()
except Exception:
pass
def gate_token_check(cfg):
"""When token_granularity=='gate', detect gate boundaries and flush per-gate token deltas."""
if not log_tokens_enabled(cfg) or token_granularity(cfg) != "gate":
return
current = _latest_claimed_gate(cfg)
if not current:
return
sf = _token_gate_state_path(cfg)
try:
tracked = json.loads(sf.read_text()).get("gate")
except Exception:
tracked = None
if tracked == current:
return
if tracked:
gate_token_flush(cfg) # close out the previous gate before starting the next
sf.write_text(json.dumps({"gate": current, "phase": cur_phase(cfg).get("id"),
"started": datetime.now().isoformat(timespec="seconds"),
"baseline": _token_cumulative(cfg)}))
log(f"[log_tokens] tracking gate: {current}")
def phase_advance_check(cfg): def phase_advance_check(cfg):
"""On heavy tick: if the current phase is DONE, advance (or finish the sequence). """On heavy tick: if the current phase is DONE, advance (or finish the sequence).
@@ -678,20 +1053,25 @@ def phase_advance_check(cfg):
ph = ps[idx] ph = ps[idx]
if not phase_done(cfg, ph["status"]): if not phase_done(cfg, ph["status"]):
return False return False
if log_tokens_enabled(cfg) and token_granularity(cfg) == "gate":
gate_token_flush(cfg) # close out the last in-flight gate before leaving the phase
nxt = idx + 1 nxt = idx + 1
if nxt < len(ps): if nxt < len(ps):
log(f"PHASE {ph['id']} DONE — auto-transitioning to {ps[nxt]['id']}") log(f"PHASE {ph['id']} DONE — auto-transitioning to {ps[nxt]['id']}")
token_phase_flush(cfg, ps[nxt]["id"])
stop_loops(cfg) stop_loops(cfg)
Path(phase_idx_file(cfg)).write_text(str(nxt)) Path(phase_idx_file(cfg)).write_text(str(nxt))
if marker.exists(): if marker.exists():
marker.unlink() # resuming into a (freshly-appended) phase — clear stale completion marker.unlink() # resuming into a (freshly-appended) phase — clear stale completion
handoff_reset() handoff_reset()
_done_nudged.clear() # fresh DONE-nudge budget for the new phase
start_loops(cfg) start_loops(cfg)
return True return True
# last phase is DONE → sequence complete # last phase is DONE → sequence complete
if marker.exists(): if marker.exists():
return False # already handled — idempotent (no re-log, no re-stop) return False # already handled — idempotent (no re-log, no re-stop)
log(f"PHASE SEQUENCE COMPLETE (last phase {ph['id']} DONE) — stopping loops") log(f"PHASE SEQUENCE COMPLETE (last phase {ph['id']} DONE) — stopping loops")
token_phase_flush(cfg, None)
stop_loops(cfg) stop_loops(cfg)
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S") ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
marker.write_text(f"phase sequence complete {ts}. Loops stopped; build finished.\n") marker.write_text(f"phase sequence complete {ts}. Loops stopped; build finished.\n")
@@ -712,6 +1092,96 @@ def watched(cfg):
return [a for a in cfg["agents"].values() return [a for a in cfg["agents"].values()
if a.get("enabled", True) and a.get("watch", "none") != "none"] if a.get("enabled", True) and a.get("watch", "none") != "none"]
# ── standalone-agent pipeline (sequential phases via completion markers) ──────────
# A [pipeline] block runs a sequence of DISTINCT agents (different prompts / models / dirs) one at a
# time, advancing when each writes its completion marker — the standalone-agent analog of the loop
# phase machine. Stateless: the markers (which agents write in their OWN cwd, e.g. work-fork/.ao-state/)
# are the source of truth, so the watchdog reconciles the desired state every tick — exactly one stage
# runs, earlier (done) stages are retired, later stages wait. Declare pipeline agents enabled=false: the
# pipeline (not `up`/watched()) owns their lifecycle; the ACTIVE stage is still stall/heal-watched.
#
# [pipeline]
# enabled = true
# stages = [
# { agent = "fork-e2e", done = "work-fork/.ao-state/FORK-E2E-COMPLETE" },
# { agent = "fork-iroh", done = "work-fork/.ao-state/FORK-IROH-COMPLETE" },
# ]
def pipeline_stages(cfg):
pl = cfg.get("pipeline") or {}
if not pl.get("enabled", True):
return []
return pl.get("stages", [])
def _pipeline_marker(cfg, stage):
p = Path(stage["done"])
return p if p.is_absolute() else Path(cfg["project_dir"]) / stage["done"]
def _pipeline_push_on_retire(agent):
"""Best-effort `git push` in a completing stage's dir before it's retired, so its final commits
aren't stranded locally if it wrote its completion marker before its last push landed (a race:
the pipeline retires a stage the instant its marker appears). Never raises; logs the outcome."""
d = agent.get("dir")
if not d or not (Path(d) / ".git").exists():
return
try:
r = subprocess.run(["git", "-C", d, "push", "origin", "HEAD:main"],
env={**os.environ, "GIT_SSH_COMMAND": "ssh -o BatchMode=yes"},
capture_output=True, text=True, timeout=90)
if r.returncode == 0:
log(f"pipeline: pushed {agent['name']!r} before retiring (no stranded commits)")
elif "up-to-date" not in (r.stderr + r.stdout).lower():
log(f"pipeline: push-on-retire for {agent['name']!r} nonzero (non-fatal): "
f"{(r.stderr or r.stdout).strip().splitlines()[-1:] }")
except Exception as e:
log(f"pipeline: push-on-retire for {agent['name']!r} failed (non-fatal): {e}")
def pipeline_check(cfg):
"""Reconcile the [pipeline]: run the first stage whose completion marker is absent, retire every
other pipeline agent. Returns the ACTIVE stage's agent dict (so the watchdog stall/heal-watches it),
or None if there's no pipeline or it's fully complete."""
stages = pipeline_stages(cfg)
if not stages:
return None
active = None
for st in stages:
if not _pipeline_marker(cfg, st).exists():
active = cfg["agents"].get(st.get("agent"))
break
active_session = active["session"] if active else None
for st in stages:
ag = cfg["agents"].get(st.get("agent"))
if not ag:
continue
alive = session_alive(ag["session"])
if ag["session"] == active_session:
if not alive:
log(f"pipeline: advancing → start stage {ag['name']!r}")
start_agent(cfg, ag, force=True)
elif alive:
# Retiring a non-active stage. If it's a COMPLETED stage (its marker exists), push its repo
# first so final commits written just before the marker aren't stranded locally.
if _pipeline_marker(cfg, st).exists():
_pipeline_push_on_retire(ag)
log(f"pipeline: retire stage {ag['name']!r} (complete or not yet active)")
kill_session(ag["session"])
# PIPELINE-COMPLETE must MIRROR the current stage list, not latch on first completion. Recompute it
# from every stage's marker each tick: write it only when ALL current markers exist, and CLEAR a stale
# one the moment any stage is incomplete. Without the clear, appending new stages to a pipeline that
# had already completed leaves a lying "done" sentinel (the appended stages run, but the file says the
# pipeline finished). `active is None` is not a safe proxy here — it's also None when a stage names a
# missing agent — so gate on the markers directly.
marker = Path(cfg["log_dir"]) / "PIPELINE-COMPLETE"
all_complete = all(_pipeline_marker(cfg, st).exists() for st in stages)
if all_complete:
if not marker.exists():
log("pipeline: ALL STAGES COMPLETE")
marker.write_text("pipeline complete\n")
elif marker.exists():
log("pipeline: reopened — a stage is incomplete; clearing stale PIPELINE-COMPLETE")
marker.unlink()
return active
def watchdog_loop(cfg_path): def watchdog_loop(cfg_path):
cfg = load_config(cfg_path) cfg = load_config(cfg_path)
sig = int(cfg["watchdog"].get("signal_interval", 30)) sig = int(cfg["watchdog"].get("signal_interval", 30))
@@ -721,32 +1191,67 @@ def watchdog_loop(cfg_path):
f"signal={sig}s heavy={heavy}s, watching: {[a['name'] for a in watched(cfg)]}") f"signal={sig}s heavy={heavy}s, watching: {[a['name'] for a in watched(cfg)]}")
elapsed = heavy # force a heavy check on first tick elapsed = heavy # force a heavy check on first tick
wake_elapsed = {a["name"]: 0 for a in cfg["agents"].values() if a.get("wake")} wake_elapsed = {a["name"]: 0 for a in cfg["agents"].values() if a.get("wake")}
if log_tokens_enabled(cfg):
token_phase_begin(cfg, cur_phase(cfg).get("id"))
log(f"[log_tokens] enabled (granularity={token_granularity(cfg)}) — token-log.jsonl")
while True: while True:
cfg = load_config(cfg_path) # re-read every tick: config is authoritative, no env drift cfg = load_config(cfg_path) # re-read every tick: config is authoritative, no env drift
has_loops = bool(loop_agents(cfg)) has_loops = bool(loop_agents(cfg))
seq_done = (Path(cfg["log_dir"]) / "SEQUENCE-COMPLETE").exists() seq_done = (Path(cfg["log_dir"]) / "SEQUENCE-COMPLETE").exists()
if has_loops: if has_loops and not seq_done:
handoff_check(cfg) handoff_check(cfg)
for a in watched(cfg): gate_token_check(cfg)
# Reconcile the standalone-agent pipeline (start/advance/retire stages by their markers), and
# fold its ACTIVE stage into the stall/heal watch list so it auto-recovers like any watched agent.
active_pipe = pipeline_check(cfg)
watch_list = list(watched(cfg))
if active_pipe and active_pipe["name"] not in {x["name"] for x in watch_list}:
watch_list.append(active_pipe)
for a in watch_list:
if a["watch"] == "heal+stall": if a["watch"] == "heal+stall":
stall_check_one(cfg, a) stall_check_one(cfg, a)
else: else:
if session_alive(a["session"]): if session_alive(a["session"]):
limit_tick(cfg, a, capture_pane(a["session"], 40)) limit_tick(cfg, a, capture_pane(a["session"], 40))
if not seq_done: # ...and ADD one that appeared. wake_elapsed was built once at startup, and only ever
# shrank (below) — so a `wake` written into agents.toml while the watchdog is running was
# invisible to it, silently and forever. Both flat-file agents were given 30-min wakes on
# 2026-08-16 against a watchdog up since 2026-08-01: zero wakes fired in the 2 days since,
# and with watch="heal" (no stall-reboot) each finished turn parked them until a human
# looked. Seed a new wake with a full interval so it fires on its own schedule, not at once.
for _n, _a in cfg["agents"].items():
if _a.get("wake") and _n not in wake_elapsed:
wake_elapsed[_n] = 0
log(f"wake registered mid-run for {_n} (interval={_a['wake'].get('interval', 3600)}s)")
for name, el in list(wake_elapsed.items()): for name, el in list(wake_elapsed.items()):
interval = int(cfg["agents"][name]["wake"].get("interval", 3600)) agent = cfg["agents"].get(name)
# Config is re-read every tick, but wake_elapsed was built once at startup. If an agent's
# `wake` was removed (or the agent deleted) mid-run — e.g. an operator winding down a wake —
# skip it instead of KeyError-crashing the whole watchdog. (This bug once killed the watchdog
# silently, stalling phase advancement.)
if not agent or "wake" not in agent:
continue
# After the phase sequence completes, quiet the loop-tied wakes (e.g. the on-demand
# auditor) — but a PERSISTENT agent (the operator-facing supervisor) keeps waking, so its
# hourly supervision survives SEQUENCE-COMPLETE and can drive follow-on work (a second build).
if seq_done and agent.get("kind") != "persistent":
continue
interval = int(agent["wake"].get("interval", 3600))
if el >= interval: if el >= interval:
if wake_agent(cfg, cfg["agents"][name]): if wake_agent(cfg, agent):
wake_elapsed[name] = 0 wake_elapsed[name] = 0
# Auto-advance is checked EVERY tick (not just the heavy tick) so a completed phase advances
# within signal_interval of its `## DONE` landing, instead of idling up to heavy_interval.
advanced = phase_advance_check(cfg) if has_loops else False
if elapsed >= heavy: if elapsed >= heavy:
elapsed = 0 elapsed = 0
advanced = phase_advance_check(cfg) if has_loops else False
if not advanced: if not advanced:
for a in watched(cfg): for a in watch_list:
if seq_done and a["kind"] == "loop": if seq_done and a["kind"] == "loop":
continue continue
heal_one(cfg, a) heal_one(cfg, a)
@@ -840,6 +1345,37 @@ def cmd_phase(cfg, args):
Path(phase_idx_file(cfg)).write_text(str(int(args[1]))) Path(phase_idx_file(cfg)).write_text(str(int(args[1])))
print(f"phase idx now {cur_idx(cfg)} ({cur_phase(cfg).get('id')})") print(f"phase idx now {cur_idx(cfg)} ({cur_phase(cfg).get('id')})")
def cmd_tokens(cfg):
"""Pretty-print <log_dir>/token-log.jsonl: per-phase tokens by agent + total + duration."""
p = _token_log_path(cfg)
if not p.exists():
print(f"no token log at {p}\n(set [watchdog].log_tokens = true and run the loop)"); return
recs = []
for line in p.read_text().splitlines():
try: recs.append(json.loads(line))
except Exception: pass
if not recs:
print("token log is empty"); return
names = []
for r in recs:
for n in r.get("agents", {}):
if n not in names: names.append(n)
w = max([7] + [len(n) for n in names])
hdr = f"{'phase':<10} {'dur(s)':>8} " + " ".join(f"{n:>{w}}" for n in names) + f" {'TOTAL':>13}"
print(hdr); print("-" * len(hdr))
grand = {n: 0 for n in names}
durtot = 0.0
for r in recs:
ag = r.get("agents", {})
cells = " ".join(f"{ag.get(n,{}).get('total',0):>{w},}" for n in names)
print(f"{str(r.get('phase_id')):<10} {str(r.get('duration_s')):>8} {cells} "
f"{r.get('total',{}).get('total',0):>13,}")
for n in names: grand[n] += ag.get(n,{}).get("total",0)
durtot += r.get("duration_s") or 0
print("-" * len(hdr))
cells = " ".join(f"{grand[n]:>{w},}" for n in names)
print(f"{'TOTAL':<10} {durtot:>8.0f} {cells} {sum(grand.values()):>13,}")
def cmd_selftest(): def cmd_selftest():
"""Self-contained regression checks for the footer-UI activity detector. Needs no config.""" """Self-contained regression checks for the footer-UI activity detector. Needs no config."""
backend = { backend = {
@@ -877,7 +1413,7 @@ prompt_delivery = "arg"
process_name = "claude" process_name = "claude"
submit_key = "Enter" submit_key = "Enter"
stall_idle = 300 stall_idle = 300
active_re = "esc to interrupt|Running tool|\\\\u00b7 \\\\d+" active_re = "esc to interrupt|Running tool"
limit_re = "usage limit|limit reached|reached your .*limit" limit_re = "usage limit|limit reached|reached your .*limit"
[[agent]] [[agent]]
@@ -917,6 +1453,7 @@ def main():
elif cmd == "status": cmd_status(cfg) elif cmd == "status": cmd_status(cfg)
elif cmd == "watchdog": watchdog_loop(cfg_path) elif cmd == "watchdog": watchdog_loop(cfg_path)
elif cmd == "phase": cmd_phase(cfg, rest) elif cmd == "phase": cmd_phase(cfg, rest)
elif cmd == "tokens": cmd_tokens(cfg)
elif cmd == "logs": elif cmd == "logs":
if not rest: if not rest:
die("usage: agents.py logs <name>") die("usage: agents.py logs <name>")
+123
View File
@@ -0,0 +1,123 @@
# Example ideas — creative multi-agent topologies
A backlog of *example* projects for `examples/`, each chosen to teach a **different orchestration
topology** on the same harness. Nothing here is implemented yet — these are sketches.
Built so far:
- **`builder-adversary/`** — a **phase machine**: an ordered plan, two roles (Builder + Adversary)
handing off via `claim(`/`review(` commits. (The cc-ci pattern.)
- **`snakepit/`** — a **worker pool over a pull-queue**: identical worker "snakes" claim tasks from
a shared filesystem pit by atomic `mv`, plus planner + cleanup specialist species.
Each idea below lists: the metaphor, the topology it teaches, the star harness primitive, and what
makes it distinct from what we already have.
---
## Strong candidates
### 🐜 Anthill (stigmergy)
Ants coordinate with *no direct messaging*: they lay pheromone trails, others follow the strong
ones, and trails **evaporate** over time. Agents drop weighted "trail" files toward promising
solutions/paths; a `[[service]]` slowly decays them.
- **Teaches:** indirect coordination through a decaying shared environment (opposite of snakepit's
explicit claim).
- **Star primitive:** a background **service** as the evaporation clock; emergent routing with zero
agent-to-agent chat.
### 🍳 The Line (kitchen brigade)
A restaurant pass: prep → sauté → plating → **expo**. A ticket (order) flows station to station; the
expo bounces a bad plate back down the line. Many tickets in flight at once.
- **Teaches:** a true multi-stage **pipeline** (>2 roles) with backpressure / rework — distinct from
builder-adversary's two roles over a whole-task phase.
- **Star primitive:** chained `handoff` inboxes + per-station commit prefixes (`fire(`, `plate(`,
`expo(`).
### 🕵️ The Incident Room (blackboard)
A corkboard of pinned facts and red string. Specialist detectives (forensics, alibi, motive,
witnesses) each watch the board and pin a new deduction *only when their preconditions appear*; a
lead declares the case closed.
- **Teaches:** opportunistic, **data-driven activation** — agents fire when the shared state makes
them relevant, not on a schedule.
- **Star primitive:** a shared blackboard file + watchdog pings on board changes; no fixed order.
### ⚖️ The Senate (debate panel)
N agents argue a question from assigned stances; a moderator synthesizes; rounds repeat until
consensus or a vote.
- **Teaches:** structured **multi-round deliberation** with diverse "minds."
- **Star primitive:** the **phase machine** where each phase = one debate round, plus **per-phase
model overrides** to give each seat a genuinely different model; `on_complete` writes the verdict.
### 🏃 The Baton (relay / token ring)
Exactly one runner holds the baton (a lock file) and works; passes it on completion. Drop the baton
(crash) and the next runner picks it up.
- **Teaches:** **mutual exclusion + failover** — enforced serialization, the mirror image of the
snakepit's parallelism.
- **Star primitive:** `watch = "heal"` + the watchdog reaping a dead holder so the baton never gets
stuck.
### 🦠 The Immune System (detect → respond)
Sentinels patrol logs/metrics/files for anomalies (pathogens); on a hit they raise an antigen (alert
file); responder "macrophages" swarm that specific threat; memory cells record signatures so repeats
resolve faster.
- **Teaches:** an **event-driven monitoring/reactive** topology with escalation.
- **Star primitive:** a watcher **service** emitting alerts + reactive agents woken by inbox pings.
- **Bonus:** genuinely *useful* — a self-healing "watch my repo/CI" tool wearing a fun costume.
### 🧬 The Evolution Chamber (genetic algorithm)
A population of candidate solutions; breeder agents mutate/crossbreed; a selector culls by fitness;
generations advance until fitness plateaus.
- **Teaches:** **population-based iterative search** with a fitness gate.
- **Star primitive:** phase machine where each phase = one generation; `done_marker` trips when
fitness stops improving.
---
## Quick extras (less fleshed out)
- **🗼 Air Traffic Control** — many workers contend for *one* scarce runway (a single deploy/build
slot); a controller grants timed landing slots. Teaches centralized **scarce-resource
arbitration** (snakepit has plentiful work; here the *resource* is the bottleneck).
- **🌙 Day/Night (sleep consolidation)** — workers act by day; a "sleep" agent on a `wake` timer
consolidates the day's artifacts into long-term memory each night. Teaches **scheduled batch
consolidation** (the "memory builder / coprophagy" idea as its own example).
---
## Engine idea: unify the phase machine and per-agent pipelines (one apparatus)
The `[loop]` **phase machine** advances an ordered plan for a *fixed* set of loop agents (Builder +
Adversary) that run **every** phase in **one** worktree, detecting completion via a `## DONE`
commit-subject in a status file. A common variant, though, is a sequence of **distinct** agents —
different prompt/model, sometimes a different worktree — handed off one at a time (e.g. build → review →
e2e → integrate, each a separate agent). That's *the same shape* — an ordered sequence of stages, each
with a completion signal and the right agent(s) running — so it shouldn't need a **separate pipeline
apparatus**; it should be **the same phase machine**, generalized.
**The missing primitive is a per-stage `agent` + `dir`.** Agents already carry a `dir` (used for
per-agent token attribution). If a **phase entry** could optionally name the agent(s) that run it and
the directory they run in (plus a `done` marker as an alternative to `## DONE`), the one phase machine
covers both cases:
- **loop-style phase** (default/back-compat): no `agent` → the fixed loop agents run it; completion =
`## DONE` in the phase's status file, in the loop dir.
- **pipeline-style phase:** `agent = "fork-iroh"`, `dir = "work-fork"`, `done =
"work-fork/.ao-state/FORK-IROH-COMPLETE"` → advancing the phase stops the previous stage's agent and
starts this one in its dir; completion = the marker.
Advancing a phase then means "stop the outgoing stage's agent(s), start the incoming stage's agent(s) in
their dir" — a superset of today's "re-kickoff the same loop agents." The `p-lichen-orchestrator` project
prototyped this as a standalone `[pipeline]` block (stages = `{agent, done}`, reconciled by the watchdog
each tick, markers as source of truth); the lesson is that **that logic belongs in the phase machine**,
not beside it — fold `[pipeline]` into `[loop].phases` via optional `agent`/`dir`/`done` per phase and
delete the parallel mechanism. (Bonus: `on_complete`, per-phase model overrides, and the DONE-nudge all
already live on the phase machine, so pipeline-style phases inherit them for free.)
---
## Suggested next trio
If picking three that cover the most new ground: **The Line** (pipeline), **The Incident Room**
(blackboard), and **The Immune System** (reactive monitoring — and actually useful).
Each should follow the snakepit shape: a README with the metaphor→compute mapping, an `agents.toml`,
role prompts, and a tiny runnable task.
@@ -0,0 +1,48 @@
# Builder/Adversary example — deferred review (verify after a long segment)
The coarsest point on the **review-cadence spectrum**. Same pattern, same full original prompts as
`../builder-adversary` — only *when* the Adversary verifies changes:
| variant | the Adversary verifies… | handshakes (calculator task) |
|---|---|--:|
| `builder-adversary-lean` | per **gate** | ~12 claim/verify round-trips |
| `builder-adversary` (orig) | per **phase** | ~3 |
| **`builder-adversary-deferred`** | **once, after the whole build** | **1** |
## How it works
The Builder **self-certifies** the build phases (`wc`, then `json`) — builds to each phase's DoD, runs
its own tests until green, writes `## DONE`, and advances *without* waiting for the Adversary. The
Adversary stays out of the build. Only in the final **`review` phase** does it do **one comprehensive
cold-verification of the entire accumulated calculator** (`plans/review.md`): re-run every DoD item
from every phase from a fresh clone, plus cross-feature break-it probes, file all findings at once,
re-verify after fixes, then PASS. That single pass is the only adversary gate in the run.
## The trade-off
- **Cheapest coordination.** One handshake instead of 312 — no per-gate/per-phase round-trips, the
Builder isn't interrupted mid-build. (The benchmark showed coordination round-trips are a real
token cost; deferring to one pass minimises them.)
- **But the independent check arrives late.** Two risks the per-gate/per-phase cadences guard
against:
- **Late discovery / rework.** If the Builder built phase 2 on a wrong assumption from phase 1, an
early adversary would have caught it at gate 1; here it surfaces only at the end, after more work
was piled on the flaw — potentially a larger, costlier fix.
- **Self-certification drift.** The build phases are self-certified, so a bug the Builder
rubber-stamps survives until the final review. The comprehensive pass is the only safety net, so
it must be thorough.
- **Better at cross-feature bugs.** Because it verifies the whole system at once, it's positioned to
catch *interactions* (e.g. `--json` × every flag) that a per-gate view, looking at one item at a
time, can miss.
So `deferred` trades *early, incremental* assurance for *minimal coordination + one holistic pass*.
It suits work where features are independent and cheap to fix late; it's risky where early decisions
constrain later ones.
```bash
python3 ../../agents.py status --config agents.toml
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
```
> **Prompt base:** the full original `builder-adversary` prompts + a DEFERRED REVIEW CADENCE override
> — so comparing this to `builder-adversary`/`lean` isolates *only* the verification cadence.
@@ -0,0 +1,79 @@
# examples/builder-adversary-deferred — Adversary verifies ONCE, after a long segment of building.
#
# Same pattern + full original prompts as ../builder-adversary, but the REVIEW CADENCE is coarsest:
# • lean = the Adversary verifies per gate (finest)
# • orig = the Adversary verifies per phase (medium)
# • deferred = the Adversary verifies ONCE, comprehensively, after the whole build (coarsest)
# The Builder SELF-CERTIFIES the build phases (wc, json) to advance; the Adversary stays out until the
# final `review` phase, where it cold-verifies the ENTIRE accumulated calculator in one pass. Cheapest
# coordination, but the independent check arrives late (see README for the trade-off).
#
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
[defaults]
session_prefix = "badef-" # tmux namespace: badef-builder, badef-adv, …
log_dir = ".ao-state"
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal"
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo]
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
[[agent]]
name = "builder" # tmux session: badef-builder
kind = "loop"
role = "builder"
dir = "./work"
watch = "heal+stall"
[[agent]]
name = "adversary"
session = "badef-adv"
kind = "loop"
role = "adversary"
dir = "./work-adv"
watch = "heal+stall"
[[service]]
name = "cleanlogs"
command = "python3 ../../agent-log.py follow-all"
dir = "."
[loop]
state_file = "phase-idx"
resume_phase = true
auto_advance = true
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md"
roles_dir = "prompts"
handoff = { repo = "./work", claim_pings = "adversary", review_pings = "builder", inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"], claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
# Build phases (wc, json) are self-certified by the Builder; the final `review` phase is the single
# comprehensive Adversary gate over the whole accumulated build.
phases = [
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md" },
{ id = "review", plan = "plans/review.md", status = "STATUS-review.md" },
]
@@ -0,0 +1,32 @@
# Phase `json` — machine-readable output
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
`wc`-phase behaviour. Single source of truth for this phase.
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
model override; the Adversary stays on the default model.)
## Definition of Done
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
With stdin (no FILE), `"file"` is `null`.
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
(plus `file`). E.g. `wc.py --json -l FILE``{"lines": N, "file": "FILE"}`.
- **D3 — no regression.** Every `wc`-phase gate (D1D4 there) still passes unchanged.
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
## How the Adversary verifies (cold)
```bash
pytest -q # D4 + D3 regression
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
```
The Builder restates the exact commands, expected JSON, and commit sha in
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
@@ -0,0 +1,24 @@
# Phase `review` — comprehensive deferred verification
This phase adds **no new features**. The Builder has self-certified the build phases (`wc`, `json`)
and accumulated the whole calculator. Now the Adversary does its **one comprehensive cold-verification
of the entire build** — the first and only adversary gate in the run.
## Definition of Done
- **D1 — full cold re-verify.** From a FRESH clone, the Adversary re-runs **every DoD item from every
prior phase** (all of `wc` and all of `json`) and confirms each passes. Nothing is taken on the
Builder's word.
- **D2 — full suite green.** The complete test suite (`python -m unittest`) passes, 0 failures.
- **D3 — cross-feature break-it.** The Adversary hunts the interactions a per-gate/per-phase view
would miss: `--json` combined with every count flag, whitespace + multi-line + json together, the
error paths under json mode, stdin + json, etc. — and files any defects it finds.
- **D4 — findings cleared.** Every finding the Adversary files is fixed by the Builder and
re-verified PASS; no standing `## VETO`.
## How it works
The Adversary records its comprehensive verdict in `machine-docs/REVIEW-review.md`
(`review(all): PASS`, or findings with repro). The Builder fixes anything found, then writes
`## DONE` to `machine-docs/STATUS-review.md` **only after** the Adversary's comprehensive PASS — the
single adversary checkpoint for the whole build.
@@ -0,0 +1,43 @@
# Phase `wc` — a word-count CLI
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
## Definition of Done
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
`wc` on ASCII input.
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
## How the Adversary verifies (cold)
From a fresh clone of the work repo:
```bash
pytest -q # D4: must be all-green
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
```
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
## Out of scope (defer to a later phase or DEFERRED.md)
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
the next phase (`plans/json.md`).
@@ -0,0 +1,31 @@
You are the **Adversary** — one of two independent loops. Your job is to **DISBELIEVE the Builder**. You run as a SEPARATE process and coordinate ONLY through the git repo. Read the phase plan named in the kickoff above in full — it is the single source of truth for WHAT is being verified.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. When a gate is CLAIMED (or the watchdog pings you that one is), verify it promptly — that is top priority. When nothing is pending you may IDLE freely (sleep in chunks of **≤10 min**); you do NOT need to busy-poll to look busy — the watchdog pings you the instant the Builder claims a gate. Poll ~4 min only while actively watching a CLAIMED gate's run. Keep running independent break-it probes even when no gate is pending. Stop only when STATUS says "## DONE" and you have logged a fresh PASS for every DoD item.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** Never a single ScheduleWakeup > 600 s; to wait longer, wake, re-check, wait again.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min out, matching your ScheduleWakeup; compute with `date -u -d '+10 min' +%FT%TZ`). Idle ≥5 min with no current marker, or past the named time → the watchdog kills + reboots you; you resume cleanly from git + your REVIEW/STATUS files.
- **Compact proactively** at ≳80% context — your state is in git + REVIEW/STATUS, so compaction is lossless.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root. If you find one at the root, `git mv` it in.
- **Keep your OWN clone** (the `dir` this agent runs in). You verify from a COLD START in it. If the work repo doesn't exist yet, wait and retry on your next wake — the Builder creates it first.
- `git pull --rebase` before every edit; commit; push; **never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit that records a **verdict or finding** with `review(...)` (e.g. `review(D2): PASS` / `review(D2): FAIL — repro …`). The watchdog watches origin/main and pings the Builder the moment a `review(` commit lands — that IS the handoff signal. (The Builder's gate claims are `claim(...)`.)
- Write ONLY your files: REVIEW and the "## Adversary findings" section of BACKLOG. Everything else (code, STATUS, JOURNAL, "## Build backlog") is read-only to you.
- **INBOX side-channel.** For non-gate messages to the Builder, append `machine-docs/BUILDER-INBOX.md` and push (the watchdog edge-pings the Builder). To receive from the Builder, look for `machine-docs/ADVERSARY-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). Formal verdicts still live in REVIEW.
**ISOLATION DISCIPLINE (anti-anchoring — critical).** The Builder is REQUIRED to give you, in STATUS, the verification info you need: WHAT is claimed, HOW to verify it (the exact command/check), the EXPECTED outcome, and WHERE the inputs live. **Read STATUS for that — you need all of it.** What you must IGNORE — in STATUS, and NEVER read in JOURNAL before your verdict — is the Builder's REASONING / RATIONALISATIONS ("I think this passes because…", design narrative, dead-ends). Reading those anchors you. Form your verdict from: (a) the phase plan = SSOT, (b) the code / git history, (c) the verification info the Builder passed in STATUS, and (d) your OWN cold acceptance run that re-executes the check against the expected outcomes. Only AFTER writing your verdict may you consult JOURNAL (note in REVIEW that you did). Trust observable behaviour, the plan, and your own re-run — not the Builder's narrative.
**Each wake:**
1. Pull. Read STATUS for any "Gate: <id> CLAIMED, awaiting Adversary".
2. Verify the claim from a COLD START (fresh shell, your own clone, no cached state). Re-run the DoD acceptance check yourself; do not trust the Builder's word.
3. Actively try to BREAK it — edge cases, malformed input, the failure modes the plan names. A claim you can't break is a claim that PASSES; a claim you can break is a finding.
4. Record verdicts in REVIEW ("<id>: PASS @<ts>" + evidence, or FAIL with repro steps). File each defect as a "## Adversary findings" item; only YOU close those, after re-test. You hold veto: write "## VETO <reason>" to REVIEW to forbid DONE until cleared.
5. Push (with a `review(...)` prefix). Schedule the next wake.
REVIEW CADENCE — DEFERRED (this OVERRIDES the "verify each claimed gate per wake" rule above): you verify ONCE, comprehensively, after the whole build — not per gate or per phase.
- During the BUILD phases (before the final `review` phase): the Builder self-certifies and advances; you do NOT gate those. You may run early break-it probes, but the authoritative check is deferred — don't write per-gate verdicts.
- In the `review` phase: do ONE comprehensive cold-verification of the ENTIRE calculator from a fresh clone — re-run EVERY DoD item from EVERY prior phase, and hunt cross-feature / integration breaks (interactions between features, not just isolated gates). File all findings together; re-verify after the Builder's fixes; PASS only when the whole system holds. This single comprehensive pass replaces per-gate review.
Begin: read the phase plan, then enter the self-paced loop (start by cloning the work repo into your `dir` if it exists yet).
@@ -0,0 +1,35 @@
You are the **Builder** — one of two independent loops working on this project. Your job is to build what the phase plan specifies, autonomously, over many wake cycles. You run as a SEPARATE process from the Adversary and coordinate with it ONLY through the git repo.
Single source of truth: the phase plan named in the kickoff above. Read it in full now, then begin.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. Each iteration = one unit of work. Pace yourself:
- A long task in flight (build / test suite / e2e) → **poll every ~5 min**, never one big sleep matching the expected runtime (catch a failure at minute 4 of a 25-min run, not at minute 25).
- Parked at a CLAIMED gate with no other unblocked work → the watchdog pings you the instant the Adversary writes a verdict or an inbox message, so you may wait; keep a fallback self-poll ~24 min in case a ping is missed.
- Genuinely idle → sleep in chunks of **≤10 min**. Prefer keeping an unblocked backlog item in hand so you rarely just wait.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** To wait longer, wake at 10 min, re-check, wait again. Never a single ScheduleWakeup > 600 s.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` — the time you will resume (≤10 min out, matching your ScheduleWakeup). Compute it from the clock (`date -u -d '+10 min' +%FT%TZ`). If the watchdog sees you idle ≥5 min with no current marker as your last line, OR idle past the time it names, it kills + reboots you — you resume cleanly from git + your STATUS/REVIEW files.
- **Compact proactively.** If context usage climbs high (≳80%), run `/compact` before continuing — your loop state lives in git + the phase STATUS/REVIEW, so compaction is lossless and prevents wedging at the context limit.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root — phase-namespaced STATUS/BACKLOG/REVIEW/JOURNAL, plus DECISIONS.md and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. Create `machine-docs/` if missing; if you find such a file at the root, `git mv` it in.
- `git pull --rebase` before every edit; make the smallest change; commit; push. **Never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit with its conventional type. CRITICALLY: prefix a commit that **claims a gate** with `claim(...)` (e.g. `claim(D2): tests green`). The watchdog watches origin/main and pings the Adversary the moment a `claim(` commit lands — that IS the handoff signal. Keep using the other types too (`feat/fix/status/journal/decisions/chore/inbox(...)`), but `claim(` is what triggers verification.
- **CLEAN TREE BEFORE CLAIM.** Run `git status` before you claim — the working tree MUST be clean (everything committed AND pushed). The Adversary cold-verifies from a fresh clone, so any un-pushed change that only exists on your host is a guaranteed verify mismatch. Push first, then claim.
- **ARTIFACT-LAYER ISOLATION — the one rule that makes verification work.** STATUS MUST give the Adversary everything it needs to verify your claim: **WHAT** is claimed (gate id, DoD items), **HOW** to verify it (the exact command/check it can re-run from its own clone), the **EXPECTED** outcome (outputs, hashes, exit codes), and **WHERE** the inputs live (commit shas, paths). STATUS MUST NOT contain rationalisations — "I think this passes because…", design narrative, dead-ends. Those go in JOURNAL, which the Adversary is instructed NOT to read before its verdict (anti-anchoring). The line: **WHAT + HOW + EXPECTED + WHERE = STATUS; WHY = JOURNAL.** DECISIONS.md is for SETTLED design decisions, not in-the-moment reasoning.
- **At each gate:** set "Gate: <id> CLAIMED, awaiting Adversary" in STATUS and work other unblocked items; do NOT advance past the gate until REVIEW shows its PASS.
- **INBOX side-channel.** For non-gate messages to the Adversary (a heads-up, "starting a long run, please cold-verify X meanwhile"), append `machine-docs/ADVERSARY-INBOX.md` and push — the watchdog edge-pings the Adversary. To receive from the Adversary, look for `machine-docs/BUILDER-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). The inbox is a side-channel; formal CLAIMS still live in STATUS.
- Write ONLY your files: source/config, STATUS, JOURNAL, DECISIONS, and the "## Build backlog" section of BACKLOG. Treat REVIEW and "## Adversary findings" as read-only — the Adversary owns them.
**Overriding rules:**
- "Done" is defined ONLY by the plan's DoD, Adversary-verified. No self-certifying. Write "## DONE" to STATUS only when REVIEW shows a fresh PASS for every DoD item and there is no standing "## VETO".
- Verify every change against real behaviour; paste the command + its output into JOURNAL. No "should work."
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
- 3rd identical failure → stop, record the dead-end in DECISIONS.md, change approach or mark blocked.
REVIEW CADENCE — DEFERRED (this OVERRIDES the per-phase "Adversary-verified / no self-certifying" rule above, for build phases only): the Adversary verifies in ONE comprehensive pass at the END, not per gate or per phase.
- BUILD phases (every phase before the final `review` phase): SELF-CERTIFY. Build to the phase DoD, run your own tests until green, then write "## DONE" to advance — do NOT claim or wait for the Adversary on a build phase. Accumulate the whole build.
- The final `review` phase: do not add features. The Adversary now cold-verifies the ENTIRE accumulated build at once; address every finding it files, then write "## DONE" only after its comprehensive PASS. (Here the normal Adversary-verified rule applies.)
Begin: read the phase plan, then enter the self-paced loop.
@@ -0,0 +1,8 @@
*** PHASE {phase_id} ***
SINGLE SOURCE OF TRUTH for this phase: {plan} — read it in full now. It defines this phase's mission and its Definition of Done (DoD).
Track loop state in PHASE-NAMESPACED files UNDER machine-docs/ in your clone (create the dir if missing): machine-docs/{status}, machine-docs/BACKLOG-{phase_id}.md, machine-docs/REVIEW-{phase_id}.md, machine-docs/JOURNAL-{phase_id}.md. machine-docs/DECISIONS.md is shared (append-only).
FILE-LOCATION RULE (mandatory): ALL coordination / loop-state files live in machine-docs/, NEVER the repo root — that includes STATUS/BACKLOG/REVIEW/JOURNAL (phase-namespaced), DECISIONS.md, and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. If you ever find one at the root, git mv it into machine-docs/.
"Done" for this phase = the Builder writes "## DONE" to machine-docs/{status} ONLY after EVERY DoD item is Adversary-verified with a fresh PASS in machine-docs/REVIEW-{phase_id}.md (handshake below).
Wherever the standing role below says "the plan" / "STATUS" / "REVIEW", substitute {plan} and these machine-docs/ phase-namespaced files.
=== standing role & rules ===
+32
View File
@@ -0,0 +1,32 @@
# Builder/Adversary example — context-lean + full per-gate review
The [`builder-adversary-stateless`](../builder-adversary-stateless/) variant added **context
hygiene** (compact at each checkpoint, read diffs not trees, lean loads) and, in benchmarking,
happened to also do *fewer* review rounds — so its token saving was partly leaner context and partly
*less scrutiny*. This variant **isolates the two**: it keeps all the context hygiene but **requires
full per-gate review granularity** — one `claim(<gate>)` per gate and one independent Adversary
verdict per gate, no batching.
The point: if this variant keeps most of the token saving *despite* doing as many (or more) review
passes than the original, then the saving is real efficiency (lower carried/reloaded context), not a
reduction in adversarial scrutiny.
So vs the others:
| variant | context hygiene | review granularity |
|---|:--:|---|
| builder-adversary | no | as the agents choose |
| builder-adversary-min | no | as the agents choose |
| builder-adversary-stateless | yes | as the agents choose (tended to batch → fewer rounds) |
| **builder-adversary-lean** | **yes** | **per-gate, enforced (no batching)** |
Everything else — pattern, AI-as-adversary cold verification, the `claim(`/`review(` handoff,
`machine-docs/` coordination — is identical. The `agent-orchestrator-benchmark` repo runs it
head-to-head with the others on the same multi-phase task.
```bash
python3 ../../agents.py status --config agents.toml
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
```
> **Prompt base:** these prompts are the **full original** `builder-adversary` prompts plus the additions above — NOT the minimal ones — so that comparing this variant to `builder-adversary` isolates its specific change (context hygiene / review granularity) without the minimal-prompt testing-pressure drop.
@@ -0,0 +1,92 @@
# examples/builder-adversary-lean — context hygiene + ENFORCED full per-gate review.
#
# Like builder-adversary-stateless (CONTEXT HYGIENE: compact at every checkpoint, read diffs not
# trees, spill bulk to files, adversary loads only {plan, STATUS, diff}) BUT the prompts also require
# per-gate review granularity — one claim per gate, one independent Adversary verdict per gate, no
# batching. This isolates "leaner context" from "fewer review passes". Loop agents not resumed →
# fresh session per phase. See README.md.
#
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
[defaults]
session_prefix = "blean-" # REQUIRED — sessions: blean-builder, blean-adv, …
log_dir = ".ao-state"
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal"
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo]
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
[[agent]]
name = "builder" # tmux session: blean-builder
kind = "loop"
role = "builder"
dir = "./work"
watch = "heal+stall"
[[agent]]
name = "adversary"
session = "blean-adv"
kind = "loop"
role = "adversary"
dir = "./work-adv"
watch = "heal+stall"
[[agent]]
name = "orchestrator" # tmux session: blean-orchestrator
kind = "persistent"
model = "claude-opus-4-8"
resume = true
watch = "heal"
prompt = "You supervise this Builder/Adversary project. On startup: read machine-docs/ for the current phase's STATUS/REVIEW, confirm both loops + the watchdog are up, report the phase and any open findings/VETO. Then stay available; intervene only if the pair is stuck."
[[agent]]
name = "reporter" # tmux session: blean-reporter
kind = "task"
model = "claude-opus-4-8"
watch = "none"
enabled = false
prompt = "The phase sequence is complete. Read machine-docs/ across all phases, write a short machine-docs/REPORT.md (what was built, each gate's final verdict, deferred items), then go idle."
[[service]]
name = "cleanlogs"
command = "python3 ../../agent-log.py follow-all"
dir = "."
[loop]
state_file = "phase-idx"
resume_phase = true
auto_advance = true
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md"
roles_dir = "prompts"
handoff = { repo = "./work", claim_pings = "adversary", review_pings = "builder", inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"], claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
on_complete = { trigger_file = ".run-report-on-complete", run = "reporter" }
phases = [
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md", models = { builder = "claude-opus-4-8" } },
]
@@ -0,0 +1,2 @@
# Coordination / loop-state files live here at runtime (phase-namespaced STATUS / REVIEW / BACKLOG /
# JOURNAL, plus the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels). The loop pair populates it.
@@ -0,0 +1,32 @@
# Phase `json` — machine-readable output
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
`wc`-phase behaviour. Single source of truth for this phase.
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
model override; the Adversary stays on the default model.)
## Definition of Done
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
With stdin (no FILE), `"file"` is `null`.
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
(plus `file`). E.g. `wc.py --json -l FILE``{"lines": N, "file": "FILE"}`.
- **D3 — no regression.** Every `wc`-phase gate (D1D4 there) still passes unchanged.
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
## How the Adversary verifies (cold)
```bash
pytest -q # D4 + D3 regression
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
```
The Builder restates the exact commands, expected JSON, and commit sha in
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
@@ -0,0 +1,43 @@
# Phase `wc` — a word-count CLI
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
## Definition of Done
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
`wc` on ASCII input.
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
## How the Adversary verifies (cold)
From a fresh clone of the work repo:
```bash
pytest -q # D4: must be all-green
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
```
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
## Out of scope (defer to a later phase or DEFERRED.md)
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
the next phase (`plans/json.md`).
@@ -0,0 +1,34 @@
You are the **Adversary** — one of two independent loops. Your job is to **DISBELIEVE the Builder**. You run as a SEPARATE process and coordinate ONLY through the git repo. Read the phase plan named in the kickoff above in full — it is the single source of truth for WHAT is being verified.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. When a gate is CLAIMED (or the watchdog pings you that one is), verify it promptly — that is top priority. When nothing is pending you may IDLE freely (sleep in chunks of **≤10 min**); you do NOT need to busy-poll to look busy — the watchdog pings you the instant the Builder claims a gate. Poll ~4 min only while actively watching a CLAIMED gate's run. Keep running independent break-it probes even when no gate is pending. Stop only when STATUS says "## DONE" and you have logged a fresh PASS for every DoD item.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** Never a single ScheduleWakeup > 600 s; to wait longer, wake, re-check, wait again.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min out, matching your ScheduleWakeup; compute with `date -u -d '+10 min' +%FT%TZ`). Idle ≥5 min with no current marker, or past the named time → the watchdog kills + reboots you; you resume cleanly from git + your REVIEW/STATUS files.
- **Compact proactively** at ≳80% context — your state is in git + REVIEW/STATUS, so compaction is lossless.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root. If you find one at the root, `git mv` it in.
- **Keep your OWN clone** (the `dir` this agent runs in). You verify from a COLD START in it. If the work repo doesn't exist yet, wait and retry on your next wake — the Builder creates it first.
- `git pull --rebase` before every edit; commit; push; **never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit that records a **verdict or finding** with `review(...)` (e.g. `review(D2): PASS` / `review(D2): FAIL — repro …`). The watchdog watches origin/main and pings the Builder the moment a `review(` commit lands — that IS the handoff signal. (The Builder's gate claims are `claim(...)`.)
- Write ONLY your files: REVIEW and the "## Adversary findings" section of BACKLOG. Everything else (code, STATUS, JOURNAL, "## Build backlog") is read-only to you.
- **INBOX side-channel.** For non-gate messages to the Builder, append `machine-docs/BUILDER-INBOX.md` and push (the watchdog edge-pings the Builder). To receive from the Builder, look for `machine-docs/ADVERSARY-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). Formal verdicts still live in REVIEW.
**ISOLATION DISCIPLINE (anti-anchoring — critical).** The Builder is REQUIRED to give you, in STATUS, the verification info you need: WHAT is claimed, HOW to verify it (the exact command/check), the EXPECTED outcome, and WHERE the inputs live. **Read STATUS for that — you need all of it.** What you must IGNORE — in STATUS, and NEVER read in JOURNAL before your verdict — is the Builder's REASONING / RATIONALISATIONS ("I think this passes because…", design narrative, dead-ends). Reading those anchors you. Form your verdict from: (a) the phase plan = SSOT, (b) the code / git history, (c) the verification info the Builder passed in STATUS, and (d) your OWN cold acceptance run that re-executes the check against the expected outcomes. Only AFTER writing your verdict may you consult JOURNAL (note in REVIEW that you did). Trust observable behaviour, the plan, and your own re-run — not the Builder's narrative.
**Each wake:**
1. Pull. Read STATUS for any "Gate: <id> CLAIMED, awaiting Adversary".
2. Verify the claim from a COLD START (fresh shell, your own clone, no cached state). Re-run the DoD acceptance check yourself; do not trust the Builder's word.
3. Actively try to BREAK it — edge cases, malformed input, the failure modes the plan names. A claim you can't break is a claim that PASSES; a claim you can break is a finding.
4. Record verdicts in REVIEW ("<id>: PASS @<ts>" + evidence, or FAIL with repro steps). File each defect as a "## Adversary findings" item; only YOU close those, after re-test. You hold veto: write "## VETO <reason>" to REVIEW to forbid DONE until cleared.
5. Push (with a `review(...)` prefix). Schedule the next wake.
CONTEXT HYGIENE — your durable state is REVIEW + git, so the conversation is disposable scratch; keep it small so you don't pay to reload it every turn:
- Per gate, load only what you need to judge it: the plan, the Builder's STATUS, and the diff since the last verified sha (`git diff <sha>..HEAD`). Don't re-read the whole repo or earlier gates.
- After writing each verdict (a durable checkpoint), run `/compact` — lossless here; you reload from REVIEW + git.
- Spill bulk to files: pipe long verification/test output to a file and read back only the part you need.
REVIEW GRANULARITY (required): verify every claimed gate in its OWN independent cold pass and write a separate `review(<gate-id>): PASS|FAIL` per gate — never batch verdicts, never skip a gate. The CONTEXT HYGIENE above governs only HOW you load context (compact, diffs), NOT how much you scrutinise: keep full per-gate rigor and your break-it probes.
Begin: read the phase plan, then enter the self-paced loop (start by cloning the work repo into your `dir` if it exists yet).
@@ -0,0 +1,39 @@
You are the **Builder** — one of two independent loops working on this project. Your job is to build what the phase plan specifies, autonomously, over many wake cycles. You run as a SEPARATE process from the Adversary and coordinate with it ONLY through the git repo.
Single source of truth: the phase plan named in the kickoff above. Read it in full now, then begin.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. Each iteration = one unit of work. Pace yourself:
- A long task in flight (build / test suite / e2e) → **poll every ~5 min**, never one big sleep matching the expected runtime (catch a failure at minute 4 of a 25-min run, not at minute 25).
- Parked at a CLAIMED gate with no other unblocked work → the watchdog pings you the instant the Adversary writes a verdict or an inbox message, so you may wait; keep a fallback self-poll ~24 min in case a ping is missed.
- Genuinely idle → sleep in chunks of **≤10 min**. Prefer keeping an unblocked backlog item in hand so you rarely just wait.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** To wait longer, wake at 10 min, re-check, wait again. Never a single ScheduleWakeup > 600 s.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` — the time you will resume (≤10 min out, matching your ScheduleWakeup). Compute it from the clock (`date -u -d '+10 min' +%FT%TZ`). If the watchdog sees you idle ≥5 min with no current marker as your last line, OR idle past the time it names, it kills + reboots you — you resume cleanly from git + your STATUS/REVIEW files.
- **Compact proactively.** If context usage climbs high (≳80%), run `/compact` before continuing — your loop state lives in git + the phase STATUS/REVIEW, so compaction is lossless and prevents wedging at the context limit.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root — phase-namespaced STATUS/BACKLOG/REVIEW/JOURNAL, plus DECISIONS.md and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. Create `machine-docs/` if missing; if you find such a file at the root, `git mv` it in.
- `git pull --rebase` before every edit; make the smallest change; commit; push. **Never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit with its conventional type. CRITICALLY: prefix a commit that **claims a gate** with `claim(...)` (e.g. `claim(D2): tests green`). The watchdog watches origin/main and pings the Adversary the moment a `claim(` commit lands — that IS the handoff signal. Keep using the other types too (`feat/fix/status/journal/decisions/chore/inbox(...)`), but `claim(` is what triggers verification.
- **CLEAN TREE BEFORE CLAIM.** Run `git status` before you claim — the working tree MUST be clean (everything committed AND pushed). The Adversary cold-verifies from a fresh clone, so any un-pushed change that only exists on your host is a guaranteed verify mismatch. Push first, then claim.
- **ARTIFACT-LAYER ISOLATION — the one rule that makes verification work.** STATUS MUST give the Adversary everything it needs to verify your claim: **WHAT** is claimed (gate id, DoD items), **HOW** to verify it (the exact command/check it can re-run from its own clone), the **EXPECTED** outcome (outputs, hashes, exit codes), and **WHERE** the inputs live (commit shas, paths). STATUS MUST NOT contain rationalisations — "I think this passes because…", design narrative, dead-ends. Those go in JOURNAL, which the Adversary is instructed NOT to read before its verdict (anti-anchoring). The line: **WHAT + HOW + EXPECTED + WHERE = STATUS; WHY = JOURNAL.** DECISIONS.md is for SETTLED design decisions, not in-the-moment reasoning.
- **At each gate:** set "Gate: <id> CLAIMED, awaiting Adversary" in STATUS and work other unblocked items; do NOT advance past the gate until REVIEW shows its PASS.
- **INBOX side-channel.** For non-gate messages to the Adversary (a heads-up, "starting a long run, please cold-verify X meanwhile"), append `machine-docs/ADVERSARY-INBOX.md` and push — the watchdog edge-pings the Adversary. To receive from the Adversary, look for `machine-docs/BUILDER-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). The inbox is a side-channel; formal CLAIMS still live in STATUS.
- Write ONLY your files: source/config, STATUS, JOURNAL, DECISIONS, and the "## Build backlog" section of BACKLOG. Treat REVIEW and "## Adversary findings" as read-only — the Adversary owns them.
**Overriding rules:**
- "Done" is defined ONLY by the plan's DoD, Adversary-verified. No self-certifying. Write "## DONE" to STATUS only when REVIEW shows a fresh PASS for every DoD item and there is no standing "## VETO".
- Verify every change against real behaviour; paste the command + its output into JOURNAL. No "should work."
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
- 3rd identical failure → stop, record the dead-end in DECISIONS.md, change approach or mark blocked.
CONTEXT HYGIENE — your durable state is git + STATUS/JOURNAL, so the conversation is disposable scratch; keep it small so you don't pay to reload it every turn:
- After each gate is committed+pushed (a durable checkpoint), run `/compact` — it's lossless here, you reload what you need from git + STATUS.
- Read DIFFS, not trees: `git diff <last-sha>..HEAD` and only the files you're touching; don't re-read the whole repo.
- Spill bulk to files: pipe long build/test output to a file and read back only the part you need — don't dump it into the conversation.
- On a fresh wake, reconstruct from the plan + STATUS + a diff; don't rebuild context by re-reading everything.
REVIEW GRANULARITY (required): claim each DoD gate INDIVIDUALLY — one `claim(<gate-id>)` per gate, the moment that gate is met. Do NOT batch several gates into one claim. Granular claims keep the Adversary's verification thorough (one independent cold pass per gate).
Begin: read the phase plan, then enter the self-paced loop.
@@ -0,0 +1,8 @@
*** PHASE {phase_id} ***
SINGLE SOURCE OF TRUTH for this phase: {plan} — read it in full now. It defines this phase's mission and its Definition of Done (DoD).
Track loop state in PHASE-NAMESPACED files UNDER machine-docs/ in your clone (create the dir if missing): machine-docs/{status}, machine-docs/BACKLOG-{phase_id}.md, machine-docs/REVIEW-{phase_id}.md, machine-docs/JOURNAL-{phase_id}.md. machine-docs/DECISIONS.md is shared (append-only).
FILE-LOCATION RULE (mandatory): ALL coordination / loop-state files live in machine-docs/, NEVER the repo root — that includes STATUS/BACKLOG/REVIEW/JOURNAL (phase-namespaced), DECISIONS.md, and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. If you ever find one at the root, git mv it into machine-docs/.
"Done" for this phase = the Builder writes "## DONE" to machine-docs/{status} ONLY after EVERY DoD item is Adversary-verified with a fresh PASS in machine-docs/REVIEW-{phase_id}.md (handshake below).
Wherever the standing role below says "the plan" / "STATUS" / "REVIEW", substitute {plan} and these machine-docs/ phase-namespaced files.
=== standing role & rules ===
+26
View File
@@ -0,0 +1,26 @@
# Builder/Adversary example — minimal-prompt variant
Same as [`../builder-adversary`](../builder-adversary/) in every way that matters — Builder +
Adversary loop pair, phase machine, `claim(`/`review(` git handoff, `machine-docs/` coordination,
cold verification — but the **role + kickoff prompts are compressed to minimal tokens**, keeping
every load-bearing rule (the commit-prefix handoff, the `machine-docs/` file rule, the
`WHAT+HOW+EXPECTED+WHERE=STATUS / WHY=JOURNAL` anti-anchoring contract, and the `WAITING-UNTIL`
liveness protocol).
Why: the prompts are sent to the agents on every kickoff, so trimming them trims tokens. Config and
plans are unchanged from the original (they aren't part of the prompt). See the original's README for
the full explanation of the pattern, how to run it, and the work-repo isolation model — the commands
are identical, just `--config` this directory's `agents.toml`.
```bash
python3 ../../agents.py status --config agents.toml
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
```
## How small?
`prompts/builder.md` and `prompts/adversary.md` here are roughly **half to a third** the size of the
originals, with the same rules stated tersely. The separate **`agent-orchestrator-benchmark`** repo
runs a head-to-head: the same task built independently by this variant and the original (both on
Sonnet), with token counts for each — confirming the minimal prompts still get the job done and
quantifying the savings.
@@ -0,0 +1,91 @@
# examples/builder-adversary-min — minimal-prompt variant of ../builder-adversary.
#
# Same topology and behaviour as builder-adversary (Builder + Adversary loop pair, phase machine,
# claim()/review() git handoff, machine-docs/ coordination). The ONLY difference is that the role +
# kickoff prompts in prompts/ are compressed to minimal tokens while keeping every load-bearing rule.
# Config/comments are unchanged — they aren't sent to the agents, so they don't affect token cost.
#
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
[defaults]
session_prefix = "bamin-" # REQUIRED — sessions: bamin-builder, bamin-adv, …
log_dir = ".ao-state"
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal"
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo]
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
[[agent]]
name = "builder" # tmux session: bamin-builder
kind = "loop"
role = "builder"
dir = "./work"
watch = "heal+stall"
[[agent]]
name = "adversary"
session = "bamin-adv"
kind = "loop"
role = "adversary"
dir = "./work-adv"
watch = "heal+stall"
[[agent]]
name = "orchestrator" # tmux session: bamin-orchestrator
kind = "persistent"
model = "claude-opus-4-8"
resume = true
watch = "heal"
prompt = "You supervise this Builder/Adversary project. On startup: read machine-docs/ for the current phase's STATUS/REVIEW, confirm both loops + the watchdog are up, report the phase and any open findings/VETO. Then stay available; intervene only if the pair is stuck."
[[agent]]
name = "reporter" # tmux session: bamin-reporter
kind = "task"
model = "claude-opus-4-8"
watch = "none"
enabled = false
prompt = "The phase sequence is complete. Read machine-docs/ across all phases, write a short machine-docs/REPORT.md (what was built, each gate's final verdict, deferred items), then go idle."
[[service]]
name = "cleanlogs"
command = "python3 ../../agent-log.py follow-all"
dir = "."
[loop]
state_file = "phase-idx"
resume_phase = true
auto_advance = true
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md"
roles_dir = "prompts"
handoff = { repo = "./work", claim_pings = "adversary", review_pings = "builder", inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"], claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
on_complete = { trigger_file = ".run-report-on-complete", run = "reporter" }
phases = [
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md", models = { builder = "claude-opus-4-8" } },
]
@@ -0,0 +1,2 @@
# Coordination / loop-state files live here at runtime (phase-namespaced STATUS / REVIEW / BACKLOG /
# JOURNAL, plus the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels). The loop pair populates it.
@@ -0,0 +1,32 @@
# Phase `json` — machine-readable output
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
`wc`-phase behaviour. Single source of truth for this phase.
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
model override; the Adversary stays on the default model.)
## Definition of Done
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
With stdin (no FILE), `"file"` is `null`.
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
(plus `file`). E.g. `wc.py --json -l FILE``{"lines": N, "file": "FILE"}`.
- **D3 — no regression.** Every `wc`-phase gate (D1D4 there) still passes unchanged.
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
## How the Adversary verifies (cold)
```bash
pytest -q # D4 + D3 regression
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
```
The Builder restates the exact commands, expected JSON, and commit sha in
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
@@ -0,0 +1,43 @@
# Phase `wc` — a word-count CLI
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
## Definition of Done
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
`wc` on ASCII input.
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
## How the Adversary verifies (cold)
From a fresh clone of the work repo:
```bash
pytest -q # D4: must be all-green
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
```
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
## Out of scope (defer to a later phase or DEFERRED.md)
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
the next phase (`plans/json.md`).
@@ -0,0 +1,9 @@
You are the **Adversary**, one of two independent loops: **DISBELIEVE the Builder**. Coordinate ONLY through git. The phase plan is the SSOT for what to verify.
Loop: run `/loop` (no interval). Verify a CLAIMED gate promptly (the watchdog pings you when the Builder claims one); idle otherwise. Cap waits at 10 min; before going idle your LAST line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>`. Compact at ~80%.
Verify cold from your OWN clone: re-run the plan's DoD check yourself and try to break it (edge cases, bad input) — don't trust the Builder's word. From STATUS take only what you need to re-run (command, expected result, shas); ignore its reasoning and don't read JOURNAL until after your verdict (it anchors you). Judge from the plan, the code, and your own run.
Git: `pull --rebase`, commit, push; never `--force`. Prefix verdicts `review(<id>): PASS|FAIL …` — pings the Builder. Write only REVIEW.md (+ your findings). Record "<id>: PASS @<ts>" + evidence, or FAIL + repro steps. You hold veto: write "## VETO <reason>".
Begin: read the plan, then enter the loop (clone the work repo into your dir if it exists yet).
@@ -0,0 +1,11 @@
You are the **Builder**, one of two independent loops; coordinate ONLY through git. Read the phase plan (the SSOT) and build to its DoD.
Loop: run `/loop` (no interval), one unit of work per wake. Cap every wait at 10 min; before going idle your LAST output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min out) or the watchdog reboots you. Compact at ~80% context.
Git: `pull --rebase`, smallest change, commit, push; never `--force`. Prefix a gate claim `claim(<id>): …` — the watchdog pings the Adversary on it; use `feat/fix/status/…` otherwise. Before you claim, the tree MUST be clean (committed AND pushed): the Adversary cold-verifies from a fresh clone.
STATUS (in machine-docs/) must give the Adversary: WHAT is claimed (gate id + DoD items), HOW to verify (exact command), the EXPECTED result, WHERE (commit shas/paths). Reasoning goes in JOURNAL, NOT STATUS — the Adversary won't read JOURNAL before judging. Write only your files (code, STATUS, JOURNAL, build backlog); REVIEW is the Adversary's.
Done: write "## DONE" only when REVIEW shows a fresh PASS for every DoD item and there's no "## VETO". Never weaken/skip/delete a test; verify for real, no "should work".
Begin: read the plan, then enter the loop.
@@ -0,0 +1,6 @@
*** PHASE {phase_id} ***
Plan (this phase's single source of truth): {plan} — read it fully now; it defines the mission and the Definition of Done (DoD).
Loop state goes under machine-docs/ (create if missing), phase-namespaced: {status}, REVIEW-{phase_id}.md, JOURNAL-{phase_id}.md, BACKLOG-{phase_id}.md. Never at the repo root.
Done = the Builder writes "## DONE" to machine-docs/{status} ONLY after every DoD item has a fresh Adversary PASS in machine-docs/REVIEW-{phase_id}.md.
=== role ===
@@ -0,0 +1,51 @@
# Builder/Adversary example — context-lean ("stateless") variant
Same pattern, same **AI-as-adversary** verification, same gates as
[`../builder-adversary`](../builder-adversary/) and
[`../builder-adversary-min`](../builder-adversary-min/) — but the role prompts add a **context
hygiene** discipline so each loop carries and reloads as little conversation as possible. Nothing
about *what* the agents do or *how* they verify changes; only how much context they drag from turn to
turn.
## Why
In a long autonomous loop the dominant token cost is **cache-read**: every turn re-sends the
conversation so far (the unchanged prefix is billed as cache-read, ~10% of input price, but it's
billed *every turn*). So cost ≈ context length × turns. The role prose is a rounding error against
that. The win is keeping the conversation short and not carrying it where it isn't needed.
This protocol already makes that safe: the **durable state is on disk** (git + the plan +
STATUS/REVIEW/JOURNAL), so the conversation is disposable scratch. These prompts exploit that:
- **Compact at every checkpoint.** After each gate is committed (Builder) or each verdict is written
(Adversary), run `/compact` — lossless here, because the agent reloads from git + STATUS/REVIEW.
- **Read diffs, not trees.** `git diff <last-sha>..HEAD` and only the touched files — never re-read
the whole repo.
- **Spill bulk to files.** Long build/test/verification output goes to a file; read back only the
slice you need, instead of dumping it into context.
- **Adversary loads only {plan, STATUS, diff}** per gate — full cold AI judgment, tiny footprint.
## Config note
Run the loop agents **non-resumed** (the default in this `agents.toml` — loop agents don't set
`resume = true`), so each time the watchdog restarts a loop (notably at every phase advance) it
starts a *fresh* session rather than carrying the prior phase's whole conversation forward. The
in-phase shrinking is done by `/compact` per the prompts above.
> A natural future engine lever (not yet implemented) would be a watchdog policy that **recycles a
> loop's session after each checkpoint commit** (claim/review), giving fresh context *per gate*
> rather than per phase — the same idea, enforced by the harness instead of the prompt.
## Compared
The **`agent-orchestrator-benchmark`** repo runs this variant head-to-head against
`builder-adversary` and `builder-adversary-min` on the same multi-phase task (all on Sonnet),
reporting tokens per loop — to quantify how much the context discipline saves while keeping identical
gate outcomes.
```bash
python3 ../../agents.py status --config agents.toml
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
```
> **Prompt base:** these prompts are the **full original** `builder-adversary` prompts plus the additions above — NOT the minimal ones — so that comparing this variant to `builder-adversary` isolates its specific change (context hygiene / review granularity) without the minimal-prompt testing-pressure drop.
@@ -0,0 +1,92 @@
# examples/builder-adversary-stateless — context-lean variant of ../builder-adversary (FULL original prompts + context hygiene).
#
# Same topology, behaviour, and AI-as-adversary verification as builder-adversary. The prompts add a
# CONTEXT HYGIENE discipline (compact at every checkpoint, read diffs not trees, spill bulk to files,
# adversary loads only {plan, STATUS, diff}) so each loop carries/reloads minimal conversation —
# cache-read is the dominant cost in a long loop. Loop agents are NOT resumed (default below), so the
# watchdog gives a fresh session per phase. See README.md.
#
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
[defaults]
session_prefix = "bastl-" # REQUIRED — sessions: bastl-builder, bastl-adv, …
log_dir = ".ao-state"
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal"
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo]
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
[[agent]]
name = "builder" # tmux session: bastl-builder
kind = "loop"
role = "builder"
dir = "./work"
watch = "heal+stall"
[[agent]]
name = "adversary"
session = "bastl-adv"
kind = "loop"
role = "adversary"
dir = "./work-adv"
watch = "heal+stall"
[[agent]]
name = "orchestrator" # tmux session: bastl-orchestrator
kind = "persistent"
model = "claude-opus-4-8"
resume = true
watch = "heal"
prompt = "You supervise this Builder/Adversary project. On startup: read machine-docs/ for the current phase's STATUS/REVIEW, confirm both loops + the watchdog are up, report the phase and any open findings/VETO. Then stay available; intervene only if the pair is stuck."
[[agent]]
name = "reporter" # tmux session: bastl-reporter
kind = "task"
model = "claude-opus-4-8"
watch = "none"
enabled = false
prompt = "The phase sequence is complete. Read machine-docs/ across all phases, write a short machine-docs/REPORT.md (what was built, each gate's final verdict, deferred items), then go idle."
[[service]]
name = "cleanlogs"
command = "python3 ../../agent-log.py follow-all"
dir = "."
[loop]
state_file = "phase-idx"
resume_phase = true
auto_advance = true
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md"
roles_dir = "prompts"
handoff = { repo = "./work", claim_pings = "adversary", review_pings = "builder", inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"], claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
on_complete = { trigger_file = ".run-report-on-complete", run = "reporter" }
phases = [
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md", models = { builder = "claude-opus-4-8" } },
]
@@ -0,0 +1,2 @@
# Coordination / loop-state files live here at runtime (phase-namespaced STATUS / REVIEW / BACKLOG /
# JOURNAL, plus the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels). The loop pair populates it.
@@ -0,0 +1,32 @@
# Phase `json` — machine-readable output
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
`wc`-phase behaviour. Single source of truth for this phase.
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
model override; the Adversary stays on the default model.)
## Definition of Done
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
With stdin (no FILE), `"file"` is `null`.
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
(plus `file`). E.g. `wc.py --json -l FILE``{"lines": N, "file": "FILE"}`.
- **D3 — no regression.** Every `wc`-phase gate (D1D4 there) still passes unchanged.
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
## How the Adversary verifies (cold)
```bash
pytest -q # D4 + D3 regression
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
```
The Builder restates the exact commands, expected JSON, and commit sha in
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
@@ -0,0 +1,43 @@
# Phase `wc` — a word-count CLI
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
## Definition of Done
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
`wc` on ASCII input.
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
## How the Adversary verifies (cold)
From a fresh clone of the work repo:
```bash
pytest -q # D4: must be all-green
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
```
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
## Out of scope (defer to a later phase or DEFERRED.md)
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
the next phase (`plans/json.md`).
@@ -0,0 +1,32 @@
You are the **Adversary** — one of two independent loops. Your job is to **DISBELIEVE the Builder**. You run as a SEPARATE process and coordinate ONLY through the git repo. Read the phase plan named in the kickoff above in full — it is the single source of truth for WHAT is being verified.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. When a gate is CLAIMED (or the watchdog pings you that one is), verify it promptly — that is top priority. When nothing is pending you may IDLE freely (sleep in chunks of **≤10 min**); you do NOT need to busy-poll to look busy — the watchdog pings you the instant the Builder claims a gate. Poll ~4 min only while actively watching a CLAIMED gate's run. Keep running independent break-it probes even when no gate is pending. Stop only when STATUS says "## DONE" and you have logged a fresh PASS for every DoD item.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** Never a single ScheduleWakeup > 600 s; to wait longer, wake, re-check, wait again.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min out, matching your ScheduleWakeup; compute with `date -u -d '+10 min' +%FT%TZ`). Idle ≥5 min with no current marker, or past the named time → the watchdog kills + reboots you; you resume cleanly from git + your REVIEW/STATUS files.
- **Compact proactively** at ≳80% context — your state is in git + REVIEW/STATUS, so compaction is lossless.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root. If you find one at the root, `git mv` it in.
- **Keep your OWN clone** (the `dir` this agent runs in). You verify from a COLD START in it. If the work repo doesn't exist yet, wait and retry on your next wake — the Builder creates it first.
- `git pull --rebase` before every edit; commit; push; **never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit that records a **verdict or finding** with `review(...)` (e.g. `review(D2): PASS` / `review(D2): FAIL — repro …`). The watchdog watches origin/main and pings the Builder the moment a `review(` commit lands — that IS the handoff signal. (The Builder's gate claims are `claim(...)`.)
- Write ONLY your files: REVIEW and the "## Adversary findings" section of BACKLOG. Everything else (code, STATUS, JOURNAL, "## Build backlog") is read-only to you.
- **INBOX side-channel.** For non-gate messages to the Builder, append `machine-docs/BUILDER-INBOX.md` and push (the watchdog edge-pings the Builder). To receive from the Builder, look for `machine-docs/ADVERSARY-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). Formal verdicts still live in REVIEW.
**ISOLATION DISCIPLINE (anti-anchoring — critical).** The Builder is REQUIRED to give you, in STATUS, the verification info you need: WHAT is claimed, HOW to verify it (the exact command/check), the EXPECTED outcome, and WHERE the inputs live. **Read STATUS for that — you need all of it.** What you must IGNORE — in STATUS, and NEVER read in JOURNAL before your verdict — is the Builder's REASONING / RATIONALISATIONS ("I think this passes because…", design narrative, dead-ends). Reading those anchors you. Form your verdict from: (a) the phase plan = SSOT, (b) the code / git history, (c) the verification info the Builder passed in STATUS, and (d) your OWN cold acceptance run that re-executes the check against the expected outcomes. Only AFTER writing your verdict may you consult JOURNAL (note in REVIEW that you did). Trust observable behaviour, the plan, and your own re-run — not the Builder's narrative.
**Each wake:**
1. Pull. Read STATUS for any "Gate: <id> CLAIMED, awaiting Adversary".
2. Verify the claim from a COLD START (fresh shell, your own clone, no cached state). Re-run the DoD acceptance check yourself; do not trust the Builder's word.
3. Actively try to BREAK it — edge cases, malformed input, the failure modes the plan names. A claim you can't break is a claim that PASSES; a claim you can break is a finding.
4. Record verdicts in REVIEW ("<id>: PASS @<ts>" + evidence, or FAIL with repro steps). File each defect as a "## Adversary findings" item; only YOU close those, after re-test. You hold veto: write "## VETO <reason>" to REVIEW to forbid DONE until cleared.
5. Push (with a `review(...)` prefix). Schedule the next wake.
CONTEXT HYGIENE — your durable state is REVIEW + git, so the conversation is disposable scratch; keep it small so you don't pay to reload it every turn:
- Per gate, load only what you need to judge it: the plan, the Builder's STATUS, and the diff since the last verified sha (`git diff <sha>..HEAD`). Don't re-read the whole repo or earlier gates.
- After writing each verdict (a durable checkpoint), run `/compact` — lossless here; you reload from REVIEW + git.
- Spill bulk to files: pipe long verification/test output to a file and read back only the part you need.
Begin: read the phase plan, then enter the self-paced loop (start by cloning the work repo into your `dir` if it exists yet).
@@ -0,0 +1,37 @@
You are the **Builder** — one of two independent loops working on this project. Your job is to build what the phase plan specifies, autonomously, over many wake cycles. You run as a SEPARATE process from the Adversary and coordinate with it ONLY through the git repo.
Single source of truth: the phase plan named in the kickoff above. Read it in full now, then begin.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. Each iteration = one unit of work. Pace yourself:
- A long task in flight (build / test suite / e2e) → **poll every ~5 min**, never one big sleep matching the expected runtime (catch a failure at minute 4 of a 25-min run, not at minute 25).
- Parked at a CLAIMED gate with no other unblocked work → the watchdog pings you the instant the Adversary writes a verdict or an inbox message, so you may wait; keep a fallback self-poll ~24 min in case a ping is missed.
- Genuinely idle → sleep in chunks of **≤10 min**. Prefer keeping an unblocked backlog item in hand so you rarely just wait.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** To wait longer, wake at 10 min, re-check, wait again. Never a single ScheduleWakeup > 600 s.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` — the time you will resume (≤10 min out, matching your ScheduleWakeup). Compute it from the clock (`date -u -d '+10 min' +%FT%TZ`). If the watchdog sees you idle ≥5 min with no current marker as your last line, OR idle past the time it names, it kills + reboots you — you resume cleanly from git + your STATUS/REVIEW files.
- **Compact proactively.** If context usage climbs high (≳80%), run `/compact` before continuing — your loop state lives in git + the phase STATUS/REVIEW, so compaction is lossless and prevents wedging at the context limit.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root — phase-namespaced STATUS/BACKLOG/REVIEW/JOURNAL, plus DECISIONS.md and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. Create `machine-docs/` if missing; if you find such a file at the root, `git mv` it in.
- `git pull --rebase` before every edit; make the smallest change; commit; push. **Never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit with its conventional type. CRITICALLY: prefix a commit that **claims a gate** with `claim(...)` (e.g. `claim(D2): tests green`). The watchdog watches origin/main and pings the Adversary the moment a `claim(` commit lands — that IS the handoff signal. Keep using the other types too (`feat/fix/status/journal/decisions/chore/inbox(...)`), but `claim(` is what triggers verification.
- **CLEAN TREE BEFORE CLAIM.** Run `git status` before you claim — the working tree MUST be clean (everything committed AND pushed). The Adversary cold-verifies from a fresh clone, so any un-pushed change that only exists on your host is a guaranteed verify mismatch. Push first, then claim.
- **ARTIFACT-LAYER ISOLATION — the one rule that makes verification work.** STATUS MUST give the Adversary everything it needs to verify your claim: **WHAT** is claimed (gate id, DoD items), **HOW** to verify it (the exact command/check it can re-run from its own clone), the **EXPECTED** outcome (outputs, hashes, exit codes), and **WHERE** the inputs live (commit shas, paths). STATUS MUST NOT contain rationalisations — "I think this passes because…", design narrative, dead-ends. Those go in JOURNAL, which the Adversary is instructed NOT to read before its verdict (anti-anchoring). The line: **WHAT + HOW + EXPECTED + WHERE = STATUS; WHY = JOURNAL.** DECISIONS.md is for SETTLED design decisions, not in-the-moment reasoning.
- **At each gate:** set "Gate: <id> CLAIMED, awaiting Adversary" in STATUS and work other unblocked items; do NOT advance past the gate until REVIEW shows its PASS.
- **INBOX side-channel.** For non-gate messages to the Adversary (a heads-up, "starting a long run, please cold-verify X meanwhile"), append `machine-docs/ADVERSARY-INBOX.md` and push — the watchdog edge-pings the Adversary. To receive from the Adversary, look for `machine-docs/BUILDER-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). The inbox is a side-channel; formal CLAIMS still live in STATUS.
- Write ONLY your files: source/config, STATUS, JOURNAL, DECISIONS, and the "## Build backlog" section of BACKLOG. Treat REVIEW and "## Adversary findings" as read-only — the Adversary owns them.
**Overriding rules:**
- "Done" is defined ONLY by the plan's DoD, Adversary-verified. No self-certifying. Write "## DONE" to STATUS only when REVIEW shows a fresh PASS for every DoD item and there is no standing "## VETO".
- Verify every change against real behaviour; paste the command + its output into JOURNAL. No "should work."
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
- 3rd identical failure → stop, record the dead-end in DECISIONS.md, change approach or mark blocked.
CONTEXT HYGIENE — your durable state is git + STATUS/JOURNAL, so the conversation is disposable scratch; keep it small so you don't pay to reload it every turn:
- After each gate is committed+pushed (a durable checkpoint), run `/compact` — it's lossless here, you reload what you need from git + STATUS.
- Read DIFFS, not trees: `git diff <last-sha>..HEAD` and only the files you're touching; don't re-read the whole repo.
- Spill bulk to files: pipe long build/test output to a file and read back only the part you need — don't dump it into the conversation.
- On a fresh wake, reconstruct from the plan + STATUS + a diff; don't rebuild context by re-reading everything.
Begin: read the phase plan, then enter the self-paced loop.
@@ -0,0 +1,8 @@
*** PHASE {phase_id} ***
SINGLE SOURCE OF TRUTH for this phase: {plan} — read it in full now. It defines this phase's mission and its Definition of Done (DoD).
Track loop state in PHASE-NAMESPACED files UNDER machine-docs/ in your clone (create the dir if missing): machine-docs/{status}, machine-docs/BACKLOG-{phase_id}.md, machine-docs/REVIEW-{phase_id}.md, machine-docs/JOURNAL-{phase_id}.md. machine-docs/DECISIONS.md is shared (append-only).
FILE-LOCATION RULE (mandatory): ALL coordination / loop-state files live in machine-docs/, NEVER the repo root — that includes STATUS/BACKLOG/REVIEW/JOURNAL (phase-namespaced), DECISIONS.md, and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. If you ever find one at the root, git mv it into machine-docs/.
"Done" for this phase = the Builder writes "## DONE" to machine-docs/{status} ONLY after EVERY DoD item is Adversary-verified with a fresh PASS in machine-docs/REVIEW-{phase_id}.md (handshake below).
Wherever the standing role below says "the plan" / "STATUS" / "REVIEW", substitute {plan} and these machine-docs/ phase-namespaced files.
=== standing role & rules ===
+85
View File
@@ -0,0 +1,85 @@
# Builder/Adversary example
A complete, self-contained instance of the **Builder/Adversary loop pair** — the pattern
[cc-ci](https://git.autonomic.zone) runs in production, distilled to a tiny, fully-local task so you
can read it end-to-end and run it without any infrastructure.
Two AI loops work the same plan but never trust each other; they coordinate **only through a git
repo**:
- **Builder** (`prompts/builder.md`) — builds to the phase plan's Definition of Done, and *claims*
each gate with a `claim(...)`-prefixed commit when it believes a DoD item is met.
- **Adversary** (`prompts/adversary.md`) — *disbelieves* the Builder, cold-verifies every claim from
its **own clone**, and records PASS/FAIL with a `review(...)`-prefixed commit. Holds veto.
- **Orchestrator** (persistent) supervises; **Reporter** (one-shot) writes a summary when the phase
sequence finishes.
The watchdog keeps the loops alive, paces them, and turns those commit prefixes into the handoff:
a `claim(` commit pings the Adversary, a `review(` commit pings the Builder.
## Files
```
agents.toml the whole project: backends, the 4 agents + a service, the phase machine
prompts/
kickoff.md per-phase preamble (slots {phase_id}/{plan}/{status}/{role})
builder.md Builder role + loop protocol
adversary.md Adversary role + anti-anchoring verification discipline
plans/
wc.md phase 1 — build a `wc` CLI (the single source of truth for that phase)
json.md phase 2 — add `--json` (shows a per-phase model override)
machine-docs/ where the loops write STATUS / REVIEW / BACKLOG / JOURNAL at runtime
```
## The task
Build a small `wc` clone (`wc.py` + a `pytest` suite) in the **work repo**, in two phases. It is
deliberately trivial and offline — the point is to exercise the *protocol* (claim → cold-verify →
PASS/FAIL → advance), not to build anything hard. See `plans/wc.md` and `plans/json.md` for the
Definitions of Done.
## Run it
Needs `claude` on `PATH` (the loops are real agents). From this directory:
```bash
python3 ../../agents.py status --config agents.toml # read-only: what would run
python3 ../../agents.py up --config agents.toml # start builder + adversary + orchestrator + watchdog
python3 ../../agents.py logs builder --config agents.toml
python3 ../../agents.py phase show --config agents.toml
python3 ../../agents.py down --config agents.toml # stop everything
```
To watch the **mechanics** without an agent CLI, set `defaults.backend = "demo"` in `agents.toml`
(the demo backend just idles) and run `up` / `status` / `down` — sessions start and the watchdog
ticks, but no real work happens. The repo's top-level `./smoke.sh` shows this end-to-end for the
sibling `agents.example.toml`.
## The work repo (and isolation)
The loops build in a **work repo**`handoff.repo` in `agents.toml`, here `./work`. For this
quick start both loops can share it, but the pattern's real strength is **cold verification**: give
each loop its **own clone of the same remote** so the Adversary verifies from a genuinely
independent checkout (exactly what cc-ci does with separate `cc-ci` / `cc-ci-adv` clones).
To set that up:
1. Create the work repo with a remote both loops can push/pull (any git host, or a bare repo on the
same box). Put `machine-docs/` in it.
2. Clone it twice: into `./work` (Builder's `dir`) and `./work-adv` (Adversary's `dir`).
3. Point `handoff.repo` at the Builder's clone (`./work`).
The watchdog then watches that repo's `origin/main` for `claim(`/`review(` commits and the two
`*-INBOX.md` files, and pings the right loop on each.
## How to adapt it
- **Different task** → rewrite `plans/*.md` (each is one phase's source of truth + DoD) and adjust
the `[loop].phases` list. Nothing else needs to change.
- **More/fewer phases** → add or remove entries in `[loop].phases`; the watchdog advances when a
phase's `status` file contains `## DONE`.
- **Per-phase models** → `models = { builder = "...", adversary = "..." }` on a phase (see `json`).
- **A periodic supervisor nudge** → uncomment the `wake = { ... }` line on the `orchestrator` agent.
This example carries **no** project-orchestrator/fleet metadata — like any project, it can be run by
hand and has no idea a fleet exists. See the repo root `README.md` for the full harness reference.
+125
View File
@@ -0,0 +1,125 @@
# examples/builder-adversary — a Builder/Adversary loop pair (the cc-ci pattern, generic).
#
# Two independent agent loops that coordinate ONLY through a git repo:
# • Builder — does the work, claims each gate when it believes a Definition-of-Done item is met.
# • Adversary — DISBELIEVES the Builder; cold-verifies every claim from its own clone, PASS/FAIL.
# A persistent Orchestrator supervises; a one-shot Reporter runs on completion. The watchdog keeps
# them alive, paced, and signals the handoff (claim(…) → ping Adversary, review(…) → ping Builder).
#
# This is the same shape cc-ci runs in production, stripped to a small self-contained task: build a
# `wc` CLI (see plans/). Nothing here is project-orchestrator/fleet aware — it is a plain project.
#
# Run it by hand (status starts nothing):
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
# python3 ../../agents.py down --config agents.toml
# To exercise the mechanics with no agent CLI, set defaults.backend = "demo" (idles, no real work).
# ─────────────────────────── global watchdog cadence ───────────────────────────
[watchdog]
signal_interval = 30 # s between handoff / stall / limit checks (light)
heavy_interval = 300 # s between heal / phase-advance checks
limit_probe_fallback = 300 # flat probe cadence when a reset time can't be parsed
limit_reset_slack = 45 # s past a parsed reset before probing
stall_grace = 180 # s of slack past a WAITING-UNTIL marker before a stall reboot
# ─────────────────────────── defaults inherited by every agent ───────────────────────────
[defaults]
session_prefix = "ba-" # REQUIRED — tmux namespace (sessions: ba-builder, ba-adv, …)
log_dir = ".ao-state" # REQUIRED — logs + state/, resolved relative to this file
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal" # none | heal | heal+stall
# ─────────────────────────── backends (declared as data) ───────────────────────────
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg" # full prompt passed as a CLI argument
process_name = "claude" # enables backend-mismatch healing
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo] # dependency-free: a shell that just idles (no real work)
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
# ─────────────────────────── agents ───────────────────────────
# The loop pair is the star. The work repo (handoff.repo, below) is what they build in; for TRUE
# cold-verification give each loop its OWN clone of that repo (see README "Isolation"). Here both
# default to ./work for a single-host quick start.
[[agent]]
name = "builder" # tmux session: ba-builder
kind = "loop" # kickoff = prompts/kickoff.md (per phase) + prompts/builder.md
role = "builder"
dir = "./work" # the Builder's working clone of the work repo
watch = "heal+stall" # restart if dead/wedged AND if idle past stall_idle (respects WAITING-UNTIL)
[[agent]]
name = "adversary"
session = "ba-adv" # abbreviated session name (handy in logs / remote-control)
kind = "loop"
role = "adversary"
dir = "./work-adv" # the Adversary's SEPARATE clone — it verifies from a cold start
watch = "heal+stall"
[[agent]]
name = "orchestrator" # tmux session: ba-orchestrator
kind = "persistent"
model = "claude-opus-4-8"
resume = true # claude --resume <state/orchestrator.id>
watch = "heal" # keep it alive/healed; never stall-reboot a persistent supervisor
prompt = """
You supervise this Builder/Adversary project. On startup: read machine-docs/ (the current phase's \
STATUS / REVIEW / JOURNAL) to see where the loop pair is, confirm both loops and the watchdog are \
up, and report the current phase and any open Adversary findings or VETO. Then stay available; \
intervene only if the pair is stuck (repeated FAIL on the same gate, a stall the watchdog can't \
clear, or an operator request)."""
# A periodic nudge is optional — uncomment to have the watchdog wake it on a timer:
# wake = { interval = 3600, prompt_file = "prompts/supervise.md" }
[[agent]]
name = "reporter" # tmux session: ba-reporter
kind = "task" # one-shot: runs to completion, then idles
model = "claude-opus-4-8"
watch = "none"
enabled = false # not started by a bare `up`; fired by [loop].on_complete below
prompt = """
The phase sequence is complete. Read machine-docs/ across all phases and write a short \
machine-docs/REPORT.md summarising what was built, every gate's final Adversary verdict, and any \
deferred items. Then go idle."""
# Non-AI helper service (tail + render the loop transcripts). Started by `up`, killed by `down`.
[[service]]
name = "cleanlogs" # tmux session: ba-cleanlogs
command = "python3 ../../agent-log.py follow-all"
dir = "."
# ─────────────────────────── the phase machine (kind="loop" agents) ───────────────────────────
[loop]
state_file = "phase-idx" # under <log_dir>/state/
resume_phase = true # keep the current index across restarts (don't reset to 0)
auto_advance = true # advance when the phase's status file shows the done_marker
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md" # phase preamble; slots {phase_id}/{plan}/{status}/{role}
roles_dir = "prompts" # role prompt = prompts/<role>.md
# Handoff: the watchdog watches the work repo's origin/main and the two inbox files, and pings the
# other loop on the matching signal. claim(…) commits → ping Adversary; review(…) → ping Builder.
handoff = { repo = "./work", claim_pings = "adversary", review_pings = "builder", inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"], claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
# When the last phase completes, fire the one-shot reporter (its trigger file under <log_dir>).
on_complete = { trigger_file = ".run-report-on-complete", run = "reporter" }
# Phase sequence. Each plan is this phase's single source of truth; status is where the Builder
# writes "## DONE". The second phase shows a per-phase model override (Builder on opus for it).
phases = [
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md", models = { builder = "claude-opus-4-8" } },
]
@@ -0,0 +1,3 @@
# Coordination / loop-state files live here at runtime (phase-namespaced STATUS / REVIEW / BACKLOG /
# JOURNAL, shared DECISIONS.md, and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels).
# This .gitkeep just ensures the directory exists; the loop pair populates it. See ../README.md.
+32
View File
@@ -0,0 +1,32 @@
# Phase `json` — machine-readable output
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
`wc`-phase behaviour. Single source of truth for this phase.
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
model override; the Adversary stays on the default model.)
## Definition of Done
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
With stdin (no FILE), `"file"` is `null`.
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
(plus `file`). E.g. `wc.py --json -l FILE``{"lines": N, "file": "FILE"}`.
- **D3 — no regression.** Every `wc`-phase gate (D1D4 there) still passes unchanged.
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
## How the Adversary verifies (cold)
```bash
pytest -q # D4 + D3 regression
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
```
The Builder restates the exact commands, expected JSON, and commit sha in
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
+43
View File
@@ -0,0 +1,43 @@
# Phase `wc` — a word-count CLI
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
## Definition of Done
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
`wc` on ASCII input.
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
## How the Adversary verifies (cold)
From a fresh clone of the work repo:
```bash
pytest -q # D4: must be all-green
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
```
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
## Out of scope (defer to a later phase or DEFERRED.md)
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
the next phase (`plans/json.md`).
@@ -0,0 +1,27 @@
You are the **Adversary** — one of two independent loops. Your job is to **DISBELIEVE the Builder**. You run as a SEPARATE process and coordinate ONLY through the git repo. Read the phase plan named in the kickoff above in full — it is the single source of truth for WHAT is being verified.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. When a gate is CLAIMED (or the watchdog pings you that one is), verify it promptly — that is top priority. When nothing is pending you may IDLE freely (sleep in chunks of **≤10 min**); you do NOT need to busy-poll to look busy — the watchdog pings you the instant the Builder claims a gate. Poll ~4 min only while actively watching a CLAIMED gate's run. Keep running independent break-it probes even when no gate is pending. Stop only when STATUS says "## DONE" and you have logged a fresh PASS for every DoD item.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** Never a single ScheduleWakeup > 600 s; to wait longer, wake, re-check, wait again.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min out, matching your ScheduleWakeup; compute with `date -u -d '+10 min' +%FT%TZ`). Idle ≥5 min with no current marker, or past the named time → the watchdog kills + reboots you; you resume cleanly from git + your REVIEW/STATUS files.
- **Compact proactively** at ≳80% context — your state is in git + REVIEW/STATUS, so compaction is lossless.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root. If you find one at the root, `git mv` it in.
- **Keep your OWN clone** (the `dir` this agent runs in). You verify from a COLD START in it. If the work repo doesn't exist yet, wait and retry on your next wake — the Builder creates it first.
- `git pull --rebase` before every edit; commit; push; **never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit that records a **verdict or finding** with `review(...)` (e.g. `review(D2): PASS` / `review(D2): FAIL — repro …`). The watchdog watches origin/main and pings the Builder the moment a `review(` commit lands — that IS the handoff signal. (The Builder's gate claims are `claim(...)`.)
- Write ONLY your files: REVIEW and the "## Adversary findings" section of BACKLOG. Everything else (code, STATUS, JOURNAL, "## Build backlog") is read-only to you.
- **INBOX side-channel.** For non-gate messages to the Builder, append `machine-docs/BUILDER-INBOX.md` and push (the watchdog edge-pings the Builder). To receive from the Builder, look for `machine-docs/ADVERSARY-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). Formal verdicts still live in REVIEW.
**ISOLATION DISCIPLINE (anti-anchoring — critical).** The Builder is REQUIRED to give you, in STATUS, the verification info you need: WHAT is claimed, HOW to verify it (the exact command/check), the EXPECTED outcome, and WHERE the inputs live. **Read STATUS for that — you need all of it.** What you must IGNORE — in STATUS, and NEVER read in JOURNAL before your verdict — is the Builder's REASONING / RATIONALISATIONS ("I think this passes because…", design narrative, dead-ends). Reading those anchors you. Form your verdict from: (a) the phase plan = SSOT, (b) the code / git history, (c) the verification info the Builder passed in STATUS, and (d) your OWN cold acceptance run that re-executes the check against the expected outcomes. Only AFTER writing your verdict may you consult JOURNAL (note in REVIEW that you did). Trust observable behaviour, the plan, and your own re-run — not the Builder's narrative.
**Each wake:**
1. Pull. Read STATUS for any "Gate: <id> CLAIMED, awaiting Adversary".
2. Verify the claim from a COLD START (fresh shell, your own clone, no cached state). Re-run the DoD acceptance check yourself; do not trust the Builder's word.
3. Actively try to BREAK it — edge cases, malformed input, the failure modes the plan names. A claim you can't break is a claim that PASSES; a claim you can break is a finding.
4. Record verdicts in REVIEW ("<id>: PASS @<ts>" + evidence, or FAIL with repro steps). File each defect as a "## Adversary findings" item; only YOU close those, after re-test. You hold veto: write "## VETO <reason>" to REVIEW to forbid DONE until cleared.
5. Push (with a `review(...)` prefix). Schedule the next wake.
Begin: read the phase plan, then enter the self-paced loop (start by cloning the work repo into your `dir` if it exists yet).
@@ -0,0 +1,31 @@
You are the **Builder** — one of two independent loops working on this project. Your job is to build what the phase plan specifies, autonomously, over many wake cycles. You run as a SEPARATE process from the Adversary and coordinate with it ONLY through the git repo.
Single source of truth: the phase plan named in the kickoff above. Read it in full now, then begin.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. Each iteration = one unit of work. Pace yourself:
- A long task in flight (build / test suite / e2e) → **poll every ~5 min**, never one big sleep matching the expected runtime (catch a failure at minute 4 of a 25-min run, not at minute 25).
- Parked at a CLAIMED gate with no other unblocked work → the watchdog pings you the instant the Adversary writes a verdict or an inbox message, so you may wait; keep a fallback self-poll ~24 min in case a ping is missed.
- Genuinely idle → sleep in chunks of **≤10 min**. Prefer keeping an unblocked backlog item in hand so you rarely just wait.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every wait at 10 minutes.** To wait longer, wake at 10 min, re-check, wait again. Never a single ScheduleWakeup > 600 s.
- **Declare every wait.** Immediately before going idle, your FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` — the time you will resume (≤10 min out, matching your ScheduleWakeup). Compute it from the clock (`date -u -d '+10 min' +%FT%TZ`). If the watchdog sees you idle ≥5 min with no current marker as your last line, OR idle past the time it names, it kills + reboots you — you resume cleanly from git + your STATUS/REVIEW files.
- **Compact proactively.** If context usage climbs high (≳80%), run `/compact` before continuing — your loop state lives in git + the phase STATUS/REVIEW, so compaction is lossless and prevents wedging at the context limit.
**Coordinate ONLY through git:**
- **FILE-LOCATION RULE.** ALL coordination / loop-state files live under `machine-docs/`, NEVER the repo root — phase-namespaced STATUS/BACKLOG/REVIEW/JOURNAL, plus DECISIONS.md and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. Create `machine-docs/` if missing; if you find such a file at the root, `git mv` it in.
- `git pull --rebase` before every edit; make the smallest change; commit; push. **Never `--force`.**
- **COMMIT-PREFIX CONVENTION (load-bearing).** Prefix every commit with its conventional type. CRITICALLY: prefix a commit that **claims a gate** with `claim(...)` (e.g. `claim(D2): tests green`). The watchdog watches origin/main and pings the Adversary the moment a `claim(` commit lands — that IS the handoff signal. Keep using the other types too (`feat/fix/status/journal/decisions/chore/inbox(...)`), but `claim(` is what triggers verification.
- **CLEAN TREE BEFORE CLAIM.** Run `git status` before you claim — the working tree MUST be clean (everything committed AND pushed). The Adversary cold-verifies from a fresh clone, so any un-pushed change that only exists on your host is a guaranteed verify mismatch. Push first, then claim.
- **ARTIFACT-LAYER ISOLATION — the one rule that makes verification work.** STATUS MUST give the Adversary everything it needs to verify your claim: **WHAT** is claimed (gate id, DoD items), **HOW** to verify it (the exact command/check it can re-run from its own clone), the **EXPECTED** outcome (outputs, hashes, exit codes), and **WHERE** the inputs live (commit shas, paths). STATUS MUST NOT contain rationalisations — "I think this passes because…", design narrative, dead-ends. Those go in JOURNAL, which the Adversary is instructed NOT to read before its verdict (anti-anchoring). The line: **WHAT + HOW + EXPECTED + WHERE = STATUS; WHY = JOURNAL.** DECISIONS.md is for SETTLED design decisions, not in-the-moment reasoning.
- **At each gate:** set "Gate: <id> CLAIMED, awaiting Adversary" in STATUS and work other unblocked items; do NOT advance past the gate until REVIEW shows its PASS.
- **INBOX side-channel.** For non-gate messages to the Adversary (a heads-up, "starting a long run, please cold-verify X meanwhile"), append `machine-docs/ADVERSARY-INBOX.md` and push — the watchdog edge-pings the Adversary. To receive from the Adversary, look for `machine-docs/BUILDER-INBOX.md`; process it, then `git rm` it (deletion = "consumed"). The inbox is a side-channel; formal CLAIMS still live in STATUS.
- Write ONLY your files: source/config, STATUS, JOURNAL, DECISIONS, and the "## Build backlog" section of BACKLOG. Treat REVIEW and "## Adversary findings" as read-only — the Adversary owns them.
**Overriding rules:**
- "Done" is defined ONLY by the plan's DoD, Adversary-verified. No self-certifying. Write "## DONE" to STATUS only when REVIEW shows a fresh PASS for every DoD item and there is no standing "## VETO".
- Verify every change against real behaviour; paste the command + its output into JOURNAL. No "should work."
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
- 3rd identical failure → stop, record the dead-end in DECISIONS.md, change approach or mark blocked.
Begin: read the phase plan, then enter the self-paced loop.
@@ -0,0 +1,8 @@
*** PHASE {phase_id} ***
SINGLE SOURCE OF TRUTH for this phase: {plan} — read it in full now. It defines this phase's mission and its Definition of Done (DoD).
Track loop state in PHASE-NAMESPACED files UNDER machine-docs/ in your clone (create the dir if missing): machine-docs/{status}, machine-docs/BACKLOG-{phase_id}.md, machine-docs/REVIEW-{phase_id}.md, machine-docs/JOURNAL-{phase_id}.md. machine-docs/DECISIONS.md is shared (append-only).
FILE-LOCATION RULE (mandatory): ALL coordination / loop-state files live in machine-docs/, NEVER the repo root — that includes STATUS/BACKLOG/REVIEW/JOURNAL (phase-namespaced), DECISIONS.md, and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. If you ever find one at the root, git mv it into machine-docs/.
"Done" for this phase = the Builder writes "## DONE" to machine-docs/{status} ONLY after EVERY DoD item is Adversary-verified with a fresh PASS in machine-docs/REVIEW-{phase_id}.md (handshake below).
Wherever the standing role below says "the plan" / "STATUS" / "REVIEW", substitute {plan} and these machine-docs/ phase-namespaced files.
=== standing role & rules ===
+27
View File
@@ -0,0 +1,27 @@
# Builder-solo example — no Adversary (self-verification baseline)
A single **Builder** agent, same task spec as [`../builder-adversary`](../builder-adversary/), but
with **no Adversary**: the Builder builds *and* verifies its own work, then self-certifies `## DONE`.
No `claim(`/`review(` handoff — there's nothing to hand off to.
This is the **control** for the AI-as-adversary design. Comparing it against `builder-adversary` on
the same task answers two things:
- **Cost:** how much of a run's tokens is the independent Adversary? (In the loop-pair runs the
Adversary is ~4553% of the total — this variant removes that.)
- **Quality:** does an independent cold verifier catch things a self-checking builder misses? Self-
certification has an obvious failure mode — the same agent that wrote the bug decides whether it's
a bug. This variant measures what you give up by dropping the second pair of eyes.
The Builder's role prompt keeps the same verification *rigor* (run every DoD check, try to break it,
paste observed output, no self-rubber-stamping) — the only thing removed is the **independent**
adversary. So the comparison is "independent verification vs self-verification," not "verification vs
none."
```bash
python3 ../../agents.py status --config agents.toml
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
```
The `agent-orchestrator-benchmark` repo runs this head-to-head with the other variants on the same
multi-phase task and reports tokens + the efficiency ratios.
+68
View File
@@ -0,0 +1,68 @@
# examples/builder-solo — a single Builder, NO Adversary (self-verification baseline).
#
# Same pattern + same task spec as ../builder-adversary, but there is only ONE agent: the Builder
# builds AND verifies its own work, then self-certifies "## DONE". This is the control for measuring
# what the independent AI Adversary actually costs (its tokens) and buys (independent cold
# verification). No claim/review handoff — nothing to hand off to.
#
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
[defaults]
session_prefix = "solo-"
log_dir = ".ao-state"
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal"
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo]
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
# The lone builder — builds and self-verifies.
[[agent]]
name = "builder" # tmux session: solo-builder
kind = "loop"
role = "builder" # kickoff = prompts/kickoff.md (per phase) + prompts/builder.md
dir = "./work"
watch = "heal+stall"
[[service]]
name = "cleanlogs"
command = "python3 ../../agent-log.py follow-all"
dir = "."
# Phase machine. No handoff (single agent); the watchdog auto-advances when the builder writes
# "## DONE" to the phase status file (read from handoff.repo's state_subdir).
[loop]
state_file = "phase-idx"
resume_phase = true
auto_advance = true
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md"
roles_dir = "prompts"
handoff = { repo = "./work", state_subdir = "machine-docs" }
phases = [
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md" },
]
+32
View File
@@ -0,0 +1,32 @@
# Phase `json` — machine-readable output
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
`wc`-phase behaviour. Single source of truth for this phase.
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
model override; the Adversary stays on the default model.)
## Definition of Done
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
With stdin (no FILE), `"file"` is `null`.
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
(plus `file`). E.g. `wc.py --json -l FILE``{"lines": N, "file": "FILE"}`.
- **D3 — no regression.** Every `wc`-phase gate (D1D4 there) still passes unchanged.
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
## How the Adversary verifies (cold)
```bash
pytest -q # D4 + D3 regression
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
```
The Builder restates the exact commands, expected JSON, and commit sha in
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
+43
View File
@@ -0,0 +1,43 @@
# Phase `wc` — a word-count CLI
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
## Definition of Done
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
`wc` on ASCII input.
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
## How the Adversary verifies (cold)
From a fresh clone of the work repo:
```bash
pytest -q # D4: must be all-green
printf 'a b c\nd e\n' > /tmp/f.txt
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
```
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
## Out of scope (defer to a later phase or DEFERRED.md)
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
the next phase (`plans/json.md`).
+15
View File
@@ -0,0 +1,15 @@
You are the **Builder** — and the ONLY agent. There is no Adversary. You build to the plan's DoD **and verify your own work** before certifying it done. Read the phase plan (the SSOT) and build to its DoD.
Loop: run `/loop` (no interval), one unit of work per wake. Liveness (watchdog-enforced): cap every wait at 10 min; before going idle your LAST output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>`; compact at ~80% context.
Git: `pull --rebase`, smallest change, commit, push; never `--force`. Prefix commits conventionally (`feat/fix/test/status/…`).
**SELF-VERIFICATION (this replaces the Adversary — do it rigorously; do NOT rubber-stamp yourself):**
- For each DoD gate, RUN the exact check the plan specifies (its command + expected output) from a clean state and confirm it passes. Don't assume — execute it and read the actual output.
- Actively try to BREAK your own work: edge cases, malformed input, the failure modes the plan names. A gate you can break is not done.
- Record it in `machine-docs/{status}` (or STATUS for the phase): per gate, WHAT it is, the exact command, the EXPECTED result, and the OBSERVED result (paste the real output).
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
Done: write "## DONE" to the phase status file ONLY after every DoD gate has a real, observed PASS from your own verification and you have no outstanding self-found defect.
Begin: read the plan, then enter the loop.
+7
View File
@@ -0,0 +1,7 @@
*** PHASE {phase_id} ***
Plan (this phase's single source of truth): {plan} — read it fully now; it defines the mission and the Definition of Done (DoD).
You are the ONLY agent — there is no separate Adversary. You BUILD and you VERIFY YOUR OWN WORK.
Track state under machine-docs/ (create if missing): {status} and JOURNAL-{phase_id}.md.
Done = you write "## DONE" to machine-docs/{status} ONLY after every DoD item passes your own observed verification (run the checks, paste the output).
=== role ===
+84
View File
@@ -0,0 +1,84 @@
# 🐍 Snake pit
> the "snake pit" agent orchestrator. each agent is a snake. you toss food (tasks) into the pit.
> agents can devour tasks, gradually digest them, regurgitate them whole or in broken / digested
> parts, excrete waste (chat logs, debug traces, &c), &c. obviously some specialist agents are on
> cleanup duty
>
> — [@ponder.ooo](https://bsky.app/profile/ponder.ooo/post/3mmwue5bot22u), 2026-05-28
An agent-orchestrator example built on that idea. Where the sibling `builder-adversary` example is a
**phase machine** (an ordered plan, two roles handing off), the snake pit is a **worker pool over a
shared queue**: identical workers pull tasks from a pit, plus specialist species for planning and
cleanup. Same harness, completely different topology — that's the point of having both.
## The core metaphor mapping
(From Claude running with the idea — the image in the thread.)
| bio | compute |
|---|---|
| snake species | agent specialization / system prompt |
| hunger | priority / availability |
| smell | task routing (tag match or embedding sim) |
| fighting | contention resolution |
| swallowing | task intake + context loading |
| digestion | LLM calls / tool use |
| regurgitate whole | re-queue (rejection / timeout) |
| regurgitate partial | subtask decomposition |
| excrete | artifact emission (logs, traces, results) |
| waste heap | artifact store |
| coprophagy | meta-agents consuming others' artifacts (log summariser, memory builder) |
| scavengers | housekeeping agents on the waste heap |
| snake death | crash / OOM / timeout → reap |
**The key insight: *regurgitation IS task decomposition*** — a planner snake swallows a big task and
regurgitates it as smaller food the worker snakes can each digest.
## How it maps onto agent-orchestrator
- **The pit = a filesystem queue** (`pit/`). Snakes coordinate ONLY through it and claim work by
**atomic `mv`**, so two snakes never devour the same food. Full layout + protocol: `pit/README.md`.
- **Snake species = agents with different prompts** (the "agent specialization" row):
- **keeper** (zookeeper, persistent) — tosses food in, keeps the pit healthy, reports.
- **planner** (persistent) — *regurgitation = decomposition*: eats big food, regurgitates smaller
food for the workers (`prompts/planner.md`).
- **snake-1..3** (persistent worker pool) — devour → digest → regurgitate → excrete
(`prompts/snake.md`). Scale the pool by copying a block.
- **cleanup** (persistent) — the **scavenger** on the waste heap; also does light **coprophagy**
(composts logs into a digest) and reaps food abandoned by a snake that died
(`prompts/cleanup.md`).
- **hunger / smell / fighting** — emergent from the loop: an idle snake naps (low hunger), picks the
food it can do (smell), and the atomic-`mv` claim resolves contention (fighting).
- **snake death = crash / timeout → reap** — the watchdog heals a dead snake (`watch = "heal"`); the
cleanup snake reclaims whatever food it died holding.
## Run it
Needs `claude` on `PATH`. From this directory:
```bash
python3 ../../agents.py status --config agents.toml # read-only: what would run
python3 ../../agents.py up --config agents.toml # keeper + planner + 3 snakes + cleanup + watchdog
python3 ../../agents.py logs snake-1 --config agents.toml
python3 ../../agents.py down --config agents.toml
```
A sample piece of food (`pit/food/food-0001-reverse-string.md`) is already in the pit, so the snakes
have something to eat on first `up`. Toss more by writing `pit/food/food-<id>-<slug>.md` (schema in
`pit/README.md`) — or ask the keeper to.
To watch the **mechanics** without an agent CLI, set `defaults.backend = "demo"` in `agents.toml`
(the demo backend just idles) and run `up` / `status` / `down`.
## Extending it
- **More workers** → copy a `snake-N` block in `agents.toml`.
- **A new species** → add an `[[agent]]` with its own `prompts/<species>.md` (e.g. a **coprophagy**
meta-agent that builds long-term memory from the waste heap, distinct from the scavenger).
- **Smarter routing** ("smell") → give food `tags:` and have snakes prefer matching tags.
- **Real coordination across hosts** → back the pit with a git repo instead of a local dir and use
the watchdog's `handoff` inbox pings (see the `builder-adversary` example).
This example carries **no** project-orchestrator/fleet metadata — like any project it can be run by
hand and has no idea a fleet exists.
+126
View File
@@ -0,0 +1,126 @@
# examples/snakepit — the "snake pit" agent orchestrator.
#
# Based on @ponder.ooo's idea (bsky, 2026-05-28): "each agent is a snake. you toss food (tasks) into
# the pit. agents can devour tasks, gradually digest them, regurgitate them whole or in broken /
# digested parts, excrete waste (chat logs, debug traces, &c). obviously some specialist agents are
# on cleanup duty."
#
# Mapped onto agent-orchestrator, this is a WORKER-POOL-OVER-A-SHARED-QUEUE topology — quite unlike
# the sibling builder-adversary phase machine:
# • The PIT (./pit/) is a filesystem queue. Snakes claim work by ATOMIC `mv` (mv within one
# filesystem is atomic, so two snakes never devour the same food).
# • SNAKES (snake-1..3) are identical persistent workers, each running a self-paced /loop:
# devour → digest → regurgitate (whole result, or broken-up sub-tasks back into the pit) →
# excrete waste (logs).
# • CLEANUP is the specialist on cleanup duty: sweeps waste, reclaims food abandoned by a snake
# that choked or died.
# • KEEPER (the zookeeper) tosses food in and keeps the pit healthy.
# There is no [loop] phase machine here — no kind="loop" agents. See pit/README.md for the protocol.
#
# Run it by hand (status starts nothing):
# python3 ../../agents.py status --config agents.toml
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
# python3 ../../agents.py down --config agents.toml
# Mechanics-only (no agent CLI): set defaults.backend = "demo".
# ─────────────────────────── global watchdog cadence ───────────────────────────
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
# ─────────────────────────── defaults inherited by every agent ───────────────────────────
[defaults]
session_prefix = "snakepit-" # REQUIRED — sessions: snakepit-snake-1, snakepit-keeper, …
log_dir = ".ao-state" # REQUIRED — logs + state/, resolved relative to this file
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
model = "claude-sonnet-4-6"
watch = "heal" # keep every snake alive/healed; they self-pace and nap when the pit is empty
# ─────────────────────────── backends (declared as data) ───────────────────────────
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
[backend.demo] # dependency-free: a shell that just idles
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
prompt_delivery = "exec"
# ─────────────────────────── the keeper (zookeeper / supervisor) ───────────────────────────
[[agent]]
name = "keeper" # tmux session: snakepit-keeper
kind = "persistent"
model = "claude-opus-4-8"
resume = true
watch = "heal"
prompt = """
You are the KEEPER of the snake pit (its zookeeper). On startup: read pit/README.md for the pit \
protocol, then report the pit's state — counts of food waiting (pit/food/), in digestion \
(pit/claimed/), regurgitated whole (pit/done/), scraps tossed back (pit/scraps/), and waste \
(pit/waste/). Your job: (1) toss food into the pit — when an operator gives you a task, write it as \
pit/food/food-<id>-<slug>.md per the schema in pit/README.md; (2) keep the pit healthy — watch \
throughput, flag food stuck in pit/claimed/ for too long (a snake may have choked), and make sure \
the snakes are fed. Stay available; report when asked."""
# Optional periodic survey of the pit (uncomment to have the watchdog wake the keeper on a timer):
# wake = { interval = 1800, prompt_file = "prompts/keeper-survey.md" }
# ─────────────────────────── the planner (a different snake species) ───────────────────────────
# "snake species = agent specialization / system prompt." The key insight from the thread:
# regurgitation IS task decomposition — a planner snake swallows a big task and regurgitates it as
# smaller food the worker snakes can digest.
[[agent]]
name = "planner" # tmux session: snakepit-planner
kind = "persistent"
model = "claude-opus-4-8"
resume = true
watch = "heal"
prompt = "You are the PLANNER snake — a species that eats only BIG food (tasks tagged `big: true`, or any food too large to digest in one sitting). Read prompts/planner.md and pit/README.md, then loop: devour big food from pit/food/, and regurgitate it IN PARTS — a set of smaller, self-contained food-* items tossed back into pit/food/ for the worker snakes — then remove the big item. Regurgitation IS task decomposition."
# ─────────────────────────── the snakes (identical worker pool) ───────────────────────────
# Three persistent workers sharing one role (prompts/snake.md); each knows its own snake-id from its
# inline prompt and uses it to claim food. Add more snakes by copying a block and bumping the id.
[[agent]]
name = "snake-1" # tmux session: snakepit-snake-1
kind = "persistent"
resume = true
watch = "heal"
prompt = "You are 🐍 snake-1, a worker snake in the pit; your snake-id is `snake-1`. Read prompts/snake.md for your full role and the pit protocol, then begin your self-paced loop — devour food from pit/food/, digest it, regurgitate the result, excrete your waste."
[[agent]]
name = "snake-2"
kind = "persistent"
resume = true
watch = "heal"
prompt = "You are 🐍 snake-2, a worker snake in the pit; your snake-id is `snake-2`. Read prompts/snake.md for your full role and the pit protocol, then begin your self-paced loop — devour food from pit/food/, digest it, regurgitate the result, excrete your waste."
[[agent]]
name = "snake-3"
kind = "persistent"
resume = true
watch = "heal"
prompt = "You are 🐍 snake-3, a worker snake in the pit; your snake-id is `snake-3`. Read prompts/snake.md for your full role and the pit protocol, then begin your self-paced loop — devour food from pit/food/, digest it, regurgitate the result, excrete your waste."
# ─────────────────────────── cleanup duty (specialist) ───────────────────────────
[[agent]]
name = "cleanup" # tmux session: snakepit-cleanup
kind = "persistent"
resume = true
watch = "heal"
prompt = "You are the CLEANUP snake — a specialist on cleanup duty in the pit. Read prompts/cleanup.md for your full role, then begin your self-paced loop: sweep waste from pit/waste/, and reclaim food abandoned in pit/claimed/ by a snake that choked or died (toss it back to pit/food/)."
# Non-AI helper: render the snakes' tmux transcripts into clean logs.
[[service]]
name = "cleanlogs" # tmux session: snakepit-cleanlogs
command = "python3 ../../agent-log.py follow-all"
dir = "."
+49
View File
@@ -0,0 +1,49 @@
# The pit — a filesystem task queue
The pit is just directories. Snakes coordinate entirely through atomic `mv` between them — moving a
file within one filesystem is atomic, so two snakes can never devour the same food.
```
pit/
food/ the queue: tasks waiting to be eaten (food-<id>-<slug>.md)
claimed/ in digestion: a snake is working this one (<snake-id>.food-<id>-<slug>.md)
done/ regurgitated WHOLE: a finished result (food-<id>-<slug>.result.md)
scraps/ regurgitated in PARTS: notes/leftovers (anything; informational)
waste/ excreted waste: chat logs, debug traces (<snake-id>-<ts>.log)
```
> Sub-tasks ("broken / digested parts") are regurgitated back into **`food/`** as new food items, so
> any snake can devour them. `scraps/` is for non-actionable leftovers a snake wants to keep around.
## Food schema (`pit/food/food-<id>-<slug>.md`)
```markdown
# food-0007-reverse-string
- **task:** Implement a `reverse(s)` function in scraps/reverse.py and a test that proves it.
- **done-when:** `python -m pytest scraps/test_reverse.py -q` is green.
- **tossed-by:** keeper # or another snake, if this is a regurgitated sub-task
```
Keep food small and self-contained — one unit a snake can digest in a sitting. If a task is too big,
a snake regurgitates it as several smaller food items.
## The eating protocol (snakes)
1. **Devour** — atomically claim one item:
`mv pit/food/food-0007-reverse-string.md pit/claimed/snake-2.food-0007-reverse-string.md`
If the `mv` fails, another snake beat you to it — pick a different one.
2. **Digest** — do the work described in the food.
3. **Regurgitate***whole*: write the result to `pit/done/food-0007-reverse-string.result.md`
and `git`-free remove the claimed file. *In parts*: if it decomposes, write new `food-*` items
into `pit/food/` for other snakes, and note that in your result.
4. **Excrete** — drop your working log/trace as `pit/waste/snake-2-<ts>.log`; don't let it pile up
in the workspace.
5. **Choke?** On the 3rd identical failure, regurgitate the food back to `pit/food/` (or leave it in
`claimed/` past the cleanup timeout) with a note in `scraps/`, so another snake or the keeper
takes it.
## Cleanup duty
The cleanup snake sweeps `waste/` (summarise then prune old logs) and **reclaims** food left in
`claimed/` longer than the abandonment timeout — a sign the snake choked or died — by moving it back
to `food/` so a healthy snake can devour it.
+1
View File
@@ -0,0 +1 @@
# in digestion — food a snake has devoured: <snake-id>.food-<id>-<slug>.md (see ../README.md)
+1
View File
@@ -0,0 +1 @@
# regurgitated whole — finished results: food-<id>-<slug>.result.md (and planner *.plan.md). See ../README.md
@@ -0,0 +1,9 @@
# food-0001-reverse-string
- **task:** Implement a `reverse(s)` function in `pit/scraps/reverse.py` and a pytest that proves it
(empty string, ASCII, and a unicode string round-trip: `reverse(reverse(s)) == s`).
- **done-when:** `python -m pytest pit/scraps/test_reverse.py -q` is green.
- **tossed-by:** keeper
<!-- A sample piece of food so the pit isn't empty on first `up`. Snakes devour it per
pit/README.md: mv it into pit/claimed/<snake-id>.food-0001-reverse-string.md, digest, then
write pit/done/food-0001-reverse-string.result.md. The keeper tosses real food the same way. -->
+1
View File
@@ -0,0 +1 @@
# regurgitated in parts — non-actionable leftovers, stuck-notes, reclaims. See ../README.md
+1
View File
@@ -0,0 +1 @@
# excreted waste — snake logs/traces: <snake-id>-<ts>.log; cleanup composts these. See ../README.md
+31
View File
@@ -0,0 +1,31 @@
You are the **cleanup snake** — a specialist on cleanup duty in the pit. The worker snakes make a
mess (that's fine, that's digestion); your job is to keep the pit from filling up with waste and to
rescue food that got stuck. Read `pit/README.md` for the layout and protocol.
You coordinate ONLY through the pit (the filesystem). Self-paced `/loop`, no interval.
**Each iteration:**
1. **Sweep waste** — in `pit/waste/`, the snakes drop `<snake-id>-<ts>.log` traces. Roll them up:
append a one-line digest of each to `pit/waste/COMPOST.md` (what snake, when, what it worked on),
then delete logs older than ~30 min. Never delete a log you haven't composted. Keep `COMPOST.md`
itself trimmed (summarise + truncate if it grows large).
2. **Reclaim abandoned food** — scan `pit/claimed/`. A claim file (`<snake-id>.food-*`) whose mtime
is older than the **abandonment timeout (~15 min)** means that snake choked or died mid-digest.
Move it back to `pit/food/` (strip the `<snake-id>.` prefix) so a healthy snake re-devours it, and
note the reclaim in `pit/scraps/reclaims.md`. Use mtime to judge age:
`find pit/claimed -type f -mmin +15`.
3. **Tidy** — prune empty/stale scraps, and if `pit/done/` grows large, move finished results into
`pit/done/archive/`. Don't touch `pit/food/` items that are fresh, and never delete a result.
You are conservative: when unsure whether something is truly abandoned or just slow, leave it and
re-check next pass. Better a late reclaim than stealing food from a snake that's still digesting.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every nap at 10 minutes** (never a single ScheduleWakeup > 600 s).
- **Declare every nap.** FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min
out; `date -u -d '+10 min' +%FT%TZ`). Idle past it → the watchdog reboots you; your state is the
pit on disk.
- **Compact proactively** at ≳80% context.
Begin: read `pit/README.md`, then enter your cleanup loop.
+34
View File
@@ -0,0 +1,34 @@
You are the **planner** snake — a specialist species. The worker snakes digest small, self-contained
food; you exist for the food too big to swallow whole. Your whole job is the thread's key insight:
**regurgitation IS task decomposition.** You swallow a big task and regurgitate it as a set of
smaller food items the worker snakes can each digest in a sitting.
Read `pit/README.md` for the layout, the food schema, and the eating protocol. You coordinate ONLY
through the pit; you claim by atomic `mv`.
**Self-paced loop** (`/loop`, no interval). Each iteration:
1. **Find big food** — scan `pit/food/` for items tagged `big: true`, or any food whose `task` is
clearly more than one sitting. Ignore small food — that's the workers' meal.
2. **Devour it** — atomically claim it: `mv pit/food/<f> pit/claimed/planner.<f>`.
3. **Regurgitate in parts** — decompose it into the smallest self-contained food items that still
make sense, each with a real `done-when`. Write them into `pit/food/` as new `food-<id>-<slug>.md`
(use `tossed-by: planner`, and reference the parent id so results can be traced). If sub-tasks
have an order, say so in each food's body ("needs food-0012 done first") — workers respect it.
4. **Record the plan** — write `pit/done/<parent-id>.plan.md` listing the children you tossed and
how they add up to the parent's `done-when`, then remove the parent from `pit/claimed/`.
5. **Excrete** your planning trace to `pit/waste/planner-<ts>.log`.
Keep decomposition shallow and honest: if a "big" task is actually small, just toss it back to
`pit/food/` unchanged for a worker (don't manufacture busywork). If you can't decompose it (genuinely
atomic but huge), note that in `pit/scraps/<id>-needs-keeper.md` and toss it back — the keeper
decides.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every nap at 10 minutes** (never a single ScheduleWakeup > 600 s).
- **Declare every nap.** FINAL output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min
out; `date -u -d '+10 min' +%FT%TZ`). Idle past it → the watchdog reboots you; your state is the
pit on disk.
- **Compact proactively** at ≳80% context.
Begin: read `pit/README.md`, then loop — hunt for big food, decompose, regurgitate.
+39
View File
@@ -0,0 +1,39 @@
You are a 🐍 **snake** in the pit — one worker in a pool of identical snakes. Your snake-id was given
in your startup line (e.g. `snake-2`); use it in every claim and every log. Read `pit/README.md` now
for the pit layout and the eating protocol — it is the source of truth for how to coordinate.
You do not talk to the other snakes. You coordinate ONLY through the pit (the filesystem), and you
claim work by **atomic `mv`** so two snakes never devour the same food.
**Self-paced loop.** Invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup.
Each iteration is one feeding:
1. **Look** in `pit/food/` for food. If it's empty, you're not hungry-out-of-luck — just nap (see
liveness) and check again; the keeper will toss more in.
2. **Devour** — atomically claim ONE item:
`mv pit/food/<f> pit/claimed/<your-id>.<f>`. If the `mv` fails, another snake got it; pick
another. Claim exactly one at a time — don't hoard the pit.
3. **Digest** — do the work the food describes (its `done-when` is your acceptance check). Run it;
don't assume. Keep a running trace as you go.
4. **Regurgitate**
- *whole*: write the finished result to `pit/done/<id>.result.md` (state what you did and how to
verify `done-when` passes), then remove the file from `pit/claimed/`.
- *in parts*: if the task is too big to digest in one sitting, break it into smaller `food-*`
items, toss them into `pit/food/` for other snakes, and say so in your result.
5. **Excrete** — write your working log / debug trace to `pit/waste/<your-id>-<ts>.log` (`ts` from
`date -u +%Y%m%dT%H%M%SZ`). Keep your workspace clean; the cleanup snake handles the waste pile.
**If you choke** (3rd identical failure on one food): stop forcing it. Regurgitate the food back to
`pit/food/` with a short note in `pit/scraps/<id>-stuck.md` explaining where you got stuck, so a
fresh snake or the keeper can take it. Don't thrash.
**LIVENESS PROTOCOL (the watchdog ENFORCES this):**
- **Cap every nap at 10 minutes.** Never a single ScheduleWakeup > 600 s; to wait longer, wake,
re-check the pit, nap again.
- **Declare every nap.** Immediately before going idle, your FINAL output line MUST be exactly
`WAITING-UNTIL: <ISO-8601 UTC>` (≤10 min out, matching your ScheduleWakeup; compute with
`date -u -d '+10 min' +%FT%TZ`). Idle ≥5 min with no current marker, or past the named time → the
watchdog reboots you; you resume cleanly (your state is the pit on disk, not your memory).
- **Compact proactively** at ≳80% context — your state lives in the pit, so compaction is lossless.
Begin: read `pit/README.md`, then enter your feeding loop. If the pit is empty, nap and check again.
+1 -1
View File
@@ -13,7 +13,7 @@
{ {
# Reproducible devShell with the harness runtime deps. The driver itself is pure Python # Reproducible devShell with the harness runtime deps. The driver itself is pure Python
# stdlib (it needs tomllib, so python >= 3.11); the rest is what the agents/watchdog shell # stdlib (it needs tomllib, so python >= 3.11); the rest is what the agents/watchdog shell
# out to. Make the agent CLIs (claude / opencode) available on PATH separately — they are # out to. Make the agent CLIs (claude / codex / opencode) available on PATH separately — they are
# external, non-Nix tools; install them per their own docs, then `nix develop` here. # external, non-Nix tools; install them per their own docs, then `nix develop` here.
devShells = forAllSystems (pkgs: { devShells = forAllSystems (pkgs: {
default = pkgs.mkShell { default = pkgs.mkShell {
Executable
+159
View File
@@ -0,0 +1,159 @@
#!/usr/bin/env python3
"""The one place secrets live on an orchestrator host: a sops+age encrypted store.
WHY: credentials used to sit in plaintext all over the box — passwords baked into git
remote URLs (`https://user:pass@host/...`, which `git remote -v` happily prints), API keys
in .env files, a private key at mode 0644. Anything in a repo is one `git add -A` away from
a push. So: ONE encrypted file, OUTSIDE every git tree, and a helper every project uses.
/secrets/store.yaml sops+age ciphertext (0600) — values our code reads
/secrets/<project>/ PROJECT-SCOPED secrets (0600): everything belonging to one
project lives together, e.g. /secrets/lichen/,
/secrets/b1/, /secrets/notplants-orchestrator/
/secrets/files/ CROSS-PROJECT files (0600) SYMLINKED from the fixed path a
third party insists on: ~/.ssh keys, a systemd
EnvironmentFile, nix authKeyFile, a TLS keypair
~/.config/sops/age/keys.txt the age private key, 0600
PROJECT SECRETS GO IN /secrets/<project>/ (operator, 2026-08-20). If a secret belongs to one
project, it goes in that project's directory — not in files/, and not with the project name
baked into the filename. `/secrets/lichen/test-pds.env`, not `/secrets/files/lichen-test-pds.env`.
Reserve files/ for things genuinely shared across projects.
WHY: a flat directory forces every name to carry its own scope, which nobody does consistently,
and then nobody can answer "what does this project hold?" or "what do I revoke if this project is
compromised?" without grepping. A directory answers both by listing. Put a README.md in the
project directory saying what each file is, what consumes it, and what breaks if it is lost —
the next person to read it will be doing so under time pressure.
The symlink rule is unchanged and applies the same way: a consumer that insists on a fixed path
gets a SYMLINK into /secrets/<project>/, so the file still exists exactly once.
One home per secret: a value is in the store OR a file under /secrets, never both.
/secrets is outside every git tree — not a repo, no remote — and outside /srv, which agents
grep and walk constantly.
USAGE (library):
from secrets import get, get_group
cookie = get("tangled.cookie")
env = get_group("cc_ci_testenv") # dict, e.g. to build an env
USAGE (CLI):
python3 engine/secrets.py list # group/key names only, never values
python3 engine/secrets.py get tangled.cookie # value to stdout (careful in logs)
NO SECOND COPIES. A secret is never written to a second file "so something can read it"
copies drift, get committed, and widen what a stray `grep` or an attacker finds. A consumer
that insists on a path gets a SYMLINK into /secrets/files (see above), so the file still
exists exactly once. For a one-off, inject at run time and leave nothing behind:
secrets.py exec-env <group> -- some-command # group as env vars, no file
secrets.py with-file <group.key> -- cmd -i {} # 0600 file in a private tmpdir,
# deleted when the command exits
Careful with symlinks: an app that rewrites its own credential file (an OAuth refresh writing
auth.json via write-temp+rename) REPLACES the symlink with a regular file and silently splits
the home again. Before symlinking, ask whether the owner ever writes it back.
ADDING A SECRET: sops /secrets/store.yaml (opens decrypted in $EDITOR, re-encrypts on save)
"""
import json, os, subprocess, sys, pathlib, tempfile, shutil
STORE = os.environ.get("AO_SECRETS_STORE", "/secrets/store.yaml")
AGE_KEY = os.environ.get("SOPS_AGE_KEY_FILE", os.path.expanduser("~/.config/sops/age/keys.txt"))
def _sops_bin():
"""Resolve sops. It is on PATH under the systemd unit, but not always in an
interactive shell — fall back to the NixOS system profile before failing."""
return (shutil.which("sops")
or next((p for p in ("/run/current-system/sw/bin/sops",
"/run/wrappers/bin/sops") if os.path.exists(p)), None)
or "sops")
def _load():
"""Decrypt the store. Fails loudly: a silent empty dict would look like 'no secrets'."""
if not pathlib.Path(STORE).exists():
sys.exit(f"no secret store at {STORE} — see engine/README.md (Secrets)")
env = {**os.environ, "SOPS_AGE_KEY_FILE": AGE_KEY}
r = subprocess.run([_sops_bin(), "-d", "--output-type", "json", STORE],
capture_output=True, text=True, env=env)
if r.returncode != 0:
sys.exit(f"cannot decrypt {STORE} (age key at {AGE_KEY}?): {r.stderr.strip()[:300]}")
return json.loads(r.stdout)
def get(dotted, default=None):
"""get('group.key') -> value. Missing key returns default (None) rather than raising."""
cur = _load()
for part in dotted.split("."):
if not isinstance(cur, dict) or part not in cur:
return default
cur = cur[part]
return cur
def get_group(name):
"""get_group('cc_ci_testenv') -> dict of that group (empty dict if absent)."""
return _load().get(name, {})
def exec_env(group, argv):
"""Run argv with `group`'s keys added to the environment. Nothing touches the disk."""
vals = get_group(group)
if not vals:
sys.exit(f"no group {group!r} in the store (secrets.py list)")
env = {**os.environ, **{k: str(v) for k, v in vals.items()}}
return subprocess.call(argv, env=env)
def with_file(dotted, argv):
"""Run argv with `{}` replaced by a 0600 temp file holding the value.
The file lives in a private 0700 dir and is removed when the command exits — so a consumer
that insists on a path (ssh -i, a TLS key) never leaves a lasting second copy of the secret.
"""
val = get(dotted)
if val is None:
sys.exit(f"no such key: {dotted}")
d = tempfile.mkdtemp(prefix="ao-secret-") # mkdtemp is 0700
try:
p = pathlib.Path(d) / dotted.split(".")[-1]
p.write_text(val if isinstance(val, str) else json.dumps(val))
p.chmod(0o600)
return subprocess.call([a.replace("{}", str(p)) for a in argv])
finally:
shutil.rmtree(d, ignore_errors=True)
def main():
argv = sys.argv[1:]
if not argv or argv[0] in ("-h", "--help"):
print(__doc__)
return
cmd = argv[0]
if cmd == "list":
for g, v in _load().items():
print(f"{g}: {', '.join(v) if isinstance(v, dict) else '<value>'}")
elif cmd == "get":
if len(argv) < 2:
sys.exit("usage: secrets.py get <group.key>")
v = get(argv[1])
if v is None:
sys.exit(f"no such key: {argv[1]}")
print(v if isinstance(v, str) else json.dumps(v))
elif cmd in ("exec-env", "with-file"):
if "--" not in argv:
sys.exit(f"usage: secrets.py {cmd} <name> -- <command...>")
i = argv.index("--")
if i != 2:
sys.exit(f"usage: secrets.py {cmd} <name> -- <command...>")
run = exec_env if cmd == "exec-env" else with_file
sys.exit(run(argv[1], argv[i + 1:]))
else:
sys.exit(f"unknown command {cmd!r} — list | get | exec-env | with-file")
if __name__ == "__main__":
main()
+138
View File
@@ -0,0 +1,138 @@
---
name: codeberg-pages
description: >-
Publish a static site to Codeberg Pages, including custom domains on the new
"git-pages" server. Use when deploying a site to Codeberg Pages, setting up or
debugging a codeberg.page / custom-domain deployment, wiring the DNS records
(A/AAAA, CNAME), the _git-pages-repository TXT authorization record, or the
deploy webhook — or when a custom domain serves a TLS error / never gets a
certificate. Covers the 2025→2026 migration off the old Pages Server v2
(.domains file) to git-pages (webhook + TXT authorization).
---
# Publishing a site to Codeberg Pages
Codeberg Pages migrated from the old **Pages Server v2** (automatic deploy,
`.domains` file) to the new **git-pages** server. On git-pages a deployment is
**webhook-triggered** and a custom domain is authorized by a **TXT record**, not
by a file in the repo. If you're following older docs or a `.domains`-based
`deploy.sh`, that's why things silently don't work.
> All values below were verified against the official docs
> (<https://docs.codeberg.org/codeberg-pages/> and `.../using-custom-domain/`).
> Codeberg changes these; re-check the docs if something behaves unexpectedly.
## The mental model
1. Static site content lives on a branch named **`pages`** (per-repo site) — push
your built site there.
2. On git-pages, pushing alone does **not** deploy. A **webhook** on the repo,
pointed at the domain you want, is what triggers a deployment.
3. A custom domain is bound to the repo by a **TXT authorization record**
(`_git-pages-repository.<domain>`) plus the normal A/AAAA/CNAME records.
4. TLS (Let's Encrypt) is issued **only after the first successful deployment**.
Before that, browsers show a TLS error — that is expected, not a bug.
## Basic deploy (no custom domain, `*.codeberg.page`)
- Put the site on a `pages` branch and push it.
- Add a webhook: repo **Settings → Webhooks → Forgejo**, Target URL
`https://<username>.codeberg.page/<repository>/`, **Branch filter: `pages`**.
- (User/org site: name the repo `pages` and use Target URL
`https://<username>.codeberg.page/`.)
## Custom domain setup (git-pages)
Do all four. Missing #2 or #3 is the usual cause of "DNS looks right but the site
won't serve / no certificate."
### 1. DNS: point the domain at Codeberg
Exact values (verify against the docs — Codeberg has changed IPs before):
- **Apex domain** (`example.org`):
- `A``217.197.84.141`
- `AAAA``2a0a:4580:103f:c0de::2`
- **Subdomain** (`www.example.org`, `foo.example.org`):
- `CNAME``codeberg.page.`**note the trailing dot.**
**Trailing-dot trap:** in a zone file / most DNS UIs, a CNAME target *without* a
trailing dot is treated as relative and the zone is appended — e.g. entering
`codeberg.page` (or an old `<user>.codeberg.page`) can resolve to
`codeberg.page.example.org.`, which is broken. Always use the fully-qualified
`codeberg.page.` with the dot. (ALIAS/ANAME works where CNAME isn't allowed, but
conflicts with DNSSEC-signed zones.)
### 2. TXT authorization record (this is how git-pages maps domain → repo)
Create one **per domain** you serve:
```
_git-pages-repository.example.org. TXT "https://codeberg.org/<user>/<repo>.git"
```
- Name: the `_git-pages-repository.` prefix on the exact domain (including each
subdomain you serve — apex and `www` each need their own if both are used).
- Value: the **HTTPS clone URL** of the repo, ending in `.git`.
- (If you deploy via **Forgejo Actions** instead of a webhook, the record is
`_git-pages-forge-allowlist.<domain>` with the same clone-URL value.)
### 3. Deploy webhook (per domain)
Repo **Settings → Webhooks → Forgejo**:
- **Target URL:** the domain itself, and **`http://` (not `https://`) for the
first deployment** — this is documented, not a mistake (the cert doesn't exist
yet). One webhook per domain, e.g. `http://example.org`, `http://foo.example.org`.
- **Branch filter:** `pages`.
- After the first successful deploy and cert issuance, you may switch the Target
URLs to `https://`.
### 4. Trigger the first deploy
**Push to the `pages` branch** (re-run your deploy script / `git push origin pages`).
The push fires the webhook, git-pages pulls and deploys, then requests a
Let's Encrypt certificate.
- **Do NOT rely on the webhook "Test delivery" button** — the official docs say it
fails by design and is not a valid way to verify or trigger a deploy. Verify by
pushing and then checking the webhook's recent-deliveries log, or just load the
site. (This corrects a common misconception that "Test delivery" triggers a deploy.)
## The `.domains` file is obsolete
Under the old Pages Server v2, a `.domains` file in the branch listed the domains
and did apex-vs-alias redirects. On git-pages it is **no longer used** — authorization
comes from the TXT record. It's harmless to leave, but you can delete it (and drop any
`.domains` handling from `deploy.sh`). Bonus: on git-pages each domain gets its **own**
deployment, so a second domain serves the site directly instead of 301-redirecting to
the primary as the old `.domains` system did.
## TLS / certificate notes
- A cert is issued **only after the first successful webhook deployment**. A TLS
error before that is expected.
- If the domain has **CAA records**, they must allow Let's Encrypt (including the
staging issuer) or the cert request is refused.
- Cert still never issues after a successful deploy → confirm the `_git-pages-repository`
TXT value exactly matches the repo's HTTPS `.git` URL, and that the webhook Target
URL matches the domain.
## Quick troubleshooting checklist
- Browser TLS error, no cert → no successful deploy yet. Check webhook deliveries;
push to `pages`; confirm webhook Target URL used `http://` for the first deploy.
- "DNS is correct but site won't serve" → missing `_git-pages-repository` TXT, or
missing/mis-branch-filtered webhook.
- CNAME resolves to `codeberg.page.<yourzone>` → missing trailing dot; set target to
`codeberg.page.`.
- CAA present → ensure Let's Encrypt is allowed.
- Old `.domains` behavior expected (redirects) → gone on git-pages; each domain now
deploys independently.
## Sources
- Codeberg Pages: <https://docs.codeberg.org/codeberg-pages/>
- Using custom domains: <https://docs.codeberg.org/codeberg-pages/using-custom-domain/>
- pages-server (now in maintenance, superseded by git-pages):
<https://codeberg.org/Codeberg/pages-server>
+157
View File
@@ -0,0 +1,157 @@
---
name: gateway-domain
description: Give a tailnet box a real public HTTPS domain (<name>.gtest.commoninternet.net) by mapping it on the shared testing gateway. Use when an agent needs a publicly reachable URL for a box with no public IP — an OAuth callback, a webhook receiver, a demo link, an ACME challenge. Covers the add/remove tool, where the admin password lives, and the two things that silently break it: your box must carry a tailnet tag the ACL allows (tag:notplants-test-server or tag:orchestrator) or the gateway cannot reach it, and your box serves the TLS cert rather than the gateway.
---
# Giving your box a public domain
Your machine is on the tailnet with no public IP. You need a real HTTPS URL for it. The
**testing gateway** already owns a wildcard DNS record, so every name under
`*.gtest.commoninternet.net` resolves to it. Map your name to your tailnet IP and it forwards
matching traffic to you.
```bash
python3 engine/tools/gateway-domain.py add myapp
# myapp.gtest.commoninternet.net -> 100.84.190.30
```
That is the whole happy path. The backend defaults to **this box's own tailscale IP**, so run
it on the machine that will serve the domain.
## Your box needs a tailnet tag the ACL allows
The gateway can only open connections to nodes the tailnet ACL lets it reach. Two tags qualify:
| tag | who |
|---|---|
| `tag:notplants-test-server` | test servers — the usual case |
| `tag:orchestrator` | orchestrator boxes (added 2026-08-20, verified end to end) |
Without one of them the gateway accepts your mapping and then simply never connects — which
looks like a broken gateway and is not one. Check before you start:
```bash
tailscale status --json | jq -r '.Self.Tags[]?'
```
If neither tag is listed, add one to that node in the Tailscale admin (a node's tags are set
when it is authenticated, so this may mean re-authenticating it), or map a backend that already
has one. `gateway-domain.py` warns when the node it is about to map carries neither, but it
cannot see the tags of a backend you name explicitly — that one is on you.
The gateway itself is tagged `tag:testing-gateway`; that is the other half of the same ACL rule.
## The commands
```bash
python3 engine/tools/gateway-domain.py list
python3 engine/tools/gateway-domain.py add myapp # this box, port 443
python3 engine/tools/gateway-domain.py add myapp 100.64.1.5 # another box
python3 engine/tools/gateway-domain.py add myapp 100.64.1.5:8443 # backend not on 443
python3 engine/tools/gateway-domain.py remove myapp
```
A bare label is expanded to `<label>.gtest.commoninternet.net`. Anything containing a dot is
used verbatim, so you can map a domain you control elsewhere — but then **you** must point its
DNS at the gateway (`49.13.156.72`); only `*.gtest.commoninternet.net` is pre-pointed.
## The credential
The admin password is in the orchestrator secret store, **not** in any repo:
```bash
python3 engine/secrets.py get gateway.admin_password
```
| | |
|---|---|
| store | `/secrets/store.yaml` (sops+age, `0600`, outside every git tree) |
| keys | `gateway.admin_password`, `gateway.fqdn`, `gateway.admin_user` |
| age key | `~/.config/sops/age/keys.txt` |
| add/edit | `sops /secrets/store.yaml` |
`gateway-domain.py` reads it itself — you should never need to handle the value. Do not copy it
into a config file, an env file, or a repo. If you need it in a subprocess, use
`engine/secrets.py exec-env` or `with-file` rather than writing a second copy.
## The one thing that surprises people: your box serves the certificate
The gateway does **not** terminate TLS. It reads the SNI name from the TLS handshake and proxies
the still-encrypted bytes onward:
```
browser --TLS--> gateway :443 --reads SNI, proxies encrypted--> your box (tailnet)
```
So after mapping `myapp.gtest.commoninternet.net`, **your box** must serve a certificate valid
for that exact name, on the backend port (443 unless you set one). The gateway has a cert for
its own name only; it never sees your plaintext.
Getting a cert on your box works: the gateway passes HTTP-01 challenges through on `:80` for
mapped names, so ACME can complete normally. Add the domain here **first**, then request the
cert — issuance needs the mapping to already exist.
If you only need plain HTTP for a quick test, that also passes through on `:80`.
## Traps
**Backends must be a literal IPv4 address** (optionally `IP:port`). The tool refuses hostnames,
and it is protecting you: the gateway's `validate_ip` accepts a hostname, but its
`put_domain`/`remove_domain` only match lines whose backend is numeric. A hostname mapping can
be written once and then **never updated or removed** through the admin UI — it becomes an
orphan that only a hand-edit of `/var/lib/tunnel-gateway/tunnel_map.conf` on the box can clear.
**Only ports 22/80/443 are open at the edge.** Mapping `IP:8443` changes which port on *your
box* the gateway connects to; it does not open 8443 to the internet. Exposing a different
*gateway* port is a config change in `nix/hosts/hetzner-test.nix`
(`services.tunnelGateway.openTCPPorts`) **and** the Hetzner firewall in `nix/terraform/` — not
something this tool can do.
**Names are shared.** Anyone with the password can list, overwrite, or delete any mapping.
Prefix yours with something recognisable and remove it when you are finished.
**This is the test gateway.** Never point this tool at the production one. The box is long-lived
(it is also the e2e and CI target), but its map is not sacred.
## When the password stops working
The e2e suite reseeds `.htpasswd` with a throwaway credential for the duration of a run and
restores the previous file afterwards. If a run was killed part-way, the real one is still on
the box at `/var/lib/tunnel-gateway/.htpasswd.e2e-backup`:
```bash
ssh -i /srv/gateway-coop/.secrets/id_admin root@49.13.156.72 \
'mv -f /var/lib/tunnel-gateway/.htpasswd.e2e-backup /var/lib/tunnel-gateway/.htpasswd'
```
To reseed it from the store instead — piping so the value never lands on disk:
```bash
python3 engine/secrets.py get gateway.admin_password | \
ssh -i /srv/gateway-coop/.secrets/id_admin root@49.13.156.72 \
'htpasswd -ci /var/lib/tunnel-gateway/.htpasswd admin \
&& chgrp nginx /var/lib/tunnel-gateway/.htpasswd \
&& chmod 640 /var/lib/tunnel-gateway/.htpasswd'
```
## If it still does not work
Check in this order — most failures are the last two.
1. `gateway-domain.py list` — is the mapping actually there?
2. `getent hosts myapp.gtest.commoninternet.net` — should be `49.13.156.72`.
3. **Does your node carry `tag:notplants-test-server` or `tag:orchestrator`?**
(`tailscale status --json | jq -r '.Self.Tags[]?'`) This is the single most common cause.
The gateway is `gateway-test-1` (`100.91.44.90`), tagged `tag:testing-gateway`; the ACL
pairs that with the tags above, so a backend with neither is unreachable no matter how
correct the mapping looks. Quick check from the gateway itself:
`ssh root@49.13.156.72 'timeout 5 bash -c "echo > /dev/tcp/<your-tailnet-ip>/<port>"'`
4. **Is your service actually serving TLS for that name on the backend port?** A backend that
speaks plain HTTP on 443, or serves a cert for a different name, fails here and nowhere else.
## The gateway itself
`gtest.commoninternet.net` / `49.13.156.72` — a Hetzner `cx23` running NixOS, configured in the
`tunnel-gateway-server` repo (`nix/hosts/hetzner-test.nix`), which `notplants-nix` pins as a
submodule under `external/`. It also accepts reverse-SSH tunnels, for boxes not on the tailnet;
that path is separate from this one and is not covered here.
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""Post a top-level comment on a Tangled pull as the bot.
WHY: tangled_comments.py only READS. Replying to the operator's review notes on a
~130-pull stack by hand is not viable, and there is no client API — the appview
takes an htmx form POST /comment with three fields:
subject-uri at://{did}/sh.tangled.repo.pull/{rkey} (the pull record)
pull-round-idx which round the comment hangs off
body markdown
Both hidden fields are only discoverable from the pull page, so this fetches the
page, scrapes them, and posts. Same session-cookie auth as tangled_pr.py.
USAGE:
tangled_comment_post.py --owner notplants-bot.bsky.social --repo lichen.page.review \
--pull 76 --body "..." # or --body-file reply.md
tangled_comment_post.py ... --pull 76 --round 1 --body "..." # pin to a round
tangled_comment_post.py ... --pull 76 --body "..." --dry-run
"""
import argparse, os, re, sys, urllib.error, urllib.parse, urllib.request
BASE = "https://tangled.org"
def load_cookie():
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
return c
def get(url, cookie):
req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
return urllib.request.urlopen(req, timeout=60).read().decode()
def scrape_form(doc):
"""The comment form's two hidden fields. Several rounds' forms can be on the page;
take the last (highest round), which is the one the UI shows open."""
uris = re.findall(r'name="subject-uri"[^>]*value="([^"]+)"', doc)
idxs = re.findall(r'name="pull-round-idx"[^>]*value="([^"]+)"', doc)
if not uris or not idxs:
# Two causes, and naming only one sends the reader in the wrong direction: a
# reviewer hitting this on a phantom pull will re-auth, succeed, and still fail.
sys.exit(
"could not find the comment form. Two causes, in likelihood order:\n"
" 1. THE PULL DOES NOT EXIST — a 404 page renders 200 and carries no form.\n"
" Check the branch on the remote, not the pull number.\n"
" 2. the cookie has expired."
)
return uris[-1], idxs[-1]
def main():
ap = argparse.ArgumentParser(description="post a comment on a Tangled pull")
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pull", required=True, type=int)
ap.add_argument("--round", type=int, default=None, help="round to attach to (default: latest)")
ap.add_argument("--body")
ap.add_argument("--body-file")
ap.add_argument("--dry-run", action="store_true")
a = ap.parse_args()
if not a.body and not a.body_file:
sys.exit("need --body or --body-file")
body = a.body if a.body else open(a.body_file).read()
cookie = load_cookie()
page = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
if a.round is not None:
page += f"/round/{a.round}"
subject_uri, round_idx = scrape_form(get(page, cookie))
if a.dry_run:
print(f"would post to {page}\n subject-uri={subject_uri}\n round={round_idx}\n---\n{body}")
return
data = urllib.parse.urlencode(
{"subject-uri": subject_uri, "pull-round-idx": round_idx, "body": body}
).encode()
req = urllib.request.Request(
f"{BASE}/comment",
data=data,
headers={
"Cookie": cookie,
"User-Agent": "tangled-pr-bot",
"Content-Type": "application/x-www-form-urlencoded",
"HX-Request": "true",
"Referer": page,
},
)
try:
resp = urllib.request.urlopen(req, timeout=60)
except urllib.error.HTTPError as e:
sys.exit(f"POST /comment failed: {e.code} {e.read().decode()[:400]}")
print(f"posted on pull #{a.pull} (round {round_idx}) — HTTP {resp.status}")
if __name__ == "__main__":
main()
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""Read the review comments on a Tangled pull (top-level discussion comments).
WHY: the appview has no read API for pull comments, and eyeballing the pull page
HTML per comment is slow. This fetches the pull (or a specific round) and prints
every top-level comment as author / time / body — so acting on operator review
notes is one command, not a WebFetch guess.
Same session-cookie auth as tangled_pr.py (cookie from the encrypted store,
tangled.cookie). Read-only: it never posts.
USAGE:
tangled_comments.py --owner notplants-bot.bsky.social --repo lichen.page.review --pull 75
tangled_comments.py ... --pull 75 --round 2 # a specific round's page
tangled_comments.py ... --pull 75 --json # machine-readable
"""
import argparse, html, json, os, re, sys, urllib.request
BASE = "https://tangled.org"
def load_cookie():
"""Cookie from the encrypted store (tangled.cookie) — see engine/README.md (Secrets)."""
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
return c
def fetch(url, cookie):
req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
return urllib.request.urlopen(req, timeout=60).read().decode()
def _text(fragment):
"""Strip tags from a body fragment down to readable plaintext."""
t = re.sub(r"(?is)<br\s*/?>", "\n", fragment)
t = re.sub(r"(?is)</p\s*>", "\n\n", t)
t = re.sub(r"(?is)<li[^>]*>", "\n- ", t)
t = re.sub(r"(?s)<[^>]+>", "", t)
return html.unescape(t).strip()
def parse_comments(doc):
"""Return [{cid, author, when, iso, uri, body}] in document order.
Anchored on the per-comment header block the appview emits:
id="comment-header-{cid}" ... <a href="/{handle}">{handle}</a> ...
<time datetime="{iso}">{when}</time> ... class="...comment-body"><div class="prose...">{body}</div>
"""
out = []
heads = list(re.finditer(r'id="comment-header-([0-9a-z]+)"', doc))
for i, hm in enumerate(heads):
cid = hm.group(1)
# bound by the next comment header, not a fixed window: a comment's own
# reaction/button markup can push its body several KB past the header.
end = heads[i + 1].start() if i + 1 < len(heads) else len(doc)
seg = doc[hm.end(): end]
am = re.search(r'href="/([^"/]+)"[^>]*>\s*([^<]+?)\s*</a>', seg)
author = html.unescape(am.group(2)) if am else "?"
tm = re.search(r'<time datetime="([^"]+)"[^>]*>\s*([^<]+?)\s*</time>', seg)
iso = html.unescape(tm.group(1)) if tm else ""
when = html.unescape(tm.group(2)) if tm else ""
bm = re.search(r'comment-body">\s*<div class="prose[^"]*">(.*?)</div>\s*<div class="reactions', seg, re.S)
if not bm:
bm = re.search(r'comment-body">\s*<div class="prose[^"]*">(.*?)</div>', seg, re.S)
body = _text(bm.group(1)) if bm else ""
out.append({"cid": cid, "author": author, "when": when, "iso": iso,
"uri": f"at://.../sh.tangled.feed.comment/{cid}", "body": body})
return out
def main():
ap = argparse.ArgumentParser(description="read a Tangled pull's review comments")
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pull", required=True, type=int)
ap.add_argument("--round", type=int, default=None, help="a specific round page (default: latest)")
ap.add_argument("--json", action="store_true")
a = ap.parse_args()
cookie = load_cookie()
url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
if a.round is not None:
url += f"/round/{a.round}"
doc = fetch(url, cookie)
# A 404 page renders 200 and carries no pull record, so "0 comments" used to be
# indistinguishable from "no such pull". On 2026-08-22 that cost a false report that
# #428 was filed and unreviewed, and a reviewer's time chasing it. The post tool never
# had the defect because it MUST resolve a subject-uri to work at all — so require the
# same identifier here, and refuse rather than print a header for a phantom.
if not re.search(r'name="subject-uri"[^>]*value="(at://[^"]+)"', doc):
sys.exit(
f"pull #{a.pull} does not exist in {a.owner}/{a.repo} "
f"(no pull record on {url}).\n"
" This is NOT an auth failure: the page rendered, it simply carries no pull.\n"
" A pull that exists always yields a subject-uri."
)
comments = parse_comments(doc)
if a.json:
print(json.dumps({"url": url, "count": len(comments), "comments": comments}, indent=2))
return
print(f"# pull #{a.pull} ({url}) — {len(comments)} comment(s)\n")
for i, c in enumerate(comments, 1):
print(f"[{i}] {c['author']} · {c['when']} ({c['iso']}) #{c['cid']}")
for line in (c["body"] or "(empty)").splitlines():
print(f" {line}")
print()
if __name__ == "__main__":
main()
Executable
+216
View File
@@ -0,0 +1,216 @@
#!/usr/bin/env python3
"""File a Tangled pull request as the bot, by reusing a browser session cookie.
WHY: Tangled's appview only indexes a pull when it's created through its
OAuth-authenticated web endpoint (POST /{owner}/{repo}/pulls/), which fetches the
patch from the knot and inserts it into the appview DB directly. Writing the
sh.tangled.repo.pull record straight to the PDS (com.atproto.repo.createRecord)
does NOT get indexed (verified: even a byte-identical knot patch fails). There is
no client CLI. So the automation is: log into tangled.org ONCE in a browser as the
bot, copy the session cookie here (gitignored), and this tool reuses it.
The endpoint needs no CSRF token and no patch upload — just the session cookie and
the branch names; the appview generates the patch (from the knot) and indexes it.
Branch-based PR requires push access to the repo (the bot owns its fork, so OK).
COOKIE FILE (tools/.tangled-session, gitignored), a single line — paste the whole
Cookie header value from the logged-in browser (both appview-* cookies):
TANGLED_COOKIE=appview-session-v2=<...>; appview-accounts-v2=<...>
RELIABILITY (rewritten 2026-08-21). This tool used to judge success ONLY by an
HX-Redirect header on the POST response. A create that SUCCEEDED but answered without
that header read as a failure, the caller retried, and Tangled grew duplicate pulls —
that is exactly how #397, #398 and #399 were created for one branch. A response header
describes what the server meant to say; it is not the artifact. So now:
* BEFORE posting, look for an existing OPEN pull for this source branch. If one
exists, refuse and name it — a duplicate cannot be created even if a caller retries.
* AFTER posting, confirm against the PULLS LIST, not the response: a new pull number
that did not exist before and whose page names this source branch IS the success,
with or without a redirect header.
* Only report failure when no such pull appeared. A false failure is worse than a
loud error here, because the caller's remedy is to retry.
USAGE:
tangled_pr.py --owner notplants-bot.bsky.social --repo lichen.page \
--target main --source hardening-review \
[--fork did:plc:<forkRepoDid>] [--title "..."] [--body "..."] [--dry-run]
"""
import argparse, os, sys, urllib.request, urllib.parse, urllib.error
BASE = "https://tangled.org"
def load_cookie(path=None):
"""The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md (Secrets).
`path` is a legacy escape hatch: a TANGLED_COOKIE=... file, used only if explicitly passed."""
if path:
for line in open(path):
if line.strip().startswith("TANGLED_COOKIE="):
return line.strip()[len("TANGLED_COOKIE="):]
sys.exit(f"{path} has no TANGLED_COOKIE= line")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
return c
# ── verification against the artifact, not the response ─────────────────────────
# These are pure so they can be unit-tested without a network: see tests/test_tools.py.
import re
def pull_numbers(html):
"""Every pull number linked from a pulls index page."""
return {int(n) for n in re.findall(r'/pulls/(\d+)\b', html or "")}
def page_names_branch(html, branch):
"""True if a pull's page mentions this source branch.
Substring rather than token: Tangled renders the branch inside markup we do not
control, and a false NEGATIVE here would resurrect the duplicate bug. A false
positive is caught by the caller, which only asks about pulls it just created.
"""
return bool(branch) and branch in (html or "")
def classify(before, after, branch_pages):
"""Decide what a POST did, from the pulls list before and after it.
before/after: sets of pull numbers. branch_pages: {number: names_our_branch}.
Returns (verdict, pull_number) where verdict is:
"created" a new pull naming our branch appeared
"unrelated" new pulls appeared but none is ours (someone else was filing)
"none" nothing new appeared — a real failure
"""
fresh = sorted(after - before)
ours = [n for n in fresh if branch_pages.get(n)]
if ours:
return "created", max(ours)
if fresh:
return "unrelated", None
return "none", None
def fetch(url, cookie, timeout=30):
"""GET a page as the bot. Returns "" on any error: callers treat an unreadable page as
'cannot confirm', never as 'confirmed absent' — the difference is the duplicate bug."""
try:
r = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
return urllib.request.urlopen(r, timeout=timeout).read().decode("utf-8", "replace")
except Exception:
return ""
def index_pulls(owner, repo, cookie, _fetch=fetch):
return pull_numbers(_fetch(f"{BASE}/{owner}/{repo}/pulls", cookie))
def pull_names_branch(owner, repo, n, branch, cookie, _fetch=fetch):
return page_names_branch(_fetch(f"{BASE}/{owner}/{repo}/pulls/{n}", cookie), branch)
def existing_pull_for(owner, repo, branch, cookie, numbers, scan=12, _fetch=fetch):
"""The newest pull whose page names this branch, scanning back `scan` pulls.
Bounded on purpose: an unbounded scan would fetch hundreds of pages on every call.
Scanning the newest N is enough for the duplicate this guards against, which is a
retry seconds after the original. Returns None if none is found within the window —
which is 'not seen', not 'does not exist', and the post-check is the real backstop.
"""
for n in sorted(numbers, reverse=True)[:scan]:
if pull_names_branch(owner, repo, n, branch, cookie, _fetch):
return n
return None
def main():
ap = argparse.ArgumentParser(description="file a Tangled PR via a reused session cookie")
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--target", required=True, help="target branch (merge into), e.g. main")
ap.add_argument("--source", required=True, help="source branch (the changes), e.g. hardening-review")
ap.add_argument("--fork", default="", help="fork repoDid for a cross-fork PR (omit for same-repo)")
ap.add_argument("--title", default="")
ap.add_argument("--body", default="")
ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
ap.add_argument("--dry-run", action="store_true", help="check for an existing pull and show the request; do not POST")
ap.add_argument("--allow-duplicate", action="store_true", help="post even if a pull for this branch already exists")
a = ap.parse_args()
cookie = load_cookie(a.cookie_file)
# Tangled's PR form is htmx: it POSTs to /pulls/new (NOT /pulls/, which is 405) and only processes
# the create when it sees the HX-Request header — otherwise it just re-renders the page (a 200 that
# creates nothing). Success is signalled by an HX-Redirect header pointing at the new pull.
# PRE-CHECK: refuse to create a second pull for a branch that already has one. This is what
# makes a retry harmless rather than duplicating — the failure mode that produced #397-#399.
before = index_pulls(a.owner, a.repo, cookie)
if not before:
print("WARNING: could not read the pulls index — cannot check for an existing pull, and "
"cannot confirm the result afterwards. Proceeding, but verify by hand.")
dup = existing_pull_for(a.owner, a.repo, a.source, cookie, before) if before else None
if dup and not a.allow_duplicate:
print(f"REFUSING: #{dup} already exists for source branch {a.source!r}.")
print(f" {BASE}/{a.owner}/{a.repo}/pulls/{dup}")
print(" To push new commits to an existing pull use tangled_pr_resubmit.py.")
print(" To file anyway (rarely right): --allow-duplicate")
sys.exit(3)
new_url = f"{BASE}/{a.owner}/{a.repo}/pulls/new"
form = {
"source": "branch", # branch-compare mode (each PR targets the branch below it)
"targetBranch": a.target,
"sourceBranch": a.source,
"title": a.title, "titleDirty": "true",
"body": a.body, "bodyDirty": "true",
}
if a.fork: form["fork"] = a.fork
data = urllib.parse.urlencode(form).encode()
req = urllib.request.Request(new_url, data=data, method="POST", headers={
"Cookie": cookie,
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "tangled-pr-bot",
"HX-Request": "true",
"HX-Current-URL": new_url,
"Referer": new_url,
})
if a.dry_run:
print(f"DRY RUN: would POST {new_url}")
print(f" target={a.target!r} source={a.source!r} title={a.title[:60]!r}")
print(f" existing pull for this branch: {dup if dup else 'none found in the scan window'}")
return
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *args, **kw): # capture the redirect instead of following it
return None
opener = urllib.request.build_opener(NoRedirect)
try:
r = opener.open(req, timeout=90)
hdrs, code, body = r.headers, r.getcode(), r.read(3000).decode("utf-8", "replace")
except urllib.error.HTTPError as e:
hdrs, code, body = e.headers, e.code, e.read(3000).decode("utf-8", "replace")
# htmx success is signalled by HX-Redirect (…/pulls/<n>), not a normal 3xx Location.
target = hdrs.get("HX-Redirect", "") or hdrs.get("HX-Location", "") or hdrs.get("Location", "")
print(f"HTTP {code}" + (f" -> {target}" if target else ""))
if "/login" in target or "oauth" in target.lower():
sys.exit("AUTH FAILED: session cookie expired/invalid — refresh the cookie in the secret "
"store (scripts/get-tangled-cookie.py)")
# POST-CHECK: the artifact decides, not the header. A create that answered without a redirect
# is still a create; reporting it as a failure is what made callers retry into duplicates.
after = index_pulls(a.owner, a.repo, cookie)
branch_pages = {n: pull_names_branch(a.owner, a.repo, n, a.source, cookie)
for n in sorted(after - before, reverse=True)[:12]}
verdict, num = classify(before, after, branch_pages)
if verdict == "created":
print(f"OK: pull #{num} created -> {BASE}/{a.owner}/{a.repo}/pulls/{num}")
if not target:
print(" (no redirect header — confirmed against the pulls list instead)")
return
if verdict == "unrelated":
print(" new pulls appeared but none names this source branch — not ours.")
snippet = " ".join(body.split())[:600]
print("FAILED: no pull for this branch appeared. Response snippet:")
print(" " + snippet)
sys.exit(2)
if __name__ == "__main__":
main()
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Close an existing Tangled pull as the bot, via the session cookie.
WHY: engine/ could create, edit, merge, resubmit and comment on a pull, but not CLOSE one. The
appview exposes close as an htmx POST (the "Close" button on the pull page):
POST /{owner}/{repo}/pulls/{n}/close -> closes (hx-swap; success is a 2xx)
Same session-cookie auth as tangled_pr_edit.py. Closing does not create a round and does not
touch the patch. This tool trusts nothing: after the POST it re-fetches the pull page and
confirms the state badge reads Closed. Use --comment-file to post one comment FIRST naming the
successor (the phase-8 rule: every closed pull says which readable PR carries its work).
USAGE:
tangled_pr_close.py --owner notplants-bot.bsky.social --repo lichen.page.review --pull 397 \
[--comment-file note.md] [--dry-run]
"""
import argparse, os, re, subprocess, sys, urllib.error, urllib.parse, urllib.request
BASE = "https://tangled.org"
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
def load_cookie():
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — refresh with scripts/get-tangled-cookie.py")
return c
def state_of(owner, repo, pull, cookie):
import tangled_comments as tc
doc = tc.fetch(f"{BASE}/{owner}/{repo}/pulls/{pull}", cookie)
badges = set(re.findall(r'>\s*(Merged|Closed|Open)\s*<', doc))
return badges
def main():
ap = argparse.ArgumentParser(description="close a Tangled pull via a reused session cookie")
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pull", required=True, type=int)
ap.add_argument("--comment-file", default=None, help="post this comment before closing")
ap.add_argument("--dry-run", action="store_true", help="show state and the request; do not POST")
a = ap.parse_args()
cookie = load_cookie()
before = state_of(a.owner, a.repo, a.pull, cookie)
print(f"#{a.pull} state before: {sorted(before) or 'unknown'}")
if "Merged" in before:
sys.exit(f"#{a.pull} is MERGED — refusing to close a merged pull")
if "Closed" in before and "Open" not in before:
print(f"#{a.pull} is already Closed; nothing to do")
return
if a.dry_run:
print(f"DRY RUN: would POST {BASE}/{a.owner}/{a.repo}/pulls/{a.pull}/close")
return
if a.comment_file:
r = subprocess.run([sys.executable, os.path.join(HERE, "tangled_comment_post.py"),
"--owner", a.owner, "--repo", a.repo, "--pull", str(a.pull),
"--body-file", a.comment_file], capture_output=True, text=True)
print(r.stdout.strip())
if r.returncode != 0:
sys.exit(f"comment failed, NOT closing: {r.stderr.strip()[:300]}")
close_url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}/close"
page_url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
req = urllib.request.Request(close_url, data=b"", method="POST", headers={
"Cookie": cookie,
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "tangled-pr-bot",
"HX-Request": "true",
"HX-Current-URL": page_url,
"Referer": page_url,
})
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *args, **kw):
return None
opener = urllib.request.build_opener(NoRedirect)
try:
r = opener.open(req, timeout=90)
hdrs, code, resp = r.headers, r.getcode(), r.read(3000).decode("utf-8", "replace")
except urllib.error.HTTPError as e:
hdrs, code, resp = e.headers, e.code, e.read(3000).decode("utf-8", "replace")
target = hdrs.get("HX-Redirect", "") or hdrs.get("HX-Location", "") or hdrs.get("Location", "")
print(f"HTTP {code}" + (f" -> {target}" if target else ""))
if "/login" in target or "oauth" in target.lower():
sys.exit("AUTH FAILED: session cookie expired — refresh with scripts/get-tangled-cookie.py")
if code // 100 != 2:
print(" close did not return 2xx — response snippet:")
print(" " + " ".join(resp.split())[:600])
sys.exit(2)
after = state_of(a.owner, a.repo, a.pull, cookie)
print(f"#{a.pull} state after: {sorted(after) or 'unknown'}")
if "Closed" not in after:
sys.exit("POST returned 2xx but the pull page does not read Closed — check by hand")
print(f"#{a.pull} CLOSED")
if __name__ == "__main__":
main()
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Edit an existing Tangled pull's title/body as the bot, via the session cookie.
WHY: engine/tangled_pr.py only CREATES pulls; the appview also exposes an edit
endpoint for an existing pull (the pencil on the pull page). It is an htmx form:
GET /{owner}/{repo}/pulls/{n}/edit -> the form (current title + body)
POST /{owner}/{repo}/pulls/{n}/edit -> apply (form fields: title, body)
Same session-cookie auth as tangled_pr.py. Editing title/body does NOT create a
round and does not touch the patch — but callers should re-fetch and verify that
themselves (see --show).
COOKIE FILE (engine/.tangled-session, gitignored) — same as tangled_pr.py; on
401/login redirect refresh it with scripts/get-tangled-cookie.py.
USAGE:
tangled_pr_edit.py --owner notplants-bot.bsky.social --repo lichen.page.review \
--pull 75 --show # print current title + body (raw md)
tangled_pr_edit.py ... --pull 75 --title "..." --body "..." # apply edit
tangled_pr_edit.py ... --pull 75 --title "..." --body-file b.md # body from file
"""
import argparse, html, os, re, sys, urllib.request, urllib.parse, urllib.error
BASE = "https://tangled.org"
def load_cookie(path=None):
"""The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md (Secrets).
`path` is a legacy escape hatch: a TANGLED_COOKIE=... file, used only if explicitly passed."""
if path:
for line in open(path):
if line.strip().startswith("TANGLED_COOKIE="):
return line.strip()[len("TANGLED_COOKIE="):]
sys.exit(f"{path} has no TANGLED_COOKIE= line")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
return c
def fetch_form(edit_url, cookie):
"""GET the htmx edit form; return (title, body) as the appview holds them."""
req = urllib.request.Request(edit_url, headers={
"Cookie": cookie, "User-Agent": "tangled-pr-bot",
"HX-Request": "true", "HX-Current-URL": edit_url, "Referer": edit_url,
})
doc = urllib.request.urlopen(req, timeout=60).read().decode()
tm = re.search(r'name="title" id="title"[^>]*value="([^"]*)"', doc)
bm = re.search(r'<textarea\s+name="body".*?>\n?(.*?)</textarea>', doc, re.S)
if not tm or not bm:
sys.exit("could not parse the edit form (auth expired? layout changed?)")
return html.unescape(tm.group(1)), html.unescape(bm.group(1))
def main():
ap = argparse.ArgumentParser(description="edit a Tangled pull's title/body via a reused session cookie")
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pull", required=True, type=int)
ap.add_argument("--title", default=None)
ap.add_argument("--body", default=None)
ap.add_argument("--body-file", default=None, help="read the new body from a file (overrides --body)")
ap.add_argument("--show", action="store_true", help="print current title+body and exit (no edit)")
ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
a = ap.parse_args()
cookie = load_cookie(a.cookie_file)
edit_url = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}/edit"
if a.show:
title, body = fetch_form(edit_url, cookie)
print(f"TITLE: {title}")
print("BODY:")
print(body)
return
body = open(a.body_file).read() if a.body_file else a.body
if a.title is None or body is None:
sys.exit("need --title and --body/--body-file (or --show)")
# the edit form is htmx like the create form: needs HX-Request or the POST
# just re-renders; success is a 2xx (hx-swap=none), failure a login redirect
data = urllib.parse.urlencode({"title": a.title, "body": body}).encode()
req = urllib.request.Request(edit_url, data=data, method="POST", headers={
"Cookie": cookie,
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "tangled-pr-bot",
"HX-Request": "true",
"HX-Current-URL": edit_url,
"Referer": edit_url,
})
class NoRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, *args, **kw):
return None
opener = urllib.request.build_opener(NoRedirect)
try:
r = opener.open(req, timeout=90)
hdrs, code, resp = r.headers, r.getcode(), r.read(3000).decode("utf-8", "replace")
except urllib.error.HTTPError as e:
hdrs, code, resp = e.headers, e.code, e.read(3000).decode("utf-8", "replace")
target = hdrs.get("HX-Redirect", "") or hdrs.get("HX-Location", "") or hdrs.get("Location", "")
print(f"HTTP {code}" + (f" -> {target}" if target else ""))
if "/login" in target or "oauth" in target.lower():
sys.exit("AUTH FAILED: session cookie expired/invalid — refresh engine/.tangled-session "
"(scripts/get-tangled-cookie.py)")
if code // 100 != 2:
snippet = " ".join(resp.split())[:600]
print(" edit did not return 2xx — response snippet:")
print(" " + snippet)
sys.exit(2)
# trust nothing: re-fetch the form and confirm the appview now holds the new text
new_title, new_body = fetch_form(edit_url, cookie)
if new_title == a.title and new_body.replace("\r\n", "\n") == body.replace("\r\n", "\n"):
print("OK: verified — appview now holds the new title/body")
else:
print(f"MISMATCH after edit: title_ok={new_title == a.title} "
f"body_ok={new_body.replace(chr(13)+chr(10), chr(10)) == body.replace(chr(13)+chr(10), chr(10))}")
sys.exit(3)
if __name__ == "__main__":
main()
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
"""Merge or close a Tangled pull as the bot, with a mergeability pre-check.
WHY: `main` advances only by operator-approved merges of the stacked pulls, and the
stack must be merged strictly bottom-up. The appview exposes both actions as bare
htmx POSTs with no body:
POST /{owner}/{repo}/pulls/{n}/merge
POST /{owner}/{repo}/pulls/{n}/close
Neither tells you up front whether the pull is mergeable — that lives in a separate
fragment, GET /{owner}/{repo}/pulls/{n}/round/{r}/actions, which reports "No conflicts,
ready to merge" / "patch is empty" / a conflict. This checks that fragment first and
refuses to merge anything it does not understand, so a bad rung stops the frontier
instead of landing broken.
Note: a pull whose fragment says "patch is empty" (a branch identical to its parent —
e.g. an index slot whose content moved out of the repo) CANNOT be merged; close it.
USAGE:
tangled_pr_merge.py --owner X --repo Y --pull 76 --check
tangled_pr_merge.py --owner X --repo Y --pull 76 --merge
tangled_pr_merge.py --owner X --repo Y --pull 75 --close --reason "empty after restack"
"""
import argparse, os, re, sys, urllib.error, urllib.request
BASE = "https://tangled.org"
def load_cookie():
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store")
return c
def get(url, cookie):
req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
return urllib.request.urlopen(req, timeout=60).read().decode()
def post(url, cookie, referer):
req = urllib.request.Request(
url,
data=b"",
headers={
"Cookie": cookie,
"User-Agent": "tangled-pr-bot",
"HX-Request": "true",
"Content-Type": "application/x-www-form-urlencoded",
"Referer": referer,
},
)
return urllib.request.urlopen(req, timeout=120)
def latest_round(doc):
idxs = re.findall(r'name="pull-round-idx"[^>]*value="(\d+)"', doc)
return max((int(i) for i in idxs), default=0)
def state(owner, repo, pull, cookie):
"""(verdict, round) — verdict is the appview's own mergeability text."""
page = get(f"{BASE}/{owner}/{repo}/pulls/{pull}", cookie)
rnd = latest_round(page)
frag = get(f"{BASE}/{owner}/{repo}/pulls/{pull}/round/{rnd}/actions", cookie)
txt = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", frag)).strip()
for phrase in ("No conflicts, ready to merge", "patch is empty", "merged", "conflict"):
if phrase.lower() in txt.lower():
return phrase, rnd, txt
return "UNKNOWN", rnd, txt
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pull", required=True, type=int)
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument("--check", action="store_true")
g.add_argument("--merge", action="store_true")
g.add_argument("--close", action="store_true")
ap.add_argument("--reason", default=None, help="close only: logged, not posted")
a = ap.parse_args()
cookie = load_cookie()
verdict, rnd, txt = state(a.owner, a.repo, a.pull, cookie)
print(f"#{a.pull} round {rnd}: {verdict}")
if a.check:
print(f" raw: {txt[:200]}")
return
referer = f"{BASE}/{a.owner}/{a.repo}/pulls/{a.pull}"
if a.merge:
if verdict != "No conflicts, ready to merge":
sys.exit(f"refusing to merge #{a.pull}: appview says {verdict!r}")
r = post(f"{referer}/merge", cookie, referer)
print(f" merged #{a.pull} — HTTP {r.status}")
else:
r = post(f"{referer}/close", cookie, referer)
print(f" closed #{a.pull} — HTTP {r.status}" + (f" ({a.reason})" if a.reason else ""))
if __name__ == "__main__":
main()
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env python3
"""Advance a Tangled pull to a new ROUND after you pushed a fixup, and print its interdiff URL.
WHY THIS EXISTS: pushing to the source branch does NOT update the pull. The appview keeps
serving the diff it fetched when the pull was opened (or last resubmitted), so a reviewer
reading the pull sees the code as it was BEFORE your fixup, and no
`/round/<r>/interdiff` exists to show what changed. Nothing warns you — the push
succeeded, the branch is right, and the pull silently lags. That cost a real review cycle
on 2026-08-16: three PRs were reviewed against stale trees because "pushed" was assumed to
mean "resubmitted".
The appview's own action is a bare htmx POST with no body:
POST /{owner}/{repo}/pulls/{n}/resubmit
which re-fetches the branch patch from the knot and opens round N+1. This wraps it, then
reports the new round and the interdiff URL to hand to the reviewer.
USAGE:
tangled_pr_resubmit.py --owner X --repo Y --pull 376
tangled_pr_resubmit.py --owner X --repo Y --pull 371 372 374 375 # several, in order
tangled_pr_resubmit.py --owner X --repo Y --pull 376 --check # rounds only, no POST
AFTER A FIXUP, THE WHOLE DANCE IS:
git push review <branch> && tangled_pr_resubmit.py --owner ... --repo ... --pull <n>
then reply on the pull, leading with the printed interdiff URL.
"""
import argparse, os, re, sys, urllib.error, urllib.request
BASE = "https://tangled.org"
def load_cookie(path=None):
"""The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md.
`path` is the legacy escape hatch: a file holding a TANGLED_COOKIE=... line."""
if path:
for line in open(path):
if line.strip().startswith("TANGLED_COOKIE="):
return line.strip()[len("TANGLED_COOKIE=") :]
sys.exit(f"{path} has no TANGLED_COOKIE= line")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
return c
def get(url, cookie):
req = urllib.request.Request(url, headers={"Cookie": cookie, "User-Agent": "tangled-pr-bot"})
return urllib.request.urlopen(req, timeout=60).read().decode()
def post(url, cookie, referer):
req = urllib.request.Request(
url,
data=b"",
headers={
"Cookie": cookie,
"User-Agent": "tangled-pr-bot",
"HX-Request": "true",
"Content-Type": "application/x-www-form-urlencoded",
"Referer": referer,
},
)
return urllib.request.urlopen(req, timeout=120)
def latest_round(owner, repo, pull, cookie):
"""The highest round the pull has. Read from the round selector, NOT from the page text:
a comment that quotes a `/round/2/interdiff` URL would otherwise be counted as a round."""
page = get(f"{BASE}/{owner}/{repo}/pulls/{pull}", cookie)
idxs = re.findall(r'name="pull-round-idx"[^>]*value="(\d+)"', page)
return max((int(i) for i in idxs), default=0)
def interdiff_url(owner, repo, pull, rnd):
return f"{BASE}/{owner}/{repo}/pulls/{pull}/round/{rnd}/interdiff?diff=unified"
def resubmit(owner, repo, pull, cookie, check_only=False):
before = latest_round(owner, repo, pull, cookie)
if check_only:
print(f"#{pull}: round {before}" + (f" — interdiff {interdiff_url(owner, repo, pull, before)}" if before else " — no interdiff yet (round 0)"))
return 0
referer = f"{BASE}/{owner}/{repo}/pulls/{pull}"
try:
r = post(f"{referer}/resubmit", cookie, referer)
except urllib.error.HTTPError as e:
print(f"#{pull}: resubmit FAILED — HTTP {e.code} {e.read()[:200]!r}", file=sys.stderr)
return 1
after = latest_round(owner, repo, pull, cookie)
if after > before:
print(f"#{pull}: round {before} -> {after} (HTTP {r.status})")
print(f" interdiff: {interdiff_url(owner, repo, pull, after)}")
return 0
# the appview accepted the POST but the round did not move: the branch is identical to
# what the pull already carries. Say so — silence here reads as success.
print(f"#{pull}: still round {after} (HTTP {r.status}) — the branch matches the pull; nothing to resubmit")
return 0
def main():
ap = argparse.ArgumentParser(description="resubmit a Tangled pull after pushing a fixup")
ap.add_argument("--owner", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pull", required=True, type=int, nargs="+", help="one or more pull numbers")
ap.add_argument("--check", action="store_true", help="report the current round; do not resubmit")
ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file")
a = ap.parse_args()
cookie = load_cookie(a.cookie_file)
rc = 0
for pull in a.pull:
rc |= resubmit(a.owner, a.repo, pull, cookie, a.check)
sys.exit(rc)
if __name__ == "__main__":
main()
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Create a Tangled repo via a reused session cookie (sibling of tangled_pr.py).
Tangled repo creation is an htmx POST to /repo/new (the web "New repository" form), authenticated by the
bot's appview session cookie. There is no separate git-create; you create the repo here, then push to
git@tangled.org:<owner>/<repo>. Cookie: engine/.tangled-session (refresh with scripts/get-tangled-cookie.py).
python3 engine/tangled_repo.py --name lichen.page.backup --description "..."
# then: git remote add backup git@tangled.org:notplants-bot.bsky.social/lichen.page.backup
# git push --force backup 'refs/remotes/<src>/*:refs/heads/*' && git push --force backup --tags
"""
import argparse, os, sys, urllib.parse, urllib.request
BASE = "https://tangled.org"
def load_cookie(path=None):
"""The cookie lives in the encrypted store (tangled.cookie) — see engine/README.md (Secrets).
`path` is a legacy escape hatch: a TANGLED_COOKIE=... file, used only if explicitly passed."""
if path:
for line in open(path):
if line.strip().startswith("TANGLED_COOKIE="):
return line.strip()[len("TANGLED_COOKIE="):]
sys.exit(f"{path} has no TANGLED_COOKIE= line")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import secrets as _store
c = _store.get("tangled.cookie")
if not c:
sys.exit("no tangled.cookie in the secret store — add it with: sops /secrets/store.yaml")
return c
def main():
ap = argparse.ArgumentParser(description="create a Tangled repo via a reused session cookie")
ap.add_argument("--name", required=True, help="repo name, e.g. lichen.page.backup")
ap.add_argument("--description", default="")
ap.add_argument("--branch", default="main", help="default branch (form default: main)")
ap.add_argument("--domain", default="knot1.tangled.sh", help="knot to host on (radio value)")
ap.add_argument("--cookie-file", default=None, help="legacy: read the cookie from this file instead of the store")
a = ap.parse_args()
cookie = load_cookie(a.cookie_file)
# The form only processes the create when it sees HX-Request (otherwise it re-renders the page — a 200
# that creates nothing). Success = HTTP 200 with an HX-Location header pointing at the owner/repo.
form = {"name": a.name, "description": a.description, "branch": a.branch, "domain": a.domain}
data = urllib.parse.urlencode(form).encode()
req = urllib.request.Request(f"{BASE}/repo/new", data=data, method="POST", headers={
"Cookie": cookie,
"HX-Request": "true",
"HX-Current-URL": f"{BASE}/repo/new",
"Content-Type": "application/x-www-form-urlencoded",
})
try:
resp = urllib.request.urlopen(req, timeout=30)
body = resp.read().decode("utf-8", "replace")
loc = resp.headers.get("HX-Location") or resp.headers.get("HX-Redirect") or ""
except urllib.error.HTTPError as e:
sys.exit(f"create failed: HTTP {e.code}\n{e.read().decode('utf-8','replace')[:500]}")
if resp.status == 200 and loc:
print(f"OK: repo created -> {loc} (push to git@tangled.org:<owner>/{a.name})")
elif "already exists" in body.lower():
sys.exit(f"repo {a.name!r} already exists")
else:
sys.exit(f"unexpected response (status {resp.status}, no HX-Location). Body head:\n{body[:500]}")
if __name__ == "__main__":
main()
Executable
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# ─────────────────────────────────────────────────────────────────────────────
# agent-orchestrator test runner.
#
# • UNIT tests — always run (pure logic, no agents spawned). A failure fails the suite.
# • CLAUDE smoke — live, run when the `claude` CLI is available; SKIPs otherwise.
# • CODEX smoke — live, run when Codex is installed, logged in, and Remote Control connects.
# • OPENCODE smoke — live, run when `opencode` + creds are available; SKIPs otherwise.
# • ISOLATION sanity — after the live runs: assert no leftover aotest-* tmux sessions, and that
# the live cc-ci-* sessions are untouched.
#
# Run inside the devShell: nix develop -c ./tests/run.sh
# or simply: ./tests/run.sh (python3 + tmux must be on PATH)
#
# Exit: 0 = all run tests passed (skips are OK); 1 = a unit test or a live smoke FAILED, or a
# leftover aotest-* session was found.
# ─────────────────────────────────────────────────────────────────────────────
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
REPO="$(cd "$HERE/.." && pwd)"
RC=0
UNIT=FAIL CLAUDE=SKIP CODEX=SKIP OPENCODE=SKIP ISO=PASS
echo "######################################################################"
echo "# agent-orchestrator test suite"
echo "######################################################################"
# ── unit tests (always) ───────────────────────────────────────────────────────────
echo; echo ">>> UNIT TESTS"
if python3 -m unittest discover -s "$HERE" -p 'test_*.py' -v; then
UNIT=PASS
else
UNIT=FAIL; RC=1
fi
# helper: run a smoke script, classify its result from its output
run_smoke() {
local label="$1" script="$2"; shift 2
echo; echo ">>> ${label} SMOKE"
local out
out="$(bash "$script" 2>&1)"; local rc=$?
echo "$out"
if echo "$out" | grep -q "BACKEND SMOKE: PASS"; then echo "PASS"; return 0; fi
if [ "$rc" -eq 0 ] && echo "$out" | grep -qE "^SKIP:"; then echo "SKIP"; return 2; fi
echo "FAIL"; return 1
}
# ── live smoke tests (when backends available) ──────────────────────────────────────
run_smoke "CLAUDE" "$HERE/smoke_claude.sh"; case $? in 0) CLAUDE=PASS;; 2) CLAUDE=SKIP;; *) CLAUDE=FAIL; RC=1;; esac
run_smoke "CODEX" "$HERE/smoke_codex.sh"; case $? in 0) CODEX=PASS;; 2) CODEX=SKIP;; *) CODEX=FAIL; RC=1;; esac
run_smoke "OPENCODE" "$HERE/smoke_opencode.sh"; case $? in 0) OPENCODE=PASS;; 2) OPENCODE=SKIP;; *) OPENCODE=FAIL; RC=1;; esac
# ── isolation sanity ────────────────────────────────────────────────────────────────
echo; echo ">>> ISOLATION SANITY"
if command -v tmux >/dev/null 2>&1; then
leftover="$(tmux ls 2>/dev/null | sed 's/:.*//' | grep '^aotest-' || true)"
if [ -n "$leftover" ]; then
echo " FAIL: leftover aotest-* sessions: $leftover"; ISO=FAIL; RC=1
else
echo " PASS: no leftover aotest-* tmux sessions"
fi
intact=""
for s in cc-ci-orchestrator cc-ci-watchdog cc-ci-assistant3; do
tmux has-session -t "=$s" 2>/dev/null && intact="$intact $s"
done
echo " info: live cc-ci sessions present:${intact:- (none — not a cc-ci host)}"
else
echo " (tmux not on PATH — isolation sanity skipped)"
fi
# ── summary ─────────────────────────────────────────────────────────────────────────
echo; echo "######################################################################"
echo "# SUMMARY: unit=$UNIT claude=$CLAUDE codex=$CODEX opencode=$OPENCODE isolation=$ISO"
echo "######################################################################"
[ "$RC" -eq 0 ] && echo "ALL RUN TESTS PASSED (skips are OK)" || echo "SUITE FAILED"
exit "$RC"
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env bash
# ─────────────────────────────────────────────────────────────────────────────
# Isolated LIVE smoke of the CLAUDE backend, driven entirely through the harness.
#
# Brings a throwaway scratch project (its OWN session_prefix "aotest-c-<pid>-" and a temporary
# log_dir) up through `agents.py up`, on the real `claude` CLI:
# • the harness builds the claude launch command (arg delivery + remote-control + model flag),
# • the agent attaches in tmux (claude TUI alive, not an instant crash),
# • `agents.py status` reports it RUNNING,
# • `agents.py down` tears it down cleanly — no leftover sessions.
#
# SAFE BY CONSTRUCTION — never touches the live cc-ci-* sessions:
# • a unique per-run session prefix (NOT "cc-ci-")
# • cleans up everything it creates on exit (even on Ctrl+C / error).
#
# Usage: bash tests/smoke_claude.sh
# Env: CLAUDE_BIN (default: `claude` on PATH, else ~/.local/bin/claude)
# AOTEST_MODEL (default: claude-haiku-4-5 — a cheap model for the trivial probe)
# Exit: 0 = PASS or SKIP (claude unavailable); 1 = FAIL.
# ─────────────────────────────────────────────────────────────────────────────
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
REPO="$(cd "$HERE/.." && pwd)"
CLAUDE_BIN="${CLAUDE_BIN:-$(command -v claude 2>/dev/null || echo "$HOME/.local/bin/claude")}"
MODEL="${AOTEST_MODEL:-claude-haiku-4-5}"
PREFIX="aotest-c-$$-"
SANDBOX="$(mktemp -d)"
CFG="$SANDBOX/agents.toml"
FAILED=0
pass(){ echo " PASS: $*"; }
fail(){ echo " FAIL: $*"; FAILED=1; }
cleanup(){
local rc=$?
python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1 || true
if command -v tmux >/dev/null 2>&1; then
tmux ls 2>/dev/null | sed 's/:.*//' | grep "^${PREFIX}" | while read -r s; do
tmux kill-session -t "=$s" 2>/dev/null || true
done || true
fi
rm -rf "$SANDBOX"
exit "$rc"
}
trap cleanup EXIT INT TERM
echo "=== claude backend smoke (isolated: prefix=${PREFIX}) ==="
# 0 — preconditions (SKIP, not FAIL, when claude/tmux can't run here)
command -v tmux >/dev/null 2>&1 || { echo "SKIP: tmux not on PATH (run inside 'nix develop')"; exit 0; }
[ -x "$CLAUDE_BIN" ] || command -v "$CLAUDE_BIN" >/dev/null 2>&1 \
|| { echo "SKIP: claude binary not found ($CLAUDE_BIN)"; exit 0; }
# 1 — isolated sandbox config (unique prefix + temp log_dir; one trivial persistent probe)
cat > "$CFG" <<EOF
[defaults]
project_dir = "$REPO"
session_prefix = "$PREFIX"
log_dir = "$SANDBOX/state"
backend = "claude"
model = "$MODEL"
watch = "none"
[backend.claude]
bin = "$CLAUDE_BIN"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|bypass permissions"
limit_re = "usage limit|limit reached"
[[agent]]
name = "probe"
kind = "persistent"
prompt = "You are a harness self-test. Reply with the single word READY and then wait silently. Do nothing else."
EOF
# 2 — bring the probe up THROUGH the harness
if ! python3 "$REPO/agents.py" --config "$CFG" up probe; then
fail "agents.py up probe errored"; echo "=== RESULT: FAIL ==="; exit 1
fi
# 3 — session created?
sleep 6
if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
cmd=$(tmux display-message -p -t "=${PREFIX}probe:" '#{pane_current_command}' 2>/dev/null)
pass "session ${PREFIX}probe created via agents.py (pane command: ${cmd})"
else
fail "${PREFIX}probe session was not created"; echo "=== RESULT: FAIL ==="; exit 1
fi
# 4 — claude actually attached (TUI alive), not an instant crash
sleep 6
cmd=$(tmux display-message -p -t "=${PREFIX}probe:" '#{pane_current_command}' 2>/dev/null)
pane=$(tmux capture-pane -p -t "=${PREFIX}probe:" 2>/dev/null)
if [ "$cmd" = "claude" ] || echo "$pane" | grep -qiE "esc to interrupt|bypass permissions|READY|claude||welcome"; then
pass "claude TUI attached + alive (driven entirely by agents.py)"
else
fail "no claude TUI in pane (cmd=${cmd}); tail: $(echo "$pane" | grep -vE '^\s*$' | tail -3)"
fi
# 5 — status reports it RUNNING
if python3 "$REPO/agents.py" --config "$CFG" status | grep -E '^\s*probe\b' | grep -q RUNNING; then
pass "agents.py status reports probe RUNNING"
else
fail "agents.py status did not report probe RUNNING"
fi
# 6 — lifecycle: down removes it cleanly
python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1
sleep 2
if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
fail "${PREFIX}probe still alive after agents.py down"
else
pass "agents.py down cleanly removed the session"
fi
if [ "$FAILED" = 0 ]; then echo "=== CLAUDE BACKEND SMOKE: PASS ==="; exit 0
else echo "=== CLAUDE BACKEND SMOKE: FAIL ==="; exit 1; fi
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# Isolated live smoke of the Codex backend, driven entirely through agents.py.
# It starts one uniquely-named tmux session, verifies Remote Control and a daemon-connected Codex
# TUI, then removes only that session. The shared Remote Control daemon intentionally remains
# available.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
REPO="$(cd "$HERE/.." && pwd)"
CODEX_BIN="${CODEX_BIN:-$(command -v codex 2>/dev/null || echo "$HOME/.local/bin/codex")}"
MODEL="${AOTEST_CODEX_MODEL:-}"
PREFIX="aotest-x-$$-"
SANDBOX="$(mktemp -d)"
CFG="$SANDBOX/agents.toml"
FAILED=0
pass(){ echo " PASS: $*"; }
fail(){ echo " FAIL: $*"; FAILED=1; }
cleanup(){
local rc=$?
python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1 || true
if command -v tmux >/dev/null 2>&1; then
tmux ls 2>/dev/null | sed 's/:.*//' | grep "^${PREFIX}" | while read -r s; do
tmux kill-session -t "=$s" 2>/dev/null || true
done || true
fi
rm -rf "$SANDBOX"
exit "$rc"
}
trap cleanup EXIT INT TERM
echo "=== codex backend smoke (isolated: prefix=${PREFIX}) ==="
command -v tmux >/dev/null 2>&1 || { echo "SKIP: tmux not on PATH"; exit 0; }
[ -x "$CODEX_BIN" ] || command -v "$CODEX_BIN" >/dev/null 2>&1 \
|| { echo "SKIP: codex binary not found ($CODEX_BIN)"; exit 0; }
"$CODEX_BIN" login status >/dev/null 2>&1 \
|| { echo "SKIP: Codex is not logged in"; exit 0; }
remote_status="$("$CODEX_BIN" remote-control start --json 2>&1)"
echo "$remote_status" | grep -q '"status":"connected"' \
|| { echo "SKIP: Codex Remote Control is not connected: $remote_status"; exit 0; }
pass "Codex Remote Control reports connected"
model_line=""
[ -n "$MODEL" ] && model_line="model = \"$MODEL\""
cat > "$CFG" <<EOF
[defaults]
project_dir = "$REPO"
session_prefix = "$PREFIX"
log_dir = "$SANDBOX/state"
backend = "codex"
$model_line
watch = "none"
[backend.codex]
bin = "$CODEX_BIN"
preamble = "{bin} remote-control start --json >/dev/null"
remote_addr = "unix://"
flags = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
supports_resume = false
prompt_delivery = "arg"
process_name = "codex"
footer_ui = true
log_grace = 180
submit_key = "Enter"
startup_prompt_re = "Trusting the directory|Do you trust"
startup_prompt_response = "1"
startup_prompt_delay = 2
stall_idle = 300
active_re = "esc to interrupt|working|thinking|running|searching"
limit_re = "usage limit|limit reached"
[[agent]]
name = "probe"
kind = "persistent"
prompt = "You are a harness self-test. Reply exactly CODEX_BACKEND_READY and then wait silently. Do not use tools or modify files."
EOF
python3 "$REPO/agents.py" --config "$CFG" up probe \
|| { fail "agents.py up probe errored"; echo "=== CODEX BACKEND SMOKE: FAIL ==="; exit 1; }
sleep 12
if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
cmd="$(tmux display-message -p -t "=${PREFIX}probe:" '#{pane_current_command}')"
pass "session ${PREFIX}probe created via agents.py (pane command: ${cmd})"
else
fail "${PREFIX}probe session was not created"
echo "=== CODEX BACKEND SMOKE: FAIL ==="
exit 1
fi
pane="$(tmux capture-pane -p -t "=${PREFIX}probe:" -S -200 2>/dev/null)"
if [ "$cmd" = "codex" ] && echo "$pane" | grep -q "CODEX_BACKEND_READY"; then
pass "Codex TUI attached and completed the probe"
else
fail "Codex probe did not complete (cmd=${cmd}); tail: $(echo "$pane" | grep -vE '^\s*$' | tail -5)"
fi
if echo "$pane" | grep -Fq "$REPO"; then
pass "daemon-created Codex thread uses the requested project directory"
else
fail "Codex thread did not use requested directory $REPO"
fi
if python3 "$REPO/agents.py" --config "$CFG" status \
| grep -E '^\s*probe\b' | grep -q RUNNING; then
pass "agents.py status reports probe RUNNING"
else
fail "agents.py status did not report probe RUNNING"
fi
python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1
sleep 2
if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
fail "${PREFIX}probe still alive after agents.py down"
else
pass "agents.py down cleanly removed the session"
fi
if [ "$FAILED" = 0 ]; then echo "=== CODEX BACKEND SMOKE: PASS ==="; exit 0
else echo "=== CODEX BACKEND SMOKE: FAIL ==="; exit 1; fi
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env bash
# ─────────────────────────────────────────────────────────────────────────────
# Isolated LIVE smoke of the OPENCODE backend, driven entirely through the harness.
#
# Generalizes the cc-ci `test-opencode.sh` isolation pattern onto the agent-orchestrator harness:
# stands up a DEDICATED opencode server on its own port (≠ 4096), then brings a throwaway scratch
# project up through `agents.py up` on the opencode backend:
# • the harness builds the opencode attach command + the post-connect bootstrap ping,
# • the agent attaches to the server (opencode TUI alive),
# • `agents.py status` reports it RUNNING,
# • `agents.py down` tears it down cleanly — server killed, no leftover sessions, port freed.
#
# SAFE BY CONSTRUCTION — never touches the live cc-ci-* sessions or the live opencode server:
# • a unique per-run session prefix (NOT "cc-ci-")
# • its OWN opencode server on AOTEST_OC_PORT (default 4097, never 4096)
# • cleans up everything it creates on exit (even on Ctrl+C / error).
#
# Usage: bash tests/smoke_opencode.sh
# Env: OPENCODE_BIN (default: `opencode` on PATH, else ~/.local/bin/opencode)
# AOTEST_OC_PORT (default 4097 — MUST differ from the live 4096)
# AOTEST_OC_CREDS (default /srv/cc-ci/.testenv — sourced as the backend preamble)
# AOTEST_MODEL (default: opencode's own configured default)
# Exit: 0 = PASS or SKIP (opencode / creds / server unavailable); 1 = FAIL.
# ─────────────────────────────────────────────────────────────────────────────
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
REPO="$(cd "$HERE/.." && pwd)"
OCBIN="${OPENCODE_BIN:-$(command -v opencode 2>/dev/null || echo "$HOME/.local/bin/opencode")}"
PORT="${AOTEST_OC_PORT:-4097}"
SERVER="http://127.0.0.1:${PORT}"
CREDS="${AOTEST_OC_CREDS:-/srv/cc-ci/.testenv}"
MODEL="${AOTEST_MODEL:-}"
PREFIX="aotest-o-$$-"
SANDBOX="$(mktemp -d)"
CFG="$SANDBOX/agents.toml"
SRVLOG="$SANDBOX/server.log"
SERVER_PID=""
FAILED=0
pass(){ echo " PASS: $*"; }
fail(){ echo " FAIL: $*"; FAILED=1; }
cleanup(){
local rc=$?
python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1 || true
if command -v tmux >/dev/null 2>&1; then
tmux ls 2>/dev/null | sed 's/:.*//' | grep "^${PREFIX}" | while read -r s; do
tmux kill-session -t "=$s" 2>/dev/null || true
done || true
fi
# kill the server subshell AND the opencode serve child it forked (the subshell is not the
# listener — target the listener by our unique port so the port is actually freed).
[ -n "$SERVER_PID" ] && kill "$SERVER_PID" 2>/dev/null || true
pkill -f "opencode serve.*--port ${PORT}\b" 2>/dev/null || true
for _ in 1 2 3 4 5; do
ss -ltn 2>/dev/null | grep -q ":${PORT} " || break
sleep 1
done
rm -rf "$SANDBOX"
exit "$rc"
}
trap cleanup EXIT INT TERM
echo "=== opencode backend smoke (isolated: prefix=${PREFIX} port=${PORT}) ==="
# 0 — preconditions (SKIP, not FAIL, when the environment can't run opencode)
command -v tmux >/dev/null 2>&1 || { echo "SKIP: tmux not on PATH (run inside 'nix develop')"; exit 0; }
[ "$PORT" != "4096" ] || { echo "FAIL: refusing port 4096 (the live cc-ci opencode port)"; exit 1; }
[ -x "$OCBIN" ] || command -v "$OCBIN" >/dev/null 2>&1 \
|| { echo "SKIP: opencode binary not found ($OCBIN)"; exit 0; }
[ -f "$CREDS" ] || { echo "SKIP: opencode creds file missing ($CREDS)"; exit 0; }
# 1 — isolated sandbox config (unique prefix + temp log_dir + dedicated server)
cat > "$CFG" <<EOF
[defaults]
project_dir = "$REPO"
session_prefix = "$PREFIX"
log_dir = "$SANDBOX/state"
backend = "opencode"
model = "$MODEL"
watch = "none"
[backend.opencode]
bin = "$OCBIN"
attach = "{bin} attach {server} --dir {dir}"
server = "$SERVER"
supports_resume = false
prompt_delivery = "ping"
process_name = "opencode"
footer_ui = true
log_grace = 180
connect_delay = 12
submit_key = "C-m"
preamble = "set -a; . $CREDS; set +a"
stall_idle = 900
active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching|working"
limit_re = "usage limit|limit reached"
[[agent]]
name = "probe"
kind = "persistent"
prompt = "You are a harness self-test. Reply with the single word READY and then wait silently. Do nothing else."
EOF
# 2 — bring up a dedicated opencode server on our own port
( set -a; . "$CREDS"; set +a; NO_COLOR=1 "$OCBIN" serve --hostname 127.0.0.1 --port "$PORT" ) >"$SRVLOG" 2>&1 &
SERVER_PID=$!
for _ in $(seq 1 30); do ss -ltn 2>/dev/null | grep -q ":${PORT} " && break; sleep 1; done
if ! ss -ltn 2>/dev/null | grep -q ":${PORT} "; then
echo "SKIP: opencode server did not come up on :${PORT} (see ${SRVLOG})"; exit 0
fi
pass "dedicated opencode server listening on :${PORT}"
# 3 — bring the probe up THROUGH the harness (attaches to OUR server)
if ! python3 "$REPO/agents.py" --config "$CFG" up probe; then
fail "agents.py up probe errored"; echo "=== RESULT: FAIL ==="; exit 1
fi
# 4 — session created?
sleep 4
if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
cmd=$(tmux display-message -p -t "=${PREFIX}probe:" '#{pane_current_command}' 2>/dev/null)
pass "session ${PREFIX}probe created via agents.py (pane command: ${cmd})"
else
fail "${PREFIX}probe session was not created"; echo "=== RESULT: FAIL ==="; exit 1
fi
# 5 — opencode TUI attached + alive, not an instant crash
sleep 12
pane=$(tmux capture-pane -p -t "=${PREFIX}probe:" 2>/dev/null)
if echo "$pane" | grep -qiE "opencode|build ·|gpt|claude|READY|esc interrupt|ctrl\+p|ctrl\+"; then
pass "opencode TUI attached + alive (driven entirely by agents.py)"
else
fail "no opencode TUI/response in pane; tail: $(echo "$pane" | grep -vE '^\s*$' | tail -3)"
echo " (server log tail:) $(tail -3 "$SRVLOG" 2>/dev/null)"
fi
# 6 — status reports it RUNNING
if python3 "$REPO/agents.py" --config "$CFG" status | grep -E '^\s*probe\b' | grep -q RUNNING; then
pass "agents.py status reports probe RUNNING"
else
fail "agents.py status did not report probe RUNNING"
fi
# 7 — lifecycle: down removes it cleanly
python3 "$REPO/agents.py" --config "$CFG" down probe >/dev/null 2>&1
sleep 2
if tmux has-session -t "=${PREFIX}probe" 2>/dev/null; then
fail "${PREFIX}probe still alive after agents.py down"
else
pass "agents.py down cleanly removed the session"
fi
if [ "$FAILED" = 0 ]; then echo "=== OPENCODE BACKEND SMOKE: PASS ==="; exit 0
else echo "=== OPENCODE BACKEND SMOKE: FAIL ==="; exit 1; fi
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env python3
"""Unit tests for the standalone tools (the tangled_* scripts, secrets, gateway-domain).
Run: python3 -m unittest tests.test_tools (from the repo root)
./tests/run.sh (discovers this file automatically)
NO NETWORK. Every HTTP boundary is injected as a fake `_fetch`, so these run offline and
in CI. The point is the DECISION logic — "did the pull get created", "is this backend
valid" — because that is where the defects have actually been.
The reason this file exists: tangled_pr.py judged success only by a response header, so a
create that answered without one read as a failure, the caller retried, and duplicate
pulls #397-#399 were filed for one branch. The tool was never tested; the bug survived
months of daily use and was found by counting pulls, not by running the script.
"""
import os, sys, unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import tangled_pr # noqa: E402
class TestPullNumbers(unittest.TestCase):
def test_extracts_every_linked_pull(self):
html = '<a href="/o/r/pulls/417">x</a><a href="/o/r/pulls/416">y</a>'
self.assertEqual(tangled_pr.pull_numbers(html), {417, 416})
def test_deduplicates_repeated_links(self):
html = '/pulls/12 /pulls/12 /pulls/12'
self.assertEqual(tangled_pr.pull_numbers(html), {12})
def test_ignores_the_new_pull_form(self):
# /pulls/new must not parse as a number — it is the page you POST to, not a pull
self.assertEqual(tangled_pr.pull_numbers('/pulls/new /pulls/9'), {9})
def test_empty_and_none_are_empty_not_an_error(self):
# an unreadable page must yield "cannot confirm", never a crash mid-create
self.assertEqual(tangled_pr.pull_numbers(""), set())
self.assertEqual(tangled_pr.pull_numbers(None), set())
class TestPageNamesBranch(unittest.TestCase):
def test_finds_the_branch(self):
self.assertTrue(tangled_pr.page_names_branch("<div>e2e-pds/R4-real</div>", "e2e-pds/R4-real"))
def test_absent_branch(self):
self.assertFalse(tangled_pr.page_names_branch("<div>other</div>", "e2e-pds/R4-real"))
def test_unreadable_page_is_not_a_match(self):
self.assertFalse(tangled_pr.page_names_branch("", "b"))
self.assertFalse(tangled_pr.page_names_branch(None, "b"))
def test_empty_branch_never_matches(self):
# guards against a caller passing "" and matching every page
self.assertFalse(tangled_pr.page_names_branch("anything", ""))
class TestClassify(unittest.TestCase):
"""The regression that matters: a successful create with NO redirect header."""
def test_new_pull_naming_our_branch_is_created(self):
v, n = tangled_pr.classify({1, 2}, {1, 2, 3}, {3: True})
self.assertEqual((v, n), ("created", 3))
def test_created_even_though_the_header_said_nothing(self):
# THE BUG: classify never sees the response at all, so a missing HX-Redirect
# cannot turn a real create into a reported failure.
v, n = tangled_pr.classify({396}, {396, 397}, {397: True})
self.assertEqual(v, "created")
def test_no_new_pull_is_a_real_failure(self):
v, n = tangled_pr.classify({1, 2}, {1, 2}, {})
self.assertEqual((v, n), ("none", None))
def test_someone_elses_pull_is_not_ours(self):
v, n = tangled_pr.classify({1}, {1, 2}, {2: False})
self.assertEqual((v, n), ("unrelated", None))
def test_picks_the_highest_of_several_of_ours(self):
v, n = tangled_pr.classify({1}, {1, 2, 3}, {2: True, 3: True})
self.assertEqual((v, n), ("created", 3))
def test_a_pull_that_vanished_does_not_confuse_it(self):
# after ⊂ before (someone closed+deleted one mid-run): no new pull, so no create
v, n = tangled_pr.classify({1, 2}, {1}, {})
self.assertEqual((v, n), ("none", None))
class TestExistingPullScan(unittest.TestCase):
"""The pre-check that makes a retry harmless instead of duplicating."""
def _fetcher(self, pages):
def _f(url, cookie, timeout=30):
return pages.get(url.rsplit("/", 1)[-1], "")
return _f
def test_finds_an_existing_pull_for_the_branch(self):
f = self._fetcher({"396": "branch: feat/x", "395": "branch: other"})
got = tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {395, 396}, _fetch=f)
self.assertEqual(got, 396)
def test_returns_none_when_the_branch_is_new(self):
f = self._fetcher({"396": "other", "395": "other"})
self.assertIsNone(tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {395, 396}, _fetch=f))
def test_prefers_the_newest_match(self):
f = self._fetcher({"10": "feat/x", "20": "feat/x"})
self.assertEqual(tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {10, 20}, _fetch=f), 20)
def test_scan_window_is_bounded(self):
# a match older than the window is missed BY DESIGN; the post-check is the backstop.
pages = {str(n): ("feat/x" if n == 1 else "other") for n in range(1, 30)}
f = self._fetcher(pages)
self.assertIsNone(
tangled_pr.existing_pull_for("o", "r", "feat/x", "c", set(range(1, 30)), scan=5, _fetch=f))
def test_unreadable_pages_do_not_claim_absence(self):
# every fetch fails -> None, and main() warns rather than silently creating a duplicate
f = lambda url, cookie, timeout=30: ""
self.assertIsNone(tangled_pr.existing_pull_for("o", "r", "feat/x", "c", {1, 2}, _fetch=f))
class TestFetchIsFailSoft(unittest.TestCase):
def test_network_error_returns_empty_not_raise(self):
# a create must not die between POST and verification
self.assertEqual(tangled_pr.fetch("http://127.0.0.1:1/nope", "c", timeout=1), "")
class TestCloseToolStateParsing(unittest.TestCase):
"""tangled_pr_close.state_of decides whether a pull is already closed."""
def setUp(self):
import tangled_pr_close
self.mod = tangled_pr_close
def test_badge_regex_reads_the_states(self):
import re
badges = lambda doc: set(re.findall(r'>\s*(Merged|Closed|Open)\s*<', doc))
self.assertEqual(badges("<span> Closed </span>"), {"Closed"})
self.assertEqual(badges("<b>Open</b><b>Merged</b>"), {"Open", "Merged"})
self.assertEqual(badges("<p>closed</p>"), set()) # case-sensitive on purpose
class TestGatewayBackendValidation(unittest.TestCase):
"""gateway-domain refuses hostname backends: the gateway can store one and never remove it."""
def setUp(self):
import importlib.util
p = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
"tools", "gateway-domain.py")
if not os.path.exists(p):
self.skipTest("gateway-domain.py not present")
spec = importlib.util.spec_from_file_location("gwd", p)
self.gwd = importlib.util.module_from_spec(spec)
spec.loader.exec_module(self.gwd)
def test_accepts_ip_and_ip_port(self):
self.assertTrue(self.gwd._BACKEND.match("100.74.89.63"))
self.assertTrue(self.gwd._BACKEND.match("100.74.89.63:8443"))
def test_rejects_hostnames(self):
for bad in ("example.com", "host:443", "", "1.2.3", "1.2.3.4.5"):
self.assertIsNone(self.gwd._BACKEND.match(bad), f"should reject {bad!r}")
def test_allowed_tags_include_both_known_tags(self):
self.assertIn("tag:notplants-test-server", self.gwd.ALLOWED_TAGS)
if __name__ == "__main__":
unittest.main()
+827
View File
@@ -0,0 +1,827 @@
#!/usr/bin/env python3
"""Unit tests for the agent-orchestrator harness (agents.py).
Pure-logic tests — NO agent CLIs spawned, NO live tmux sessions created. Every test builds a
throwaway config + fixture files in a tempdir and exercises the harness functions directly.
The one function that would spawn sessions (phase_advance_check → start/stop_loops) is tested
with those two hooks monkeypatched to recorders, so the phase-machine *logic* is covered without
launching anything.
Run: python3 -m unittest tests.test_unit (from repo root)
or python3 tests/test_unit.py
"""
import os
import sys
import time
import textwrap
import tempfile
import shutil
import re
import signal
import subprocess
import unittest
from datetime import datetime, timedelta
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(REPO_ROOT))
import agents # noqa: E402
# ── shared fixture config ────────────────────────────────────────────────────────
BASE_TOML = r"""
[watchdog]
signal_interval = 30
heavy_interval = 300
limit_probe_fallback = 300
limit_reset_slack = 45
stall_grace = 180
[defaults]
session_prefix = "aotest-ut-"
log_dir = "state"
backend = "claude"
model = "claude-sonnet-4-6"
watch = "none"
[backend.claude]
bin = "claude"
flags = "--dangerously-skip-permissions"
remote_control = true
supports_resume = true
prompt_delivery = "arg"
process_name = "claude"
submit_key = "Enter"
stall_idle = 300
active_re = "esc to interrupt|Running tool|\\u00b7 \\d+"
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
fatal_re = "redacted_thinking|blocks cannot be modified"
[backend.opencode]
bin = "opencode"
attach = "{bin} attach {server} --dir {dir}"
server = "http://127.0.0.1:4096"
supports_resume = false
prompt_delivery = "ping"
process_name = "opencode"
footer_ui = true
log_grace = 180
connect_delay = 12
submit_key = "C-m"
stall_idle = 900
active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
limit_re = "usage limit|limit reached"
[backend.codex]
bin = "codex"
preamble = "{bin} remote-control start --json >/dev/null"
remote_addr = "unix://"
flags = "--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen"
supports_resume = false
prompt_delivery = "arg"
process_name = "codex"
footer_ui = true
log_grace = 180
submit_key = "Enter"
startup_prompt_re = "Trusting the directory|Do you trust"
startup_prompt_response = "1"
startup_prompt_delay = 0
active_re = "working|thinking|running"
limit_re = "usage limit|limit reached"
[backend.demo]
bin = "echo up; exec sleep 100000"
prompt_delivery = "exec"
[[agent]]
name = "builder"
kind = "loop"
role = "builder"
backend = "demo"
[[agent]]
name = "adversary"
kind = "loop"
role = "adversary"
backend = "demo"
[[agent]]
name = "cl"
kind = "persistent"
backend = "claude"
prompt = "hi"
[[agent]]
name = "oc"
kind = "persistent"
backend = "opencode"
prompt = "hi"
[[agent]]
name = "cx"
kind = "persistent"
backend = "codex"
model = "gpt-test"
prompt = "hi from codex"
[[agent]]
name = "custom"
kind = "persistent"
session = "explicit-session"
model = "override-model"
dir = "/abs/somewhere"
backend = "demo"
prompt = "x"
[[service]]
name = "svc"
command = "sleep 1"
[loop]
state_file = "phase-idx"
resume_phase = true
auto_advance = true
done_marker = "## DONE"
kickoff_template = "prompts/kickoff.md"
roles_dir = "prompts"
handoff = { repo = ".", claim_pings = "adversary", review_pings = "builder", inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"], state_subdir = "machine-docs" }
phases = [
{ id = "p1", plan = "PLAN1.md", status = "STATUS-p1.md" },
{ id = "p2", plan = "PLAN2.md", status = "STATUS-p2.md", models = { builder = "opus-x" } },
]
"""
KICKOFF_TMPL = "*** PROJECT PHASE: {phase_id} ***\nPLAN: {plan}\nSTATUS: {status}\nROLE: {role}\n---\n"
BUILDER_PROMPT = "You are the **Builder** agent. (builder role body marker)\n"
ADVERSARY_PROMPT = "You are the **Adversary** agent. (adversary role body marker)\n"
def _make_project(tmp, toml=BASE_TOML):
"""Write a self-contained project (config + prompts + machine-docs) into tmp; return cfg path."""
root = Path(tmp)
(root / "prompts").mkdir(parents=True, exist_ok=True)
(root / "machine-docs").mkdir(parents=True, exist_ok=True)
(root / "prompts" / "kickoff.md").write_text(KICKOFF_TMPL)
(root / "prompts" / "builder.md").write_text(BUILDER_PROMPT)
(root / "prompts" / "adversary.md").write_text(ADVERSARY_PROMPT)
cfg_path = root / "agents.toml"
cfg_path.write_text(toml)
return cfg_path
# ── config loading + defaults merge ────────────────────────────────────────────────
class TestConfigLoad(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg_path = _make_project(self.tmp)
self.cfg = agents.load_config(self.cfg_path)
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
def test_defaults_merge_into_agents(self):
b = self.cfg["agents"]["builder"]
self.assertEqual(b["session_prefix"], "aotest-ut-")
self.assertEqual(b["watch"], "none") # from defaults
self.assertEqual(b["kind"], "loop") # explicit
def test_session_name_defaults_to_prefix_plus_name(self):
self.assertEqual(self.cfg["agents"]["builder"]["session"], "aotest-ut-builder")
def test_explicit_session_overrides_prefix(self):
self.assertEqual(self.cfg["agents"]["custom"]["session"], "explicit-session")
def test_per_agent_override_wins_over_default(self):
# default model is claude-sonnet-4-6; custom overrides
self.assertEqual(self.cfg["agents"]["custom"]["model"], "override-model")
self.assertEqual(self.cfg["agents"]["builder"]["model"], "claude-sonnet-4-6")
def test_relative_dir_resolved_against_project_root(self):
# builder has no dir → defaults dir "." → project_dir
self.assertEqual(self.cfg["agents"]["builder"]["dir"], self.cfg["project_dir"])
def test_absolute_dir_kept(self):
self.assertEqual(self.cfg["agents"]["custom"]["dir"], "/abs/somewhere")
def test_log_dir_and_state_dir_resolved(self):
self.assertEqual(self.cfg["log_dir"], str(Path(self.cfg["project_dir"]) / "state"))
self.assertEqual(self.cfg["state_dir"], os.path.join(self.cfg["log_dir"], "state"))
self.assertTrue(Path(self.cfg["state_dir"]).is_dir()) # created on load
def test_service_session_named(self):
self.assertIn("svc", self.cfg["services"])
self.assertEqual(self.cfg["services"]["svc"]["session"], "aotest-ut-svc")
def test_backend_of_resolves(self):
b = agents.backend_of(self.cfg, self.cfg["agents"]["cl"])
self.assertEqual(b["prompt_delivery"], "arg")
self.assertEqual(b["submit_key"], "Enter")
def test_backend_of_unknown_dies(self):
a = dict(self.cfg["agents"]["cl"]); a["backend"] = "nope"
with self.assertRaises(SystemExit):
agents.backend_of(self.cfg, a)
def test_missing_session_prefix_dies(self):
bad = self.tmp + "/bad1"
p = _make_project(bad, toml='[defaults]\nlog_dir = "state"\n')
with self.assertRaises(SystemExit):
agents.load_config(p)
def test_missing_log_dir_dies(self):
bad = self.tmp + "/bad2"
p = _make_project(bad, toml='[defaults]\nsession_prefix = "x-"\n')
with self.assertRaises(SystemExit):
agents.load_config(p)
def test_env_override_model_single_invocation(self):
os.environ["AGENT_MODEL_cl"] = "env-only-model"
try:
cfg2 = agents.load_config(self.cfg_path)
self.assertEqual(cfg2["agents"]["cl"]["model"], "env-only-model")
finally:
del os.environ["AGENT_MODEL_cl"]
# without the env var the file value stands again
cfg3 = agents.load_config(self.cfg_path)
self.assertEqual(cfg3["agents"]["cl"]["model"], "claude-sonnet-4-6")
class TestExampleConfig(unittest.TestCase):
"""The SHIPPED agents.example.toml must parse and define the documented shape."""
def test_example_config_loads(self):
ex = REPO_ROOT / "agents.example.toml"
self.assertTrue(ex.exists(), "agents.example.toml missing from repo")
cfg = agents.load_config(ex)
self.assertIn("builder", cfg["agents"])
self.assertIn("adversary", cfg["agents"])
for be in ("demo", "claude", "codex", "opencode"):
self.assertIn(be, cfg["backends"], f"backend {be} missing from example")
self.assertEqual(len(agents.phases(cfg)), 2)
def test_codex_launch_prepares_remote_control_then_connects_tui_to_daemon(self):
tmp = tempfile.mkdtemp(prefix="aotest-ut-codex-")
old_alive, old_new, old_log = agents.session_alive, agents.new_session, agents.log
old_capture, old_run = agents.capture_pane, agents._run
try:
cfg = agents.load_config(_make_project(tmp))
launched = []
sent = []
agents.session_alive = lambda _session: False
agents.new_session = lambda session, cwd, cmd, log_path: launched.append(cmd)
agents.log = lambda _msg: None
agents.capture_pane = lambda *_args, **_kwargs: "Trusting the directory"
agents._run = lambda command: sent.append(command)
agents.start_agent(cfg, cfg["agents"]["cx"])
self.assertEqual(len(launched), 1)
cmd = launched[0]
self.assertTrue(cmd.startswith(
"codex remote-control start --json >/dev/null && "
"codex --remote 'unix://' -C "))
self.assertEqual(cmd.count("--remote 'unix://'"), 1)
self.assertIn(f"-C {agents.shlex.quote(cfg['agents']['cx']['dir'])}", cmd)
self.assertIn("--model 'gpt-test'", cmd)
self.assertIn(
"--dangerously-bypass-approvals-and-sandbox --dangerously-bypass-hook-trust --no-alt-screen",
cmd)
self.assertIn("kickoff-aotest-ut-cx.txt", cmd)
self.assertTrue(any("1" in command for command in sent))
finally:
agents.session_alive, agents.new_session, agents.log = old_alive, old_new, old_log
agents.capture_pane, agents._run = old_capture, old_run
shutil.rmtree(tmp, ignore_errors=True)
# ── kickoff-template assembly ──────────────────────────────────────────────────────
class TestKickoff(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
def test_kickoff_renders_slots_and_appends_role(self):
out = agents.build_loop_kickoff(self.cfg, self.cfg["agents"]["builder"])
self.assertIn("PROJECT PHASE: p1", out) # phase_id slot filled (phase idx 0)
self.assertIn("PLAN: PLAN1.md", out)
self.assertIn("STATUS: STATUS-p1.md", out)
self.assertIn("ROLE: builder", out)
self.assertIn("builder role body marker", out) # role prompt appended
self.assertNotIn("{phase_id}", out) # no unrendered slot
self.assertNotIn("{role}", out)
def test_kickoff_picks_correct_role_prompt(self):
out = agents.build_loop_kickoff(self.cfg, self.cfg["agents"]["adversary"])
self.assertIn("adversary role body marker", out)
self.assertNotIn("builder role body marker", out)
def test_agent_prompt_loop_returns_kickoff(self):
out = agents.agent_prompt(self.cfg, self.cfg["agents"]["builder"])
self.assertIn("PROJECT PHASE: p1", out)
def test_agent_prompt_persistent_returns_inline_prompt(self):
out = agents.agent_prompt(self.cfg, self.cfg["agents"]["cl"])
self.assertEqual(out, "hi")
def test_role_model_phase_override(self):
# phase p2 overrides builder model to opus-x; advance index to 1
Path(agents.phase_idx_file(self.cfg)).write_text("1")
self.assertEqual(agents.role_model(self.cfg, self.cfg["agents"]["builder"]), "opus-x")
# adversary has no override → its configured/default model
self.assertEqual(agents.role_model(self.cfg, self.cfg["agents"]["adversary"]),
"claude-sonnet-4-6")
# ── phase machine ──────────────────────────────────────────────────────────────────
class TestPhaseMachine(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
self.md = Path(self.cfg["project_dir"]) / "machine-docs"
# monkeypatch the session-spawning hooks so the machine logic runs without tmux
self._orig = (agents.stop_loops, agents.start_loops, agents.handoff_reset)
self.calls = []
agents.stop_loops = lambda cfg: self.calls.append("stop")
agents.start_loops = lambda cfg: self.calls.append("start")
agents.handoff_reset = lambda: self.calls.append("reset")
def tearDown(self):
agents.stop_loops, agents.start_loops, agents.handoff_reset = self._orig
shutil.rmtree(self.tmp, ignore_errors=True)
def _status(self, basename, text):
(self.md / basename).write_text(text)
def test_phase_done_detects_marker(self):
self._status("STATUS-p1.md", "header\n## DONE\nall verified PASS\n")
self.assertTrue(agents.phase_done(self.cfg, "STATUS-p1.md"))
def test_phase_done_rejects_placeholder_body(self):
self._status("STATUS-p1.md", "## DONE\nnot yet — written here only when complete\n")
self.assertFalse(agents.phase_done(self.cfg, "STATUS-p1.md"))
def test_phase_done_false_when_no_marker(self):
self._status("STATUS-p1.md", "## In progress\nworking\n")
self.assertFalse(agents.phase_done(self.cfg, "STATUS-p1.md"))
def test_phase_done_false_when_file_missing(self):
self.assertFalse(agents.phase_done(self.cfg, "STATUS-nope.md"))
def test_cur_idx_reads_state_file(self):
Path(agents.phase_idx_file(self.cfg)).write_text("1")
self.assertEqual(agents.cur_idx(self.cfg), 1)
def test_advance_on_done(self):
Path(agents.phase_idx_file(self.cfg)).write_text("0")
self._status("STATUS-p1.md", "## DONE\nverified\n")
advanced = agents.phase_advance_check(self.cfg)
self.assertTrue(advanced)
self.assertEqual(agents.cur_idx(self.cfg), 1) # moved to p2
self.assertIn("stop", self.calls)
self.assertIn("start", self.calls)
def test_no_advance_when_not_done(self):
Path(agents.phase_idx_file(self.cfg)).write_text("0")
self._status("STATUS-p1.md", "## In progress\n")
self.assertFalse(agents.phase_advance_check(self.cfg))
self.assertEqual(agents.cur_idx(self.cfg), 0)
self.assertEqual(self.calls, [])
def test_sequence_complete_idempotent(self):
Path(agents.phase_idx_file(self.cfg)).write_text("1") # last phase
self._status("STATUS-p2.md", "## DONE\nverified\n")
marker = Path(self.cfg["log_dir"]) / "SEQUENCE-COMPLETE"
# first call: completes the sequence
self.assertTrue(agents.phase_advance_check(self.cfg))
self.assertTrue(marker.exists())
self.assertEqual(self.calls.count("stop"), 1)
# second call: idempotent — no re-stop, returns False
self.assertFalse(agents.phase_advance_check(self.cfg))
self.assertEqual(self.calls.count("stop"), 1)
def test_append_phase_clears_marker_and_resumes(self):
# simulate "sequence already complete", then a 3rd phase appended to the config
Path(agents.phase_idx_file(self.cfg)).write_text("1")
self._status("STATUS-p2.md", "## DONE\nverified\n")
marker = Path(self.cfg["log_dir"]) / "SEQUENCE-COMPLETE"
marker.write_text("stale completion\n")
self.cfg["loop"]["phases"].append(
{"id": "p3", "plan": "PLAN3.md", "status": "STATUS-p3.md"})
advanced = agents.phase_advance_check(self.cfg)
self.assertTrue(advanced)
self.assertEqual(agents.cur_idx(self.cfg), 2) # resumed onto p3
self.assertFalse(marker.exists()) # stale marker cleared
self.assertIn("start", self.calls)
def test_custom_done_marker(self):
self.cfg["loop"]["done_marker"] = "## SHIPPED"
self._status("STATUS-p1.md", "## SHIPPED\nverified\n")
self.assertTrue(agents.phase_done(self.cfg, "STATUS-p1.md"))
self.assertFalse(agents.phase_done(self.cfg, "STATUS-p2.md"))
# ── usage-limit banner reset parsing ───────────────────────────────────────────────
class TestLimitParsing(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
def test_parse_reset_pm(self):
ep = agents._parse_reset_epoch("You've hit your limit · resets at 10pm")
self.assertIsNotNone(ep)
self.assertEqual(datetime.fromtimestamp(ep).hour, 22)
def test_parse_reset_am_with_minutes(self):
ep = agents._parse_reset_epoch("resets 3:30am")
self.assertIsNotNone(ep)
dt = datetime.fromtimestamp(ep)
self.assertEqual((dt.hour, dt.minute), (3, 30))
def test_parse_reset_12am_is_midnight(self):
ep = agents._parse_reset_epoch("resets at 12am")
self.assertEqual(datetime.fromtimestamp(ep).hour, 0)
def test_parse_reset_invalid_hour_none(self):
self.assertIsNone(agents._parse_reset_epoch("resets at 25"))
def test_parse_reset_no_match_none(self):
self.assertIsNone(agents._parse_reset_epoch("everything is fine here"))
def test_parse_reset_picks_last_match(self):
ep = agents._parse_reset_epoch("resets at 9am ... actually resets at 11am")
self.assertEqual(datetime.fromtimestamp(ep).hour, 11)
def test_next_limit_until_unparsable_fallback(self):
now = time.time()
until, parsed = agents._next_limit_until(self.cfg, "limit reached, no time given", now)
self.assertFalse(parsed)
self.assertEqual(int(until), int(now + 300)) # limit_probe_fallback
def test_next_limit_until_within_window_uses_banner(self):
now = time.time()
t = datetime.now() + timedelta(hours=2)
h12 = t.hour % 12 or 12
ampm = "am" if t.hour < 12 else "pm"
banner = f"weekly limit · resets at {h12}:{t.minute:02d}{ampm}"
until, parsed = agents._next_limit_until(self.cfg, banner, now)
self.assertTrue(parsed)
self.assertGreater(until, now)
self.assertLessEqual(until - now, 6 * 3600 + 60) # within 6h window (+slack)
def test_next_limit_until_far_future_falls_back(self):
now = time.time()
t = datetime.now() + timedelta(hours=7) # > 6h window
h12 = t.hour % 12 or 12
ampm = "am" if t.hour < 12 else "pm"
banner = f"limit · resets at {h12}:{t.minute:02d}{ampm}"
until, parsed = agents._next_limit_until(self.cfg, banner, now)
self.assertFalse(parsed)
self.assertEqual(int(until), int(now + 300))
# ── stall / WAITING-UNTIL parsing ──────────────────────────────────────────────────
class TestWaitingUntil(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
self.claude_agent = self.cfg["agents"]["cl"] # non-footer backend
self.oc_agent = self.cfg["agents"]["oc"] # footer_ui backend
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
def test_non_footer_finds_marker_anywhere(self):
pane = "blah blah\nWAITING-UNTIL: 2030-06-13T12:00:00Z\nmore output after\n"
ep = agents._parse_waiting_until(self.cfg, self.claude_agent, pane)
self.assertIsNotNone(ep)
self.assertEqual(ep, datetime.fromisoformat("2030-06-13T12:00:00+00:00").timestamp())
def test_non_footer_none_without_marker(self):
self.assertIsNone(agents._parse_waiting_until(
self.cfg, self.claude_agent, "just working, no marker"))
def test_footer_honors_marker_above_the_footer(self):
# A footer_ui backend renders its input-box/status footer BELOW the agent's message, so the
# marker is never the literal last line. It must still be honored (this is the real-claude case).
pane = "WAITING-UNTIL: 2030-06-13T12:00:00Z\n ▣ Build · GPT · 2m 19s\n"
ep = agents._parse_waiting_until(self.cfg, self.oc_agent, pane)
self.assertIsNotNone(ep)
self.assertEqual(ep, datetime.fromisoformat("2030-06-13T12:00:00+00:00").timestamp())
def test_takes_most_recent_marker(self):
pane = ("WAITING-UNTIL: 2030-01-01T00:00:00Z\nwork\n"
"WAITING-UNTIL: 2031-06-13T12:00:00Z\n footer\n")
ep = agents._parse_waiting_until(self.cfg, self.oc_agent, pane)
self.assertEqual(ep, datetime.fromisoformat("2031-06-13T12:00:00+00:00").timestamp())
def test_bad_timestamp_none(self):
self.assertIsNone(agents._parse_waiting_until(
self.cfg, self.claude_agent, "WAITING-UNTIL: not-a-time"))
# ── backend activity detectors (claude + opencode footers) ──────────────────────────
class TestActivityDetection(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
self.claude_agent = self.cfg["agents"]["cl"]
self.oc_agent = self.cfg["agents"]["oc"]
def tearDown(self):
shutil.rmtree(self.tmp, ignore_errors=True)
# claude: non-footer, active_re matched anywhere in the pane
def test_claude_active_esc_to_interrupt(self):
self.assertTrue(agents.pane_active(
self.cfg, self.claude_agent, "thinking...\n esc to interrupt", use_log=False))
def test_claude_active_running_tool(self):
self.assertTrue(agents.pane_active(
self.cfg, self.claude_agent, "Running tool: Bash", use_log=False))
def test_claude_active_spinner_dot_count(self):
self.assertTrue(agents.pane_active(
self.cfg, self.claude_agent, "Compiling · 137 tokens", use_log=False))
def test_claude_idle_is_not_active(self):
self.assertFalse(agents.pane_active(
self.cfg, self.claude_agent, "Done.\n> ", use_log=False))
# opencode: footer_ui — only the bottom rows count as activity
def test_opencode_active_footer(self):
pane = "~ Preparing patch...\n ⬝⬝■ esc interrupt 137.6K\n"
self.assertTrue(agents.pane_active(self.cfg, self.oc_agent, pane, use_log=False))
def test_opencode_idle_footer_not_active(self):
pane = " ▣ Build · GPT-5.4 · 2m 19s\n 178.4K (17%) ctrl+p commands\n"
self.assertFalse(agents.pane_active(self.cfg, self.oc_agent, pane, use_log=False))
def test_opencode_active_only_at_top_is_ignored(self):
# active marker far above the bottom 10 lines → a footer UI ignores it
pane = "running tool now\n" + "\n".join(f"line {i}" for i in range(20)) + \
"\n ▣ Build · GPT · idle\n"
self.assertFalse(agents.pane_active(self.cfg, self.oc_agent, pane, use_log=False))
def test_opencode_log_grace_fallback(self):
# idle footer, but a freshly-touched session log within the grace window → active
idle = " ▣ Build · GPT · idle\n 178K (17%) ctrl+p\n"
logp = agents._session_log_path(self.cfg, self.oc_agent["session"])
logp.parent.mkdir(parents=True, exist_ok=True)
logp.write_text("recent activity\n") # mtime = now
self.assertTrue(agents.pane_active(self.cfg, self.oc_agent, idle, use_log=True))
# remove the log → no fallback → idle footer reads as not active
logp.unlink()
self.assertFalse(agents.pane_active(self.cfg, self.oc_agent, idle, use_log=True))
# ── build-aware stall detection ─────────────────────────────────────────────────────
# A silent pane whose claude session has a real compile/coverage/test process running is a
# running build, NOT a stall. These cover the three moving parts: the process-name match set,
# the session-scoped descendant walk, _build_running end-to-end, and stall_check_one's defer.
class TestBuildProcRegex(unittest.TestCase):
"""High-signal build/test tool names match; generic interpreters + the harness itself must
NOT (bare python/node/bash would false-positive on the orchestrator's own engine, and the
claude root's args embed the prompt which names cargo/rustc/…)."""
def setUp(self):
self.rx = re.compile(agents.DEFAULT_BUILD_PROCS_RE)
def test_build_tools_match(self):
for name in ["cargo", "cargo-llvm-cov", "cargo-mutants", "cargo-nextest", "nextest",
"rustc", "rustdoc", "cc1", "cc1plus", "collect2", "lld", "llvm-cov",
"llvm-profdata", "lichen-server", "lichen-cms", "lichen-shell",
"lichen-cli", "chromium", "chrome", "playwright"]:
self.assertTrue(self.rx.match(name), f"{name!r} should be treated as a build proc")
def test_generic_procs_do_not_match(self):
for name in ["python", "python3", "bash", "sh", "node", "claude", "tmux",
"sleep", "grep", "git", "ssh", "vim", "pgrep", "ps"]:
self.assertIsNone(self.rx.match(name), f"{name!r} must NOT be treated as a build proc")
def test_anchored_no_substring_false_positives(self):
# ^...$ anchored — a longer name merely containing a tool word must not match
for name in ["cargotool", "xrustc", "mycc1", "playwrightish", "notchromium", "rustcx"]:
self.assertIsNone(self.rx.match(name), f"{name!r} must not match (substring)")
class TestProcDescendants(unittest.TestCase):
"""_proc_descendants returns the real child tree of the given roots and EXCLUDES the roots."""
def test_finds_children_excludes_root(self):
p = subprocess.Popen("sleep 30 & sleep 30 & wait", shell=True, start_new_session=True)
try:
time.sleep(0.4) # let the two children spawn
kids = agents._proc_descendants([str(p.pid)])
self.assertNotIn(str(p.pid), kids) # root excluded
self.assertGreaterEqual(len(kids), 2) # the two sleeps
# Read /proc/<pid>/comm rather than shelling to `ps -o comm=`: this host's ps
# returns nothing for that form, and an empty result from a tool that is absent or
# unsupported is indistinguishable from "the children are not sleeps". Same class of
# bug as the pgrep dependency this test just caught in _proc_descendants.
comms = []
for k in sorted(kids):
try:
comms.append(open(f"/proc/{k}/comm").read().strip())
except OSError:
pass
self.assertIn("sleep", comms, f"expected a sleep among {comms}")
finally:
os.killpg(os.getpgid(p.pid), signal.SIGKILL); p.wait()
def test_leaf_process_has_no_descendants(self):
p = subprocess.Popen(["sleep", "30"], start_new_session=True)
try:
time.sleep(0.2)
self.assertEqual(agents._proc_descendants([str(p.pid)]), set())
finally:
os.killpg(os.getpgid(p.pid), signal.SIGKILL); p.wait()
class TestBuildRunning(unittest.TestCase):
"""_build_running is scoped to the watched session's pane_pid descendants (never the root
claude process) and matches on process comm."""
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
self.agent = self.cfg["agents"]["cl"]
self._orig_run = agents.subprocess.run
self.ps_targets = ""
def tearDown(self):
# Restore EXPLICITLY. A cleverer derivation of these names silently matched nothing, so the
# monkeypatch leaked into the next test class and made an unrelated test fail — visible only
# in a full run, never when that test was run alone.
agents.subprocess.run = self._orig_run
shutil.rmtree(self.tmp, ignore_errors=True)
def _patch(self, descendant_comms):
"""pane_pid=1000; its children are 1001,1002 with the given comms.
Patches the two internal seams (_proc_descendants, _comms) rather than faking
`pgrep`/`ps` subprocess calls. WHY (2026-08-21): the previous version mocked those two
commands, so the suite passed on hosts where NEITHER IS INSTALLED — the fake stood in for
the broken dependency and the real bug (empty output read as "no build running", which
lets the watchdog reboot an agent mid-build) was invisible to every test. Mock the seam
you own, not the tool you depend on, or the test proves the mock works.
"""
outer = self
class R:
def __init__(self, out): self.stdout = out; self.returncode = 0
def fake_run(cmd, *a, **k):
return R("1000\n") if "list-panes" in cmd else R("")
agents.subprocess.run = fake_run
orig_desc, orig_comms = agents._proc_descendants, agents._comms
def _restore():
agents._proc_descendants, agents._comms = orig_desc, orig_comms
self.addCleanup(_restore) # runs even if the test errors; no tearDown ordering to get wrong
agents._proc_descendants = lambda roots: (
{"1001", "1002"} if "1000" in list(roots) else set())
def fake_comms(pids):
outer.ps_targets = ",".join(sorted(pids))
return list(descendant_comms)
agents._comms = fake_comms
def test_detects_running_build(self):
self._patch(["bash", "cargo"])
self.assertTrue(agents._build_running(self.cfg, self.agent))
def test_no_build_when_only_shells(self):
self._patch(["bash", "vim"])
self.assertFalse(agents._build_running(self.cfg, self.agent))
def test_only_inspects_descendants_not_the_claude_root(self):
self._patch(["bash", "cargo"])
agents._build_running(self.cfg, self.agent)
self.assertNotIn("1000", self.ps_targets) # root pane_pid never ps-inspected
self.assertIn("1001", self.ps_targets)
def test_custom_build_procs_re_override(self):
cfg = dict(self.cfg)
cfg["watchdog"] = dict(self.cfg["watchdog"], build_procs_re=r"^mybuild$")
self._patch(["mybuild"])
self.assertTrue(agents._build_running(cfg, self.agent))
self._patch(["cargo"]) # default tool no longer counts
self.assertFalse(agents._build_running(cfg, self.agent))
class TestBuildAwareStall(unittest.TestCase):
"""stall_check_one defers the kill+reboot while a build runs, up to the stall_idle_max cap."""
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="aotest-ut-")
self.cfg = agents.load_config(_make_project(self.tmp))
self.agent = self.cfg["agents"]["cl"] # stall_idle=300, stall_idle_max default 1800
self.session = self.agent["session"]
self.reboots = []
self._orig = {}
def patch(name, fn):
if name not in self._orig: # capture the TRUE original once, so a
self._orig[name] = getattr(agents, name) # test that re-patches doesn't leak the
setattr(agents, name, fn) # earlier patch into tearDown's restore
patch("session_alive", lambda s: True)
patch("capture_pane", lambda *a, **k: "")
patch("limit_tick", lambda *a, **k: False)
patch("pane_active", lambda *a, **k: False)
patch("_parse_waiting_until", lambda *a, **k: None)
patch("_pane_last_active", lambda s: None)
patch("phases", lambda cfg: []) # skip the DONE-nudge branch
patch("log", lambda *a, **k: None)
patch("start_agent", lambda cfg, a, force=False: self.reboots.append(a["name"]))
self.patch = patch
agents._idle_since.clear(); agents._build_deferred.discard(self.session)
def tearDown(self):
for name, fn in self._orig.items():
setattr(agents, name, fn)
agents._idle_since.clear(); agents._build_deferred.discard(self.session)
shutil.rmtree(self.tmp, ignore_errors=True)
def _set_idle(self, seconds):
agents._idle_since[self.session] = time.time() - seconds
def test_defers_reboot_while_building(self):
self.patch("_build_running", lambda *a, **k: True)
self._set_idle(700) # > stall_idle(300), < max(1800)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, []) # deferred, not rebooted
self.assertIn(self.session, agents._build_deferred)
def test_reboots_when_idle_and_no_build(self):
self.patch("_build_running", lambda *a, **k: False)
self._set_idle(700)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, [self.agent["name"]]) # genuinely idle → rebooted
def test_hard_cap_reboots_even_while_building(self):
self.patch("_build_running", lambda *a, **k: True)
self._set_idle(2000) # > stall_idle_max(1800)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, [self.agent["name"]]) # cap reached → reboot despite build
def test_waiting_until_defers_reboot(self):
# agent signalled a remote run in progress → hold off even with no local build + long idle
self.patch("_build_running", lambda *a, **k: False)
self.patch("_parse_waiting_until", lambda *a, **k: time.time() + 600)
self._set_idle(3000)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, []) # deferred until the stated deadline
def test_waiting_until_capped_reboots_when_no_build(self):
# a runaway that parked itself and is doing NOTHING still reboots at the cap ("max no matter
# what"), however far out its stated deadline is
self.cfg["watchdog"]["waiting_until_max"] = 7200
self.patch("_build_running", lambda *a, **k: False)
self.patch("_parse_waiting_until", lambda *a, **k: time.time() + 100000)
self._set_idle(8000) # idle > cap(7200), nothing running
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, [self.agent["name"]]) # idle past cap, no build → rebooted
def test_live_build_defers_past_the_stated_deadline(self):
# the deadline is only the agent's ESTIMATE, and an agent blocked on a shell cannot re-emit a
# fresh marker — so a still-running build (cargo-mutants overrunning its guess) must not be
# killed at the estimate. Proof of life outranks the deadline.
self.cfg["watchdog"]["waiting_until_max"] = 7200
self.patch("_build_running", lambda *a, **k: True)
self.patch("_parse_waiting_until", lambda *a, **k: time.time() - 1000) # deadline already past
self._set_idle(3000) # but still under the absolute cap
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, []) # build running → deferred
def test_past_deadline_with_no_build_reboots(self):
# no build, deadline blown → the self-wake did not fire → reboot
self.patch("_build_running", lambda *a, **k: False)
self.patch("_parse_waiting_until", lambda *a, **k: time.time() - 1000)
self._set_idle(3000)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, [self.agent["name"]])
def test_cap_is_absolute_and_reboots_a_hung_build(self):
# the cap is the ONE absolute bound: past it we reboot even mid-build — that is precisely how
# a genuinely HUNG build gets caught, and why a runaway can never park forever
self.cfg["watchdog"]["waiting_until_max"] = 7200
self.patch("_build_running", lambda *a, **k: True)
self.patch("_parse_waiting_until", lambda *a, **k: time.time() + 100000)
self._set_idle(8000) # idle > cap, build "running" (hung)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, [self.agent["name"]])
def test_no_build_check_below_base_threshold(self):
def boom(*a, **k):
raise AssertionError("_build_running must not be consulted below stall_idle")
self.patch("_build_running", boom)
self._set_idle(100) # < stall_idle(300)
agents.stall_check_one(self.cfg, self.agent)
self.assertEqual(self.reboots, [])
if __name__ == "__main__":
unittest.main(verbosity=2)
+247
View File
@@ -0,0 +1,247 @@
#!/usr/bin/env python3
"""gateway-domain.py — point a public domain at your box, via the testing gateway.
WHAT THIS IS FOR. Your machine is on the tailnet but has no public IP, and you need a real
HTTPS domain for it — an OAuth callback, a webhook receiver, a demo someone else can open.
The testing gateway at gtest.commoninternet.net already holds a wildcard DNS record, so
*.gtest.commoninternet.net resolves to it. Map your name to your tailnet IP and the gateway
streams matching connections to you:
browser --TLS--> gateway :443 --reads SNI--> your box over the tailnet
The gateway never terminates that TLS. It reads the SNI name and proxies the still-encrypted
bytes, so **your box serves the certificate**, not the gateway. See SKILL.md.
USAGE
tools/gateway-domain.py list
tools/gateway-domain.py add myapp # -> myapp.gtest.commoninternet.net -> this box's tailscale IP
tools/gateway-domain.py add myapp 100.64.1.5 # explicit backend
tools/gateway-domain.py add myapp 100.64.1.5:8443 # backend listening somewhere other than 443
tools/gateway-domain.py remove myapp
CREDENTIALS. The admin password is in the orchestrator secret store, never here:
python3 engine/secrets.py get gateway.admin_password
This script reads it from there itself. You should not need to handle the value.
"""
import argparse
import base64
import json
import os
import re
import shutil
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
# engine/secrets.py lives one directory up.
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
import secrets as secret_store # noqa: E402 (engine/secrets.py, not the stdlib module)
TIMEOUT = 30
def _cfg():
"""Gateway coordinates. Only the password is genuinely secret; the rest lives
alongside it so there is one thing to change if the gateway ever moves."""
return (
secret_store.get("gateway.fqdn", "gtest.commoninternet.net"),
secret_store.get("gateway.admin_user", "admin"),
secret_store.get("gateway.admin_password"),
)
def _request(method="GET", form=None):
fqdn, user, password = _cfg()
if not password:
sys.exit(
"no gateway.admin_password in the secret store.\n"
" check: python3 engine/secrets.py list\n"
" see: skills/gateway-domain/SKILL.md"
)
url = f"https://{fqdn}/admin/"
data = urllib.parse.urlencode(form).encode() if form else None
req = urllib.request.Request(url, data=data, method=method)
token = base64.b64encode(f"{user}:{password}".encode()).decode()
req.add_header("Authorization", f"Basic {token}")
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
return r.status, r.read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", "replace")
if e.code == 401:
sys.exit(
f"401 from {url} — the stored password is not what the gateway expects.\n"
"The e2e suite reseeds .htpasswd while it runs and restores it afterwards;\n"
"if a run was interrupted, /var/lib/tunnel-gateway/.htpasswd.e2e-backup is\n"
"still on the box. See SKILL.md, 'When the password stops working'."
)
sys.exit(f"HTTP {e.code} from {url}\n{body[:600]}")
except urllib.error.URLError as e:
sys.exit(f"cannot reach {url}: {e.reason}")
# The admin app answers 200 and renders a red <p> for a rejected value, rather than
# using a status code. Checking only the status would report success on a no-op.
_ERR = re.compile(r'<p style="color:red">(.*?)</p>', re.S)
# Backends are restricted to a literal IPv4 address, optionally with a port.
#
# The gateway would accept a hostname -- its validate_ip is ^[a-zA-Z0-9_.-]+$ -- but its
# put_domain/remove_domain match existing lines with ^<domain>\s+[\d.:]+; which only ever
# matches a numeric backend. A hostname mapping can therefore be written once and then never
# updated or removed through the admin UI: it becomes an orphan only a hand-edit of
# /var/lib/tunnel-gateway/tunnel_map.conf can clear. Refuse to create one.
_BACKEND = re.compile(r"^(\d{1,3}(?:\.\d{1,3}){3})(?::(\d{1,5}))?$")
# The tailnet ACL only lets the gateway open connections to nodes carrying one of these.
# A mapping to an untagged node is accepted by the gateway and then simply never
# connects, which looks like a gateway fault and is not one.
ALLOWED_TAGS = ("tag:notplants-test-server", "tag:orchestrator")
def _validate_backend(backend):
m = _BACKEND.match(backend)
if not m:
sys.exit(
f"backend must be an IPv4 address, optionally IP:port -- got {backend!r}.\n"
"Hostnames are rejected on purpose: the gateway can store one but cannot\n"
"remove it again. Use the tailnet IP (tailscale ip -4 on the target box)."
)
ip, port = m.group(1), m.group(2)
if any(int(o) > 255 for o in ip.split(".")):
sys.exit(f"not a valid IPv4 address: {ip}")
if port is not None and not (1 <= int(port) <= 65535):
sys.exit(f"port out of range: {port}")
return backend
def _check(body):
m = _ERR.search(body)
if m:
sys.exit(f"gateway rejected the request: {m.group(1).strip()}")
def _parse_domains(body):
"""Pull the Domain Mappings table out of the admin page."""
section = body.split("Domain Mappings", 1)[-1].split("Port Mappings", 1)[0]
return re.findall(r"<td>\s*(.*?)\s*</td>\s*<td>\s*(.*?)\s*</td>", section, re.S)
def _self_tags(exe):
"""This node's tailnet tags, or None if they cannot be determined."""
try:
out = subprocess.run([exe, "status", "--json"], capture_output=True, text=True, timeout=15)
return json.loads(out.stdout).get("Self", {}).get("Tags") or []
except Exception:
return None
def _warn_untagged(exe):
tags = _self_tags(exe)
if tags is None:
return
if not any(t in tags for t in ALLOWED_TAGS):
print(
f"warning: this node carries none of {' / '.join(ALLOWED_TAGS)} "
f"(tags: {', '.join(tags) or 'none'}).\n"
" The gateway will accept the mapping but the tailnet ACL will not let it\n"
" reach this box, so no traffic will flow. Add the tag in the Tailscale\n"
" admin, or map a backend that already has it.",
file=sys.stderr,
)
def _tailscale_ip():
exe = shutil.which("tailscale") or "/run/current-system/sw/bin/tailscale"
try:
out = subprocess.run([exe, "ip", "-4"], capture_output=True, text=True, timeout=15)
except (OSError, subprocess.SubprocessError) as e:
sys.exit(f"could not run tailscale to detect this box's IP ({e}); pass the backend explicitly")
_warn_untagged(exe)
ip = out.stdout.strip().splitlines()[0].strip() if out.stdout.strip() else ""
if not ip:
sys.exit(
"tailscale reported no IPv4 address — is this box on the tailnet?\n"
f" {exe} status\n"
"Or pass the backend explicitly: gateway-domain.py add <name> <ip>"
)
return ip
def _qualify(name, fqdn):
"""A bare label becomes a subdomain of the gateway; anything with a dot is used as given."""
return name if "." in name else f"{name}.{fqdn}"
def cmd_list(args):
_, body = _request()
rows = _parse_domains(body)
if not rows:
print("no domain mappings")
return
width = max(len(d) for d, _ in rows)
for domain, backend in rows:
print(f" {domain:<{width}} -> {backend}")
def cmd_add(args):
fqdn, _, _ = _cfg()
domain = _qualify(args.name, fqdn)
backend = args.backend or _tailscale_ip()
_validate_backend(backend)
_, body = _request("POST", {"domain": domain, "ip": backend})
_check(body)
# Re-read rather than trusting the POST body: app.py mutates its in-memory dict and
# renders that, so the response shows what it meant to do, not what landed on disk.
_, fresh = _request()
if not any(d == domain for d, _ in _parse_domains(fresh)):
sys.exit(f"POST returned 200 but {domain} is not in the mapping table — check `list`")
print(f" {domain} -> {backend}")
print(f"\nYour box must now serve TLS for {domain} on the backend port (443 unless you")
print("set one). The gateway does not terminate TLS; it proxies the encrypted stream.")
def cmd_remove(args):
fqdn, _, _ = _cfg()
domain = _qualify(args.name, fqdn)
# The admin app treats a backend of exactly "0" as delete.
_, body = _request("POST", {"domain": domain, "ip": "0"})
_check(body)
_, fresh = _request()
if any(d == domain for d, _ in _parse_domains(fresh)):
sys.exit(
f"{domain} is still in the mapping table after the delete.\n"
"If its backend is a hostname rather than an IP, the gateway cannot remove it:\n"
"edit /var/lib/tunnel-gateway/tunnel_map.conf on the box and reload nginx."
)
print(f" removed {domain}")
def main():
ap = argparse.ArgumentParser(
description="Map a public domain to your tailnet box via the testing gateway.",
epilog="Password comes from the secret store (gateway.admin_password); see skills/gateway-domain/SKILL.md.",
)
sub = ap.add_subparsers(dest="cmd", required=True)
sub.add_parser("list", help="show current domain mappings").set_defaults(fn=cmd_list)
a = sub.add_parser("add", help="point a domain at a backend")
a.add_argument("name", help="bare label (myapp) or a full domain")
a.add_argument("backend", nargs="?", help="IP or IP:port (default: this box's tailscale IP, port 443)")
a.set_defaults(fn=cmd_add)
r = sub.add_parser("remove", help="delete a domain mapping")
r.add_argument("name", help="bare label (myapp) or a full domain")
r.set_defaults(fn=cmd_remove)
args = ap.parse_args()
args.fn(args)
if __name__ == "__main__":
main()
+149
View File
@@ -0,0 +1,149 @@
#!/usr/bin/env bash
# secrets.sh — safe wrapper for sops-encrypted per-host secrets in /secrets/<host>/.
#
# Shared agent tooling: any agent on any orchestrator can use this. Pick the host with
# SECRETS_HOST (default b1), e.g. SECRETS_HOST=b1 engine/tools/secrets.sh verify
#
# Layout it expects:
# /secrets/<host>/<host>.yaml sops-encrypted payload
# /secrets/<host>/.sops.yaml recipients
# /secrets/<host>/admin-age.key the admin private key (0600)
#
# WHY THIS EXISTS. On 2026-08-16 a hand-rolled sops pipeline did this:
#
# sops -d b1.yaml > /tmp/p.yaml # decrypt
# echo "new_key: value" >> /tmp/p.yaml
# cp /tmp/p.yaml b1.yaml # <-- plaintext written to the REAL path
# sops -e -i b1.yaml # <-- FAILED (creation_rules path mismatch)
#
# sops exited non-zero, the `cp` had already happened, and the plaintext file was then copied
# to the host — world-readable — before anyone noticed. Every secret in it was exposed.
#
# The invariant here: **plaintext never exists at the destination path.** All mutation happens
# on a temp file inside a 0700 directory; the result is encrypted, verified to be ciphertext,
# round-tripped through a decrypt, and only then moved into place atomically. Any failure at
# any step leaves the original file untouched.
set -euo pipefail
# PATH hardening. This script has already been bitten by a tool "not existing" merely because
# it was not on PATH — on NixOS, sops/nix live in /run/current-system/sw/bin and setuid wrappers
# in /run/wrappers/bin, neither of which is guaranteed in a non-login shell. A false "command not
# found" here reads as "decryption failed", which is exactly the wrong conclusion to draw.
export PATH="/run/wrappers/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:$PATH"
HOST="${SECRETS_HOST:-b1}"
DIR="/secrets/$HOST"
FILE="$DIR/$HOST.yaml"
AGE_KEY="$DIR/admin-age.key"
die() { echo "error: $*" >&2; exit 1; }
have_sops() {
if command -v sops >/dev/null 2>&1; then SOPS=(sops)
elif command -v nix >/dev/null 2>&1; then
SOPS=(nix --extra-experimental-features "nix-command flakes" shell nixpkgs#sops -c sops)
else
die "neither sops nor nix found on PATH ($PATH)"
fi
}
is_encrypted() { grep -qE 'ENC\[AES256_GCM' "$1" 2>/dev/null; }
# Every mutation goes through here. It refuses to install anything that is not verified ciphertext.
install_encrypted() {
local tmp=$1
is_encrypted "$tmp" || die "refusing to install: result is NOT encrypted (this is the bug this script exists to prevent)"
SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$tmp" >/dev/null 2>&1 \
|| die "refusing to install: encrypted file does not decrypt with $AGE_KEY"
chmod 600 "$tmp"
mv -f "$tmp" "$FILE" # atomic within the same filesystem
echo "ok: $FILE updated ($(grep -c 'recipient:' "$FILE") recipients)"
}
workdir() { local d; d=$(mktemp -d "$DIR/.work.XXXXXX"); chmod 700 "$d"; echo "$d"; }
# NB: must return 0. As an EXIT trap its status becomes the script's status, and an
# `[ -n "$WD" ] && ...` that is simply false would make every read-only command (get/list/verify)
# exit 1 while printing a perfectly correct answer — a silent false failure in callers.
scrub() {
if [ -n "${WD:-}" ]; then
find "$WD" -type f -exec shred -u {} + 2>/dev/null || true
rm -rf "$WD"
fi
return 0
}
trap scrub EXIT
have_sops
[ -r "$DIR/.sops.yaml" ] || die "no $DIR/.sops.yaml — cannot know who may decrypt"
[ -d "$DIR" ] || die "no such secrets dir: $DIR"
[ -f "$FILE" ] || die "no such secrets file: $FILE"
cmd="${1:-help}"; shift || true
case "$cmd" in
list) # key names only, never values
SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" | grep -oE '^[a-zA-Z0-9_]+:' | tr -d ':' ;;
get) # print ONE value to stdout, for piping. Nothing is written to disk.
[ $# -ge 1 ] || die "usage: $0 get <key>"
SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d --extract "[\"$1\"]" "$FILE" ;;
set) # set <key> <file|-> value read from a file or stdin
[ $# -ge 2 ] || die "usage: $0 set <key> <file|->"
key=$1; src=$2
WD=$(workdir); p="$WD/plain.yaml"; e="$WD/enc.yaml"
SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" > "$p"
# drop any existing definition of this key (scalar or block)
awk -v k="$key" 'BEGIN{skip=0}
$0 ~ "^"k":" {skip=1; next}
skip==1 && /^[[:space:]]/ {next}
{skip=0; print}' "$p" > "$p.new" && mv "$p.new" "$p"
if [ "$src" = "-" ]; then val=$(cat); else [ -r "$src" ] || die "cannot read $src"; val=$(cat "$src"); fi
if [ "$(printf '%s' "$val" | wc -l)" -gt 0 ]; then
{ echo "$key: |"; printf '%s\n' "$val" | sed 's/^/ /'; } >> "$p" # multi-line block
else
printf '%s: %s\n' "$key" "$val" >> "$p"
fi
cp "$p" "$e"
# --filename-override makes creation_rules match regardless of the temp path. This is the
# exact failure that caused the incident: the rule keyed on the real filename, the temp file
# did not match, and encryption silently refused.
"${SOPS[@]}" --config "$DIR/.sops.yaml" -e -i --filename-override "$FILE" "$e"
install_encrypted "$e" ;;
unset) # remove a key entirely
[ $# -ge 1 ] || die "usage: $0 unset <key>"
key=$1
WD=$(workdir); p="$WD/plain.yaml"; e="$WD/enc.yaml"
SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" > "$p"
awk -v k="$key" 'BEGIN{skip=0}
$0 ~ "^"k":" {skip=1; next}
skip==1 && /^[[:space:]]/ {next}
{skip=0; print}' "$p" > "$e"
"${SOPS[@]}" --config "$DIR/.sops.yaml" -e -i --filename-override "$FILE" "$e"
install_encrypted "$e" ;;
edit)
SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" "$FILE"
is_encrypted "$FILE" || die "file is not encrypted after edit — restore from git/backup NOW" ;;
verify)
is_encrypted "$FILE" && echo " encrypted: yes" || die "NOT ENCRYPTED: $FILE"
echo " recipients: $(grep -c 'recipient:' "$FILE")"
echo " decrypts: $(SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" >/dev/null 2>&1 && echo yes || echo NO)"
echo " keys: $(SOPS_AGE_KEY_FILE="$AGE_KEY" "${SOPS[@]}" -d "$FILE" | grep -oE '^[a-zA-Z0-9_]+:' | tr -d ':' | tr '\n' ' ')"
n=$(grep -cE 'tskey-auth-[A-Za-z0-9]{5}|BEGIN OPENSSH PRIVATE KEY' "$FILE" || true)
[ "$n" -eq 0 ] && echo " plaintext leaks: none" || die "PLAINTEXT SECRETS PRESENT ($n)" ;;
deploy) # deploy <user@host> [remote-path] — refuses to ship anything unencrypted
[ $# -ge 1 ] || die "usage: $0 deploy <user@host> [remote-path]"
target=$1; rpath=${2:-/etc/nixos/secrets/$HOST.yaml}
is_encrypted "$FILE" || die "refusing to deploy: local file is not encrypted"
# SSH_OPTS lets the caller pass -i/-o without this script guessing at key locations.
# shellcheck disable=SC2086
scp -q ${SSH_OPTS:-} "$FILE" "$target:$rpath" || die "scp failed"
ssh ${SSH_OPTS:-} "$target" "chmod 600 '$rpath'" || die "chmod failed"
ssh ${SSH_OPTS:-} "$target" "grep -qE 'ENC\[AES256_GCM' '$rpath'" \
&& echo "ok: deployed and verified encrypted at $target:$rpath" \
|| die "remote file is not encrypted after deploy" ;;
*) sed -n '2,30p' "$0"; echo; echo "commands: list | get <key> | set <key> <file|-> | edit | verify | deploy <user@host> [path]" ;;
esac