agent-orchestrator/README.md

# agent-orchestrator

A generic, reusable harness for running and supervising a fleet of AI-agent sessions in **tmux**.
One driver script + one declarative config (`agents.toml`) describe every agent — a Builder /
Adversary loop pair, a persistent supervisor, a one-shot task — and a **watchdog** keeps them
alive, healed, paced, and coordinated. The watchdog reads the same config every tick, so there is
never any env-vs-file drift.

Nothing about any particular project lives in this repo. Paths, the loop **kickoff preamble**, the
**handoff conventions**, and the **on-complete** hook are all supplied by the project's config and
prompt files. A project consumes this repo as a pinned **git submodule** (`engine/`) and keeps its
own config, prompts, state, and tmux namespace — total isolation between projects.

```
agents.py            the driver + watchdog (pure Python stdlib; needs python >= 3.11 for tomllib)
agent-log.py         render claude JSONL transcripts into clean, greppable logs
agents.example.toml  a self-contained 2-agent example project
prompts/             generic role + kickoff templates (builder / adversary / kickoff)
examples/            runnable example projects — the Builder/Adversary variant family, snakepit, …
smoke.sh             bring the example up + tear it down in an isolated sandbox, then clean up
tests/               the test suite — unit tests + isolated live backend smokes + a runner
flake.nix/.lock      a Nix devShell with the runtime deps (python311, tmux, git)
```

---

## Quick start

```bash
nix develop                                   # python311 + tmux + git on PATH (see "Nix" below)

python3 agents.py selftest                    # regression-test the activity detector (no config)
python3 agents.py status --config agents.example.toml   # one table: every agent + the phase
./smoke.sh                                    # prove up/down works end-to-end, isolated + clean

python3 agents.py init myproject              # scaffold a starter agents.toml + prompts/
```

`up` is **use-or-create**: an already-running session is left alone, never double-started.

```bash
python3 agents.py --config agents.toml up           # start all enabled agents + services + watchdog
python3 agents.py --config agents.toml up builder    # start just one agent (by name)
python3 agents.py --config agents.toml down          # stop everything
python3 agents.py --config agents.toml logs builder  # tail one session's log
python3 agents.py --config agents.toml phase show    # where the loop phase machine is
```

`--config` defaults to `./agents.toml`, falling back to one next to `agents.py`.

---

## Examples

`examples/` holds runnable example projects — copy one, point `agents.py` at its `agents.toml`, and
go. The headline set is a family of **Builder/Adversary** variants that build the *same* task but each
differ in one dimension — useful both as templates and as a study of the pattern:

- **`builder-adversary`** — the canonical loop pair: a Builder that builds and an Adversary that
  cold-verifies every claim, coordinating only through git (`claim(`/`review(` commits + the watchdog
  handoff). **Start here.**
- **`builder-adversary-min`** — the same pattern with the prompts compressed to minimal tokens.
- **`builder-adversary-stateless`** — `builder-adversary` + **context hygiene** (compact at each
  checkpoint, read diffs not trees, lean loads) to minimise carried/reloaded context.
- **`builder-adversary-lean`** — context hygiene + **per-gate** review (one claim/verdict per gate).
- **`builder-adversary-deferred`** — the Adversary verifies **once**, after the whole build, in a
  final comprehensive `review` phase (vs per-phase / per-gate).
- **`builder-solo`** — a single Builder that self-certifies, with **no Adversary** (the control).
- **`snakepit`** — a different topology entirely: a pool of identical worker "snakes" pulling tasks
  from a shared filesystem queue, plus cleanup specialists. (`examples/IDEAS.md` sketches more.)

Each example has its own `README.md`. Run one by hand:

```bash
cd examples/builder-adversary
python3 ../../agents.py status --config agents.toml      # read-only
python3 ../../agents.py up     --config agents.toml       # needs `claude` on PATH
```

**Benchmark.** The separate
[`agent-orchestrator-benchmark`](https://git.autonomic.zone/recipe-maintainers/agent-orchestrator-benchmark)
repo runs these Builder/Adversary variants head-to-head (N=5, real `agents.py up` runs) to measure
what drives token cost. Short version: an independent adversary costs **~4.7×** a solo builder, but
the review *cadence* (per-gate / per-phase / deferred) is **nearly token-neutral**, and **context
hygiene** is the one clean **~−22%** win. See that repo's `FINDINGS.md`.

---

## The config: `agents.toml`

Five section types: `[watchdog]`, `[backend.<name>]`, `[defaults]`, `[[agent]]` / `[[service]]`,
and `[loop]`. See `agents.example.toml` for a complete, runnable example.

### `[watchdog]` — global supervisor cadence

```toml
[watchdog]
signal_interval      = 30    # seconds between light checks (handoff / stall / limit)
heavy_interval       = 300   # seconds between heal + phase-advance checks
limit_probe_fallback = 300   # re-probe cadence for a usage-limited agent when reset time is unparsable
limit_reset_slack    = 45    # seconds to wait past a parsed reset before probing
stall_grace          = 180   # seconds of slack past a WAITING-UNTIL marker before a stall reboot
log_tokens           = false # opt-in: record per-phase token + time usage (see below)
```

**Per-phase token + time logging (`log_tokens`).** Set `log_tokens = true` (under `[watchdog]` or
`[loop]`) and the watchdog records, for **each phase**, how many tokens **each agent** used and how
long the phase took — appended as one JSON object per phase to `<log_dir>/token-log.jsonl`. Tokens
are summed from each agent's Claude Code session transcript and attributed **by working dir**, so
give each agent its own `dir` (the Builder/Adversary loop pair already uses separate clones) for
accurate per-agent numbers. The watchdog snapshots a baseline when a phase starts and writes the
delta (per agent, and the total) when the phase advances or the sequence completes — robust across
watchdog restarts. Pretty-print it with `agents.py tokens`:

```
phase        dur(s)   builder adversary         TOTAL
-----------------------------------------------------
lex           372.0 3,910,118 3,221,447     7,131,565
parse         410.5 ...
```

### `[defaults]` — inherited by every agent

```toml
[defaults]
session_prefix = "myproj-"   # REQUIRED: tmux namespace for this project. No implicit default.
log_dir        = ".ao-state" # REQUIRED: logs + state/. Relative paths resolve against the config dir.
backend        = "claude"
model          = "claude-sonnet-4-6"
dir            = "."         # default working dir for agents (relative → project dir)
watch          = "heal"      # none | heal | heal+stall
project_dir    = "."         # OPTIONAL: project root for resolving prompts/paths (default: config's dir)
```

`session_prefix` and `log_dir` are **required** — the harness has no project-specific fallbacks.
Every relative path (`log_dir`, an agent's `dir`, `handoff.repo`, prompt/template files) resolves
against `project_dir`, which defaults to the directory holding the config file. When the config
lives in a sandbox but the prompts live elsewhere (as `smoke.sh` does), set `project_dir`
explicitly.

### `[backend.<name>]` — backends declared as data

A backend is fully described by config — no code change to add one. The one field that selects
behavior is `prompt_delivery`:

| `prompt_delivery` | how the kickoff reaches the agent | example |
|---|---|---|
| `"arg"`  | passed as a CLI argument (claude-style) | `claude … "$(cat kickoff)"` |
| `"ping"` | typed in after a TUI connects (opencode-style) | attach, wait, send-keys |
| `"exec"` | a plain command; the prompt is written to a file | generic / demo |

```toml
[backend.claude]
bin             = "claude"
flags           = "--dangerously-skip-permissions"
remote_control  = true          # add a --remote-control <session> flag
supports_resume = true          # honor an agent's resume=true
prompt_delivery = "arg"
process_name    = "claude"      # the pane process a healthy session runs (backend-mismatch healing)
submit_key      = "Enter"       # key to submit a typed message
stall_idle      = 300           # seconds idle before a heal+stall agent is rebooted
active_re = "esc to interrupt|Running tool|· \\d+"   # pane shows the agent is WORKING
limit_re  = "usage limit|limit reached|reached your .*limit"   # usage/rate-limit banner
fatal_re  = "redacted_thinking|cannot be modified"  # unrecoverable session state → kill + restart

[backend.opencode]              # a TUI backend
bin             = "opencode"
attach          = "{bin} attach {server} --dir {dir}"
server          = "http://127.0.0.1:4096"
prompt_delivery = "ping"
process_name    = "opencode"
footer_ui       = true          # a static footer lingers after a turn → only the bottom = activity
log_grace       = 180           # within this many seconds of a log write, treat as active
connect_delay   = 12            # seconds to wait for the TUI before typing
submit_key      = "C-m"
model_env       = true          # pass the model via OPENCODE_CONFIG_CONTENT
preamble        = "set -a; . ./.env; set +a"   # shell run before launch (e.g. load creds)
active_re = "esc interrupt|thinking|running tool|preparing patch"
limit_re  = "usage limit|limit reached"

[backend.demo]                  # a dependency-free backend for testing the harness mechanics
bin             = "echo '[demo] {session} up'; exec sleep 1000000"
prompt_delivery = "exec"        # {kickoff}=prompt file, {session}=session name, {model}=model
```

For an `"arg"` backend the flag *templates* are configurable (so you can point at a non-claude
CLI): `resume_flag` (default `--resume '{id}'`), `model_flag` (default `--model '{model}'`),
`remote_control_flag` (default `--remote-control '{session}'`). A backend that sets `process_name`
participates in backend-mismatch healing; one that doesn't (e.g. `demo`) never does.

### `[[agent]]` — one block per agent

```toml
[[agent]]
name    = "builder"            # tmux session defaults to <session_prefix><name>; override with session=
kind    = "loop"              # loop | persistent | task
backend = "claude"            # overrides defaults.backend
model   = "claude-opus-4-8"   # overrides defaults.model
dir     = "."                 # working dir (relative → project dir)
role    = "builder"           # loop agents only: role prompt = <roles_dir>/<role>.md
resume  = true                # (arg backends with supports_resume) --resume <state/<name>.id>
watch   = "heal+stall"        # none | heal | heal+stall
enabled = true                # false = not started by a bare `up`, not supervised
wake    = { interval = 3600, prompt_file = "prompts/supervise.md" }   # periodic nudge
prompt  = """inline startup text"""          # persistent/task agents; OR prompt_file = "path.md"
log_signature = "PROJECT PHASE"              # optional: disambiguate agents that share a dir (agent-log.py)
```

| kind | prompt source | typical `watch` |
|---|---|---|
| `loop` | auto-built: kickoff template + `prompts/<role>.md` | `heal+stall` |
| `persistent` | `prompt` / `prompt_file` (+ optional `resume`, `wake`) | `heal` |
| `task` | `prompt` (runs once, then idles) | `none`, `enabled=false` |

**`watch` policy:**

| value | behavior |
|---|---|
| `none` | ignored by the watchdog entirely |
| `heal` | restart if the session is dead, FATAL-wedged, or running the wrong backend; pause all healing while inside a usage-limit window; **never** reboot just for being idle |
| `heal+stall` | everything in `heal`, **plus** reboot if idle past `stall_idle` — respecting any `WAITING-UNTIL: <ISO-8601>` self-wake marker the agent prints as its last line |

### `[[service]]` — non-AI helper processes

```toml
[[service]]
name    = "cleanlogs"
command = "python3 agent-log.py follow-all"
```

Started by a bare `up`, killed by `down`. Just a supervised command in a tmux session.

### `[loop]` — the phase state machine (governs `kind="loop"` agents)

```toml
[loop]
state_file       = "phase-idx"          # under <log_dir>/state/
resume_phase     = true                 # keep the phase index across restarts (don't reset to 0)
auto_advance     = true                 # advance when the current phase's status file says done_marker
done_marker      = "## DONE"
kickoff_template = "prompts/kickoff.md" # project preamble; slots {phase_id}/{plan}/{status}/{role}
roles_dir        = "prompts"            # role prompt = <roles_dir>/<role>.md
handoff = { repo = ".", claim_pings = "adversary", review_pings = "builder",
            inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"],
            claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
on_complete = { trigger_file = ".run-on-complete", run = "reporter" }   # run task agent on completion
phases = [
  { id = "p1", plan = "plans/p1.md", status = "STATUS-p1.md" },
  { id = "p2", plan = "plans/p2.md", status = "STATUS-p2.md", models = { builder = "claude-opus-4-8" } },
]
```

- **Kickoff template.** A loop agent's prompt is `kickoff_template` (with `{phase_id}`, `{plan}`,
  `{status}`, `{role}` substituted from the current phase) followed by `<roles_dir>/<role>.md`.
  Both are project files; this repo ships generic starters in `prompts/`. There is no built-in
  preamble text.
- **Per-phase model override.** A phase's `models = { builder = "...", adversary = "..." }`
  overrides those agents' model for just that phase (matched on the agent's `role`).
- **Auto-advance.** Each heavy tick, if the current phase's `status` file (looked up in
  `handoff.repo`'s `state_subdir/` then its root) contains a real `done_marker` — not a "Not
  yet…" placeholder — the watchdog stops the loops, bumps the phase index, and restarts them on
  the next phase. After the last phase it writes a `SEQUENCE-COMPLETE` marker under `log_dir` and
  stops the loops (idempotent — no churn). Appending a phase later clears the stale marker and
  resumes. On completion, an optional `on_complete.run` task agent fires if its `trigger_file`
  exists under `log_dir`.
- **Handoff signalling.** The watchdog watches `handoff.repo`'s `origin/main` for commits whose
  subject matches `claim_pattern` / `review_pattern`, and watches the two `inboxes` files. When a
  claim lands it pings the `claim_pings` agent; a review pings `review_pings`; an inbox change
  pings the relevant side. This is how the Builder and Adversary coordinate purely through git.
  `claim_pings` / `review_pings` may be a single agent name **or a list** — e.g.
  `claim_pings = ["correctness-adversary", "readability-adversary"]` pings every reviewer on a claim
  (each in its own session), for multi-reviewer setups.

---

## Config vs state

- **Config** = `agents.toml` — declarative, version-controlled, the only source of truth.
- **State** = `<log_dir>/state/` — machine-written runtime only: `phase-idx` (current phase),
  `<name>.id` (resume id), `limited-<session>.json` (active usage-limit window),
  `kickoff-<session>.txt` (the exact prompt last sent). Git-ignore your `log_dir`.
- **Env** = a one-off override for a *single* invocation only: `AGENT_MODEL_<name>=…` /
  `AGENT_BACKEND_<name>=…`. The persisted watchdog ignores env and re-reads the file every tick —
  deliberately, so env-vs-file drift can never silently revert a backend.

---

## The driver: verbs

The recommended (not required) verb set — an AI project-orchestrator can rely on these being
present, but a harness is free to add more:

```
agents.py up [name…]               start enabled agents (+ services + watchdog); use-or-create
agents.py down [name…]             stop agents/services/watchdog (all, or named)
agents.py status                   table of every agent: kind, backend, model, watch, state, phase
agents.py watchdog                 the supervisor loop (what the <prefix>watchdog session runs)
agents.py logs <name>              tail that session's log
agents.py phase [show|next|set N]  inspect / move the loop phase index
agents.py tokens                   per-phase token + time report (when [watchdog].log_tokens = true)
agents.py selftest                 regression-test the backend activity detector (needs no config)
agents.py init [dir]               scaffold a starter agents.toml + prompts/ in a project dir
  --config PATH                    use a specific config (default: ./agents.toml)
```

### The watchdog tick

`agents.py watchdog` runs as the `<prefix>watchdog` tmux session and **re-reads the config every
tick**. Each loop:

- **signal tick** (`signal_interval`): handoff pings; for each watched agent the usage-limit check,
  and for `heal+stall` agents the stall check; fire any due `wake`.
- **heavy tick** (`heavy_interval`): advance the loop phase if the current one is done; otherwise
  heal each watched agent per its `watch` policy. When the sequence is complete the finished loops
  stay stopped, but persistent agents stay supervised.

**Usage-limit handling:** when an agent prints a limit banner, the watchdog parses the reset time,
arms a quiet window (never rebooting a limited agent), and at the end sends one probe to resume it
— re-arming if the banner re-prints.

---

## Driving the harness from an AI project-orchestrator

This harness is designed to be driven by an AI "project-orchestrator" (PO) that creates and runs
many projects, each pinning its own copy of this engine. The contract is intentionally **not
rigid** — the PO reads these docs and works out how to drive a project. What it can rely on:

1. **One config, one driver.** Everything the PO needs to know about a project's agents is in that
   project's `agents.toml`; everything it can *do* is a verb above. To inspect, `status`. To start
   or stop, `up` / `down`. To move the phase, `phase`.
2. **Isolation by `session_prefix`.** Two projects never collide as long as their `session_prefix`
   differ. The PO assigns each project a unique prefix at creation.
3. **State is on disk, not in the PO.** Phase index, resume ids and limit windows live under the
   project's `log_dir`. The PO can restart a project (or the whole host) and the watchdog resumes
   from there.
4. **Knowledge is one-directional.** A project repo contains nothing about the PO or the fleet —
   it can be run by hand and would have no idea a PO exists. The PO's fleet registry is the only
   record of which projects exist and at what engine ref. This repo never reaches "up" toward a PO.
5. **Submodule pin = the engine version.** A project pins this repo at a tag (e.g. `v0.1.0`) as a
   submodule under `engine/`. Bumping is per-project and opt-in (`git submodule update --remote`);
   one project's bump can't break another.

A minimal project layout the PO scaffolds:

```
my-project/                 # its own repo; knows nothing about the PO
  agents.toml               # harness config (this schema)
  engine/                   # this repo as a pinned submodule
  prompts/                  # role prompts + kickoff template
  machine-docs/             # the loop pair's coordination files (STATUS/REVIEW/inboxes)
  .ao-state/                # runtime state + logs (gitignored)
  .env                      # project creds (never in git)
```

Run it by hand with `engine/agents.py up --config agents.toml`.

---

## Tangled (the atproto forge) — the PR tools

One tool per action, all reading `tangled.cookie` from the secret store (`--cookie-file` is a
legacy fallback). They exist because Tangled has no client CLI: each action is an htmx POST to
the appview that only a logged-in session can make.

| tool | action |
|---|---|
| `tangled_pr.py` | open a pull (branch-based; `--target` is the branch it merges into) |
| `tangled_pr_edit.py` | edit a pull's title/body, or `--show` the current one |
| `tangled_pr_resubmit.py` | **advance a pull to a new round after you pushed a fixup**, and print the interdiff URL |
| `tangled_pr_merge.py` | `--check` mergeability, then `--merge` or `--close` |
| `tangled_comments.py` | read a pull's review comments (`--json` for machine use) |
| `tangled_comment_post.py` | post a comment on a pull |
| `tangled_repo.py` | create a repo as the bot (see also the `tangled-repo` skill) |

**The trap `tangled_pr_resubmit.py` exists for:** pushing the branch does NOT update the pull.
The appview keeps serving the patch it fetched when the pull was opened or last resubmitted,
so reviewers read the pre-fixup code and no interdiff exists — with no warning, because the
push itself succeeded. Always: push, resubmit, then reply with the printed interdiff URL. See
`machine-docs/PR-WORKFLOW.md`, "A push does NOT advance the round".

## The testing gateway — public domains for tailnet boxes

A box on the tailnet with no public IP can still have a real HTTPS domain: the shared testing
gateway holds a wildcard record for `*.gtest.commoninternet.net` and forwards by SNI.

```bash
python3 engine/tools/gateway-domain.py add myapp     # -> myapp.gtest.commoninternet.net -> this box
python3 engine/tools/gateway-domain.py list
python3 engine/tools/gateway-domain.py remove myapp
```

The admin password is `gateway.admin_password` in the secret store below; the tool reads it
itself. **The gateway does not terminate TLS** — it proxies the encrypted stream, so your box
serves the certificate for that name. Full detail, including how to get a cert and why hostname
backends are refused, is in the `gateway-domain` skill (`skills/gateway-domain/SKILL.md`).

## Secrets — one encrypted store, never in git

**Every credential on an orchestrator host lives in one sops+age encrypted file. Do not put a
secret anywhere else** — not in a git remote URL, not in a project `.env`, not in a prompt.

```
/secrets/store.yaml         the store: sops+age ciphertext, mode 0600
~/.config/sops/age/keys.txt     the age private key — the ONE plaintext secret, mode 0600
```

`/secrets/` is deliberately **not a git repo and has no remote**, so there is no path by
which a `git add`/`git push` can leak it; the store is ciphertext at rest anyway.

Read it with `engine/secrets.py` (stdlib + the `sops` binary, no Python deps):

```python
from secrets import get, get_group
cookie = get("tangled.cookie")            # a single value
env    = get_group("cc_ci_testenv")       # a whole group as a dict
```

```sh
python3 engine/secrets.py list                 # group/key NAMES only — never prints values
python3 engine/secrets.py get tangled.cookie   # one value on stdout
python3 engine/secrets.py materialize tangled-session   # write a runtime file from the store
sops /secrets/store.yaml                   # add/edit: decrypts to $EDITOR, re-encrypts on save
```

**One home per secret — two shapes.**

*Values our code reads* live **in the store**; import this module and ask for them. Nothing is
written to disk (`engine/.tangled-session` is gone — the tangled tools read `tangled.cookie`).

*Secrets a third party reads from a fixed path* (ssh keys, a systemd `EnvironmentFile`, nix's
`authKeyFile`, a TLS keypair) live as **real files in `/secrets/files/`, symlinked from the path
the consumer expects**:

```
~/.ssh/tangled-ed25519                -> /secrets/files/tangled-ed25519
/etc/ts-auth-key                      -> /secrets/files/ts-auth-key
/srv/cc-ci/.testenv                   -> /secrets/files/cc-ci.testenv
```

The consumer is unchanged and unaware; the file exists once, in one directory, at 0600. Do **not**
also copy such a secret into `store.yaml` — that is two sources of truth again.

For a one-off where neither shape fits, inject at run time and leave nothing behind:

```sh
python3 engine/secrets.py exec-env <group> -- some-command        # group as env vars
python3 engine/secrets.py with-file <group.key> -- cmd -i {}      # 0600 file in a private
                                                                  # tmpdir, deleted on exit
```

**The symlink exception: apps that rewrite their own credential file.** An app that refreshes an
OAuth token by writing `auth.json` atomically (write-temp + rename) **replaces the symlink with a
regular file**, silently splitting the home again. `~/.local/share/opencode/auth.json` is such a
file, so it stays where it is and is deliberately *not* centralised. Before symlinking a secret,
ask whether its owner ever writes it back.

**Rules of thumb**

- Prefer ssh remotes over `https://user:pass@host/...`. A password in a remote URL is printed by
  `git remote -v`, copied into every clone, and survives in `.git/config` where nobody looks.
- A private key is `chmod 600`. Check with
  `find . -name '*.key' -o -name 'id_*' ! -name '*.pub' -perm /044`.
- Anything a project must keep on disk goes in `.gitignore` **and** gets its real home in the store.

---

## Nix

A `flake.nix` provides a reproducible devShell with the runtime deps (`python311` for stdlib
`tomllib`, plus `tmux` and `git`):

```bash
nix develop                                   # enter the shell
nix develop -c python3 agents.py selftest      # or run one command in it
nix flake check                                # evaluate + build the devShell
```

The agent CLIs themselves (`claude`, `opencode`) are **external, non-Nix tools** — install them
per their own docs and make sure they are on `PATH` before launching live agents. The devShell
documents this in its banner.

### PATH on a NixOS host: why a tool that IS installed says "command not found"

**An agent's `PATH` does not include the system profile.** On `notplants-orchestrator` the agent
shell gets a pinned list of individual store paths (bash, git, python, coreutils, findutils, grep,
sed, systemd, tmux, openssh, net-tools) and **not** `/run/current-system/sw/bin`. Everything the
host declares in `environment.systemPackages` therefore exists and is unreachable.

Measured 2026-08-21: `ps`, `pgrep`, `free`, `cmp`, `awk`, `curl`, `diff`, `strings`, `nm`, `getent`
and `ping` were ALL installed (notplants-nix `hosts/notplants-orchestrator/configuration.nix`,
"give agents a real toolbox") and all reported `command not found`.

**Why this is worse than an inconvenience.** A missing binary run through `shell=True` returns
**rc=127 with empty stdout**, and empty stdout is indistinguishable from a real answer of "none":

- `agents.py` shelled out to `pgrep -P` to find a pane's child processes. With `pgrep` unreachable
  it returned *no children*, so `_build_running` was **always False** and the stall detector could
  reboot an agent in the middle of a build. It shipped that way and no test caught it, because the
  unit tests **mocked `pgrep`** — the fake stood in for the broken dependency.
- The same session read an empty `ps` as "0 agent processes running", and an empty `strings` as
  proof that a binary had its features stripped.

**Check before concluding a tool is absent:**

```bash
ls /run/current-system/sw/bin/<tool>     # installed but unreachable?
command -v <tool>                        # reachable?
```

**Put the system profile on PATH** (appended, so the sandbox's pinned store paths keep priority):

```bash
case ":$PATH:" in *":/run/current-system/sw/bin:"*) ;;
  *) export PATH="$PATH:/run/current-system/sw/bin" ;; esac
```

For something genuinely not installed, fetch it without changing the host:

```bash
nix shell nixpkgs#tcpdump -c tcpdump ...       # one command, nothing persisted
nix run nixpkgs#git-filter-repo -- --help
```

`nix` itself may also be off `PATH`; it lives at `/nix/var/nix/profiles/default/bin/nix`.

**Rule for harness code: do not shell out for something the kernel already exposes.** `agents.py`
now reads `/proc/<pid>/stat` and `/proc/<pid>/comm` directly instead of calling `pgrep` and `ps`.
That has no PATH dependency and cannot fail silently in the direction that matters. If you must
call an external tool, check the return code — never treat empty output as an answer.

---

## Testing

The `tests/` directory holds the harness's own test suite. One runner drives everything:

- `tests/test_unit.py` — the harness: config load, kickoff, the phase machine, limit parsing,
  waiting-until, the build-process detector.
- `tests/test_tools.py` — the standalone tools (`tangled_pr.py`, `tangled_pr_close.py`,
  `tools/gateway-domain.py`). **No network**: every HTTP boundary is injected as a fake `_fetch`.

**Mock the seam you own, not the tool you depend on.** The build-detector tests used to fake
`pgrep` and `ps` subprocess calls, so they passed on a host where neither was reachable and the
real defect — empty output read as "no build running" — was invisible to every test. They now
patch `_proc_descendants` and `_comms`, the functions this repo owns. A test that mocks a
dependency proves the mock works.

```bash
nix develop -c ./tests/run.sh      # unit tests always; live backend smokes when available
# or just:  ./tests/run.sh         # (python3 + tmux must be on PATH)
```

What it runs:

- **Unit tests** (`tests/test_unit.py`) — pure logic, **no agents spawned, no live tmux sessions**.
  Cover config load + defaults merge, kickoff-template assembly, the phase machine (advance on the
  done marker, idempotent sequence-complete, append-a-phase resumes), usage-limit reset-banner
  parsing, `WAITING-UNTIL` / stall parsing, and the per-backend activity detectors (claude +
  opencode footers). Always run; a failure fails the suite. Run them alone with
  `python3 -m unittest discover -s tests` (or `python3 tests/test_unit.py`).
- **Live backend smokes** (`tests/smoke_claude.sh`, `tests/smoke_opencode.sh`) — each brings a
  throwaway scratch project up **through `agents.py`** on a real backend, in a fully isolated
  sandbox (its own unique `session_prefix`, a temp `log_dir`, and — for opencode — a dedicated
  server on a non-default port `AOTEST_OC_PORT`, default `4097`), confirms the session attaches and
  `status` reports it RUNNING, then `down`s it and cleans up (no leftover sessions, port freed).
  Each **SKIPs gracefully** (exit 0) when its backend's binary or creds are unavailable. Useful env:
  `CLAUDE_BIN` / `OPENCODE_BIN`, `AOTEST_MODEL`, `AOTEST_OC_PORT`, `AOTEST_OC_CREDS`.
- **Isolation sanity** — after the live runs, the runner asserts no `aotest-*` tmux sessions leaked
  and reports that any live sessions are untouched.

The smokes are safe by construction: a unique per-run session prefix (never `cc-ci-` or any real
project's), a dedicated opencode port (never `4096`), and a cleanup trap that fires on success,
failure, and Ctrl+C.

---

## Adding things

- **Add an agent** — add an `[[agent]]` block; `agents.py up <name>`. No code change.
- **Add a backend** — add a `[backend.<name>]` block (`bin`, `prompt_delivery`, the regexes);
  point an agent at it with `backend = "<name>"`.
- **Add / append a phase** — add an entry to `[loop].phases`; the watchdog advances into it
  automatically (clearing a stale `SEQUENCE-COMPLETE` if the sequence had finished).
- **Change a model or backend** — edit the field (or a phase's `models = {}`), then
  `agents.py down <name> && agents.py up <name>`. The watchdog re-reads the file; it won't fight you.