notplants
22bd897a86
An agent on a box with no public IP frequently needs a reachable HTTPS URL —
an OAuth callback, a webhook receiver, a demo link. The testing gateway
already holds a wildcard record for *.gtest.commoninternet.net and forwards
by SNI, but nothing here knew that, so every agent had to be told by hand.
tools/gateway-domain.py add myapp
# myapp.gtest.commoninternet.net -> 100.84.190.30
The backend defaults to the running box's own tailscale IP, which is the case
that comes up almost every time.
The admin password comes from gateway.admin_password in the secret store; the
tool reads it itself, so no caller handles the value and there is no second
copy to drift or get committed.
Two things the tool refuses to do, both learned by doing them:
Backends must be a literal IPv4 address. The gateway's validate_ip accepts a
hostname, but put_domain/remove_domain only match lines whose backend is
numeric ([\d.:]+). A hostname mapping can therefore be written once and never
updated or removed through the admin UI — it becomes an orphan that only a
hand-edit of tunnel_map.conf clears. One got created while testing this.
Verification re-reads the mapping table instead of trusting the POST body.
The admin app mutates its in-memory dict and renders that, so a delete that
silently failed still renders as success. Checking the response alone
reported "removed" for an entry that was still on disk.
skills/gateway-domain/ carries the rest: that the gateway does NOT terminate
TLS (your box serves the cert for that name), how ACME still works through
it, that only 22/80/443 are open at the edge, and how to recover if an
interrupted e2e run leaves the admin password reseeded.
527 lines
28 KiB
Markdown
527 lines
28 KiB
Markdown
# agent-orchestrator
|
||
|
||
A generic, reusable harness for running and supervising a fleet of AI-agent sessions in **tmux**.
|
||
One driver script + one declarative config (`agents.toml`) describe every agent — a Builder /
|
||
Adversary loop pair, a persistent supervisor, a one-shot task — and a **watchdog** keeps them
|
||
alive, healed, paced, and coordinated. The watchdog reads the same config every tick, so there is
|
||
never any env-vs-file drift.
|
||
|
||
Nothing about any particular project lives in this repo. Paths, the loop **kickoff preamble**, the
|
||
**handoff conventions**, and the **on-complete** hook are all supplied by the project's config and
|
||
prompt files. A project consumes this repo as a pinned **git submodule** (`engine/`) and keeps its
|
||
own config, prompts, state, and tmux namespace — total isolation between projects.
|
||
|
||
```
|
||
agents.py the driver + watchdog (pure Python stdlib; needs python >= 3.11 for tomllib)
|
||
agent-log.py render claude JSONL transcripts into clean, greppable logs
|
||
agents.example.toml a self-contained 2-agent example project
|
||
prompts/ generic role + kickoff templates (builder / adversary / kickoff)
|
||
examples/ runnable example projects — the Builder/Adversary variant family, snakepit, …
|
||
smoke.sh bring the example up + tear it down in an isolated sandbox, then clean up
|
||
tests/ the test suite — unit tests + isolated live backend smokes + a runner
|
||
flake.nix/.lock a Nix devShell with the runtime deps (python311, tmux, git)
|
||
```
|
||
|
||
---
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
nix develop # python311 + tmux + git on PATH (see "Nix" below)
|
||
|
||
python3 agents.py selftest # regression-test the activity detector (no config)
|
||
python3 agents.py status --config agents.example.toml # one table: every agent + the phase
|
||
./smoke.sh # prove up/down works end-to-end, isolated + clean
|
||
|
||
python3 agents.py init myproject # scaffold a starter agents.toml + prompts/
|
||
```
|
||
|
||
`up` is **use-or-create**: an already-running session is left alone, never double-started.
|
||
|
||
```bash
|
||
python3 agents.py --config agents.toml up # start all enabled agents + services + watchdog
|
||
python3 agents.py --config agents.toml up builder # start just one agent (by name)
|
||
python3 agents.py --config agents.toml down # stop everything
|
||
python3 agents.py --config agents.toml logs builder # tail one session's log
|
||
python3 agents.py --config agents.toml phase show # where the loop phase machine is
|
||
```
|
||
|
||
`--config` defaults to `./agents.toml`, falling back to one next to `agents.py`.
|
||
|
||
---
|
||
|
||
## Examples
|
||
|
||
`examples/` holds runnable example projects — copy one, point `agents.py` at its `agents.toml`, and
|
||
go. The headline set is a family of **Builder/Adversary** variants that build the *same* task but each
|
||
differ in one dimension — useful both as templates and as a study of the pattern:
|
||
|
||
- **`builder-adversary`** — the canonical loop pair: a Builder that builds and an Adversary that
|
||
cold-verifies every claim, coordinating only through git (`claim(`/`review(` commits + the watchdog
|
||
handoff). **Start here.**
|
||
- **`builder-adversary-min`** — the same pattern with the prompts compressed to minimal tokens.
|
||
- **`builder-adversary-stateless`** — `builder-adversary` + **context hygiene** (compact at each
|
||
checkpoint, read diffs not trees, lean loads) to minimise carried/reloaded context.
|
||
- **`builder-adversary-lean`** — context hygiene + **per-gate** review (one claim/verdict per gate).
|
||
- **`builder-adversary-deferred`** — the Adversary verifies **once**, after the whole build, in a
|
||
final comprehensive `review` phase (vs per-phase / per-gate).
|
||
- **`builder-solo`** — a single Builder that self-certifies, with **no Adversary** (the control).
|
||
- **`snakepit`** — a different topology entirely: a pool of identical worker "snakes" pulling tasks
|
||
from a shared filesystem queue, plus cleanup specialists. (`examples/IDEAS.md` sketches more.)
|
||
|
||
Each example has its own `README.md`. Run one by hand:
|
||
|
||
```bash
|
||
cd examples/builder-adversary
|
||
python3 ../../agents.py status --config agents.toml # read-only
|
||
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
|
||
```
|
||
|
||
**Benchmark.** The separate
|
||
[`agent-orchestrator-benchmark`](https://git.autonomic.zone/recipe-maintainers/agent-orchestrator-benchmark)
|
||
repo runs these Builder/Adversary variants head-to-head (N=5, real `agents.py up` runs) to measure
|
||
what drives token cost. Short version: an independent adversary costs **~4.7×** a solo builder, but
|
||
the review *cadence* (per-gate / per-phase / deferred) is **nearly token-neutral**, and **context
|
||
hygiene** is the one clean **~−22%** win. See that repo's `FINDINGS.md`.
|
||
|
||
---
|
||
|
||
## The config: `agents.toml`
|
||
|
||
Five section types: `[watchdog]`, `[backend.<name>]`, `[defaults]`, `[[agent]]` / `[[service]]`,
|
||
and `[loop]`. See `agents.example.toml` for a complete, runnable example.
|
||
|
||
### `[watchdog]` — global supervisor cadence
|
||
|
||
```toml
|
||
[watchdog]
|
||
signal_interval = 30 # seconds between light checks (handoff / stall / limit)
|
||
heavy_interval = 300 # seconds between heal + phase-advance checks
|
||
limit_probe_fallback = 300 # re-probe cadence for a usage-limited agent when reset time is unparsable
|
||
limit_reset_slack = 45 # seconds to wait past a parsed reset before probing
|
||
stall_grace = 180 # seconds of slack past a WAITING-UNTIL marker before a stall reboot
|
||
log_tokens = false # opt-in: record per-phase token + time usage (see below)
|
||
```
|
||
|
||
**Per-phase token + time logging (`log_tokens`).** Set `log_tokens = true` (under `[watchdog]` or
|
||
`[loop]`) and the watchdog records, for **each phase**, how many tokens **each agent** used and how
|
||
long the phase took — appended as one JSON object per phase to `<log_dir>/token-log.jsonl`. Tokens
|
||
are summed from each agent's Claude Code session transcript and attributed **by working dir**, so
|
||
give each agent its own `dir` (the Builder/Adversary loop pair already uses separate clones) for
|
||
accurate per-agent numbers. The watchdog snapshots a baseline when a phase starts and writes the
|
||
delta (per agent, and the total) when the phase advances or the sequence completes — robust across
|
||
watchdog restarts. Pretty-print it with `agents.py tokens`:
|
||
|
||
```
|
||
phase dur(s) builder adversary TOTAL
|
||
-----------------------------------------------------
|
||
lex 372.0 3,910,118 3,221,447 7,131,565
|
||
parse 410.5 ...
|
||
```
|
||
|
||
### `[defaults]` — inherited by every agent
|
||
|
||
```toml
|
||
[defaults]
|
||
session_prefix = "myproj-" # REQUIRED: tmux namespace for this project. No implicit default.
|
||
log_dir = ".ao-state" # REQUIRED: logs + state/. Relative paths resolve against the config dir.
|
||
backend = "claude"
|
||
model = "claude-sonnet-4-6"
|
||
dir = "." # default working dir for agents (relative → project dir)
|
||
watch = "heal" # none | heal | heal+stall
|
||
project_dir = "." # OPTIONAL: project root for resolving prompts/paths (default: config's dir)
|
||
```
|
||
|
||
`session_prefix` and `log_dir` are **required** — the harness has no project-specific fallbacks.
|
||
Every relative path (`log_dir`, an agent's `dir`, `handoff.repo`, prompt/template files) resolves
|
||
against `project_dir`, which defaults to the directory holding the config file. When the config
|
||
lives in a sandbox but the prompts live elsewhere (as `smoke.sh` does), set `project_dir`
|
||
explicitly.
|
||
|
||
### `[backend.<name>]` — backends declared as data
|
||
|
||
A backend is fully described by config — no code change to add one. The one field that selects
|
||
behavior is `prompt_delivery`:
|
||
|
||
| `prompt_delivery` | how the kickoff reaches the agent | example |
|
||
|---|---|---|
|
||
| `"arg"` | passed as a CLI argument (claude-style) | `claude … "$(cat kickoff)"` |
|
||
| `"ping"` | typed in after a TUI connects (opencode-style) | attach, wait, send-keys |
|
||
| `"exec"` | a plain command; the prompt is written to a file | generic / demo |
|
||
|
||
```toml
|
||
[backend.claude]
|
||
bin = "claude"
|
||
flags = "--dangerously-skip-permissions"
|
||
remote_control = true # add a --remote-control <session> flag
|
||
supports_resume = true # honor an agent's resume=true
|
||
prompt_delivery = "arg"
|
||
process_name = "claude" # the pane process a healthy session runs (backend-mismatch healing)
|
||
submit_key = "Enter" # key to submit a typed message
|
||
stall_idle = 300 # seconds idle before a heal+stall agent is rebooted
|
||
active_re = "esc to interrupt|Running tool|· \\d+" # pane shows the agent is WORKING
|
||
limit_re = "usage limit|limit reached|reached your .*limit" # usage/rate-limit banner
|
||
fatal_re = "redacted_thinking|cannot be modified" # unrecoverable session state → kill + restart
|
||
|
||
[backend.opencode] # a TUI backend
|
||
bin = "opencode"
|
||
attach = "{bin} attach {server} --dir {dir}"
|
||
server = "http://127.0.0.1:4096"
|
||
prompt_delivery = "ping"
|
||
process_name = "opencode"
|
||
footer_ui = true # a static footer lingers after a turn → only the bottom = activity
|
||
log_grace = 180 # within this many seconds of a log write, treat as active
|
||
connect_delay = 12 # seconds to wait for the TUI before typing
|
||
submit_key = "C-m"
|
||
model_env = true # pass the model via OPENCODE_CONFIG_CONTENT
|
||
preamble = "set -a; . ./.env; set +a" # shell run before launch (e.g. load creds)
|
||
active_re = "esc interrupt|thinking|running tool|preparing patch"
|
||
limit_re = "usage limit|limit reached"
|
||
|
||
[backend.demo] # a dependency-free backend for testing the harness mechanics
|
||
bin = "echo '[demo] {session} up'; exec sleep 1000000"
|
||
prompt_delivery = "exec" # {kickoff}=prompt file, {session}=session name, {model}=model
|
||
```
|
||
|
||
For an `"arg"` backend the flag *templates* are configurable (so you can point at a non-claude
|
||
CLI): `resume_flag` (default `--resume '{id}'`), `model_flag` (default `--model '{model}'`),
|
||
`remote_control_flag` (default `--remote-control '{session}'`). A backend that sets `process_name`
|
||
participates in backend-mismatch healing; one that doesn't (e.g. `demo`) never does.
|
||
|
||
### `[[agent]]` — one block per agent
|
||
|
||
```toml
|
||
[[agent]]
|
||
name = "builder" # tmux session defaults to <session_prefix><name>; override with session=
|
||
kind = "loop" # loop | persistent | task
|
||
backend = "claude" # overrides defaults.backend
|
||
model = "claude-opus-4-8" # overrides defaults.model
|
||
dir = "." # working dir (relative → project dir)
|
||
role = "builder" # loop agents only: role prompt = <roles_dir>/<role>.md
|
||
resume = true # (arg backends with supports_resume) --resume <state/<name>.id>
|
||
watch = "heal+stall" # none | heal | heal+stall
|
||
enabled = true # false = not started by a bare `up`, not supervised
|
||
wake = { interval = 3600, prompt_file = "prompts/supervise.md" } # periodic nudge
|
||
prompt = """inline startup text""" # persistent/task agents; OR prompt_file = "path.md"
|
||
log_signature = "PROJECT PHASE" # optional: disambiguate agents that share a dir (agent-log.py)
|
||
```
|
||
|
||
| kind | prompt source | typical `watch` |
|
||
|---|---|---|
|
||
| `loop` | auto-built: kickoff template + `prompts/<role>.md` | `heal+stall` |
|
||
| `persistent` | `prompt` / `prompt_file` (+ optional `resume`, `wake`) | `heal` |
|
||
| `task` | `prompt` (runs once, then idles) | `none`, `enabled=false` |
|
||
|
||
**`watch` policy:**
|
||
|
||
| value | behavior |
|
||
|---|---|
|
||
| `none` | ignored by the watchdog entirely |
|
||
| `heal` | restart if the session is dead, FATAL-wedged, or running the wrong backend; pause all healing while inside a usage-limit window; **never** reboot just for being idle |
|
||
| `heal+stall` | everything in `heal`, **plus** reboot if idle past `stall_idle` — respecting any `WAITING-UNTIL: <ISO-8601>` self-wake marker the agent prints as its last line |
|
||
|
||
### `[[service]]` — non-AI helper processes
|
||
|
||
```toml
|
||
[[service]]
|
||
name = "cleanlogs"
|
||
command = "python3 agent-log.py follow-all"
|
||
```
|
||
|
||
Started by a bare `up`, killed by `down`. Just a supervised command in a tmux session.
|
||
|
||
### `[loop]` — the phase state machine (governs `kind="loop"` agents)
|
||
|
||
```toml
|
||
[loop]
|
||
state_file = "phase-idx" # under <log_dir>/state/
|
||
resume_phase = true # keep the phase index across restarts (don't reset to 0)
|
||
auto_advance = true # advance when the current phase's status file says done_marker
|
||
done_marker = "## DONE"
|
||
kickoff_template = "prompts/kickoff.md" # project preamble; slots {phase_id}/{plan}/{status}/{role}
|
||
roles_dir = "prompts" # role prompt = <roles_dir>/<role>.md
|
||
handoff = { repo = ".", claim_pings = "adversary", review_pings = "builder",
|
||
inboxes = ["ADVERSARY-INBOX.md", "BUILDER-INBOX.md"],
|
||
claim_pattern = "^claim", review_pattern = "^review", state_subdir = "machine-docs" }
|
||
on_complete = { trigger_file = ".run-on-complete", run = "reporter" } # run task agent on completion
|
||
phases = [
|
||
{ id = "p1", plan = "plans/p1.md", status = "STATUS-p1.md" },
|
||
{ id = "p2", plan = "plans/p2.md", status = "STATUS-p2.md", models = { builder = "claude-opus-4-8" } },
|
||
]
|
||
```
|
||
|
||
- **Kickoff template.** A loop agent's prompt is `kickoff_template` (with `{phase_id}`, `{plan}`,
|
||
`{status}`, `{role}` substituted from the current phase) followed by `<roles_dir>/<role>.md`.
|
||
Both are project files; this repo ships generic starters in `prompts/`. There is no built-in
|
||
preamble text.
|
||
- **Per-phase model override.** A phase's `models = { builder = "...", adversary = "..." }`
|
||
overrides those agents' model for just that phase (matched on the agent's `role`).
|
||
- **Auto-advance.** Each heavy tick, if the current phase's `status` file (looked up in
|
||
`handoff.repo`'s `state_subdir/` then its root) contains a real `done_marker` — not a "Not
|
||
yet…" placeholder — the watchdog stops the loops, bumps the phase index, and restarts them on
|
||
the next phase. After the last phase it writes a `SEQUENCE-COMPLETE` marker under `log_dir` and
|
||
stops the loops (idempotent — no churn). Appending a phase later clears the stale marker and
|
||
resumes. On completion, an optional `on_complete.run` task agent fires if its `trigger_file`
|
||
exists under `log_dir`.
|
||
- **Handoff signalling.** The watchdog watches `handoff.repo`'s `origin/main` for commits whose
|
||
subject matches `claim_pattern` / `review_pattern`, and watches the two `inboxes` files. When a
|
||
claim lands it pings the `claim_pings` agent; a review pings `review_pings`; an inbox change
|
||
pings the relevant side. This is how the Builder and Adversary coordinate purely through git.
|
||
`claim_pings` / `review_pings` may be a single agent name **or a list** — e.g.
|
||
`claim_pings = ["correctness-adversary", "readability-adversary"]` pings every reviewer on a claim
|
||
(each in its own session), for multi-reviewer setups.
|
||
|
||
---
|
||
|
||
## Config vs state
|
||
|
||
- **Config** = `agents.toml` — declarative, version-controlled, the only source of truth.
|
||
- **State** = `<log_dir>/state/` — machine-written runtime only: `phase-idx` (current phase),
|
||
`<name>.id` (resume id), `limited-<session>.json` (active usage-limit window),
|
||
`kickoff-<session>.txt` (the exact prompt last sent). Git-ignore your `log_dir`.
|
||
- **Env** = a one-off override for a *single* invocation only: `AGENT_MODEL_<name>=…` /
|
||
`AGENT_BACKEND_<name>=…`. The persisted watchdog ignores env and re-reads the file every tick —
|
||
deliberately, so env-vs-file drift can never silently revert a backend.
|
||
|
||
---
|
||
|
||
## The driver: verbs
|
||
|
||
The recommended (not required) verb set — an AI project-orchestrator can rely on these being
|
||
present, but a harness is free to add more:
|
||
|
||
```
|
||
agents.py up [name…] start enabled agents (+ services + watchdog); use-or-create
|
||
agents.py down [name…] stop agents/services/watchdog (all, or named)
|
||
agents.py status table of every agent: kind, backend, model, watch, state, phase
|
||
agents.py watchdog the supervisor loop (what the <prefix>watchdog session runs)
|
||
agents.py logs <name> tail that session's log
|
||
agents.py phase [show|next|set N] inspect / move the loop phase index
|
||
agents.py tokens per-phase token + time report (when [watchdog].log_tokens = true)
|
||
agents.py selftest regression-test the backend activity detector (needs no config)
|
||
agents.py init [dir] scaffold a starter agents.toml + prompts/ in a project dir
|
||
--config PATH use a specific config (default: ./agents.toml)
|
||
```
|
||
|
||
### The watchdog tick
|
||
|
||
`agents.py watchdog` runs as the `<prefix>watchdog` tmux session and **re-reads the config every
|
||
tick**. Each loop:
|
||
|
||
- **signal tick** (`signal_interval`): handoff pings; for each watched agent the usage-limit check,
|
||
and for `heal+stall` agents the stall check; fire any due `wake`.
|
||
- **heavy tick** (`heavy_interval`): advance the loop phase if the current one is done; otherwise
|
||
heal each watched agent per its `watch` policy. When the sequence is complete the finished loops
|
||
stay stopped, but persistent agents stay supervised.
|
||
|
||
**Usage-limit handling:** when an agent prints a limit banner, the watchdog parses the reset time,
|
||
arms a quiet window (never rebooting a limited agent), and at the end sends one probe to resume it
|
||
— re-arming if the banner re-prints.
|
||
|
||
---
|
||
|
||
## Driving the harness from an AI project-orchestrator
|
||
|
||
This harness is designed to be driven by an AI "project-orchestrator" (PO) that creates and runs
|
||
many projects, each pinning its own copy of this engine. The contract is intentionally **not
|
||
rigid** — the PO reads these docs and works out how to drive a project. What it can rely on:
|
||
|
||
1. **One config, one driver.** Everything the PO needs to know about a project's agents is in that
|
||
project's `agents.toml`; everything it can *do* is a verb above. To inspect, `status`. To start
|
||
or stop, `up` / `down`. To move the phase, `phase`.
|
||
2. **Isolation by `session_prefix`.** Two projects never collide as long as their `session_prefix`
|
||
differ. The PO assigns each project a unique prefix at creation.
|
||
3. **State is on disk, not in the PO.** Phase index, resume ids and limit windows live under the
|
||
project's `log_dir`. The PO can restart a project (or the whole host) and the watchdog resumes
|
||
from there.
|
||
4. **Knowledge is one-directional.** A project repo contains nothing about the PO or the fleet —
|
||
it can be run by hand and would have no idea a PO exists. The PO's fleet registry is the only
|
||
record of which projects exist and at what engine ref. This repo never reaches "up" toward a PO.
|
||
5. **Submodule pin = the engine version.** A project pins this repo at a tag (e.g. `v0.1.0`) as a
|
||
submodule under `engine/`. Bumping is per-project and opt-in (`git submodule update --remote`);
|
||
one project's bump can't break another.
|
||
|
||
A minimal project layout the PO scaffolds:
|
||
|
||
```
|
||
my-project/ # its own repo; knows nothing about the PO
|
||
agents.toml # harness config (this schema)
|
||
engine/ # this repo as a pinned submodule
|
||
prompts/ # role prompts + kickoff template
|
||
machine-docs/ # the loop pair's coordination files (STATUS/REVIEW/inboxes)
|
||
.ao-state/ # runtime state + logs (gitignored)
|
||
.env # project creds (never in git)
|
||
```
|
||
|
||
Run it by hand with `engine/agents.py up --config agents.toml`.
|
||
|
||
---
|
||
|
||
## Tangled (the atproto forge) — the PR tools
|
||
|
||
One tool per action, all reading `tangled.cookie` from the secret store (`--cookie-file` is a
|
||
legacy fallback). They exist because Tangled has no client CLI: each action is an htmx POST to
|
||
the appview that only a logged-in session can make.
|
||
|
||
| tool | action |
|
||
|---|---|
|
||
| `tangled_pr.py` | open a pull (branch-based; `--target` is the branch it merges into) |
|
||
| `tangled_pr_edit.py` | edit a pull's title/body, or `--show` the current one |
|
||
| `tangled_pr_resubmit.py` | **advance a pull to a new round after you pushed a fixup**, and print the interdiff URL |
|
||
| `tangled_pr_merge.py` | `--check` mergeability, then `--merge` or `--close` |
|
||
| `tangled_comments.py` | read a pull's review comments (`--json` for machine use) |
|
||
| `tangled_comment_post.py` | post a comment on a pull |
|
||
| `tangled_repo.py` | create a repo as the bot (see also the `tangled-repo` skill) |
|
||
|
||
**The trap `tangled_pr_resubmit.py` exists for:** pushing the branch does NOT update the pull.
|
||
The appview keeps serving the patch it fetched when the pull was opened or last resubmitted,
|
||
so reviewers read the pre-fixup code and no interdiff exists — with no warning, because the
|
||
push itself succeeded. Always: push, resubmit, then reply with the printed interdiff URL. See
|
||
`machine-docs/PR-WORKFLOW.md`, "A push does NOT advance the round".
|
||
|
||
## The testing gateway — public domains for tailnet boxes
|
||
|
||
A box on the tailnet with no public IP can still have a real HTTPS domain: the shared testing
|
||
gateway holds a wildcard record for `*.gtest.commoninternet.net` and forwards by SNI.
|
||
|
||
```bash
|
||
python3 engine/tools/gateway-domain.py add myapp # -> myapp.gtest.commoninternet.net -> this box
|
||
python3 engine/tools/gateway-domain.py list
|
||
python3 engine/tools/gateway-domain.py remove myapp
|
||
```
|
||
|
||
The admin password is `gateway.admin_password` in the secret store below; the tool reads it
|
||
itself. **The gateway does not terminate TLS** — it proxies the encrypted stream, so your box
|
||
serves the certificate for that name. Full detail, including how to get a cert and why hostname
|
||
backends are refused, is in the `gateway-domain` skill (`skills/gateway-domain/SKILL.md`).
|
||
|
||
## Secrets — one encrypted store, never in git
|
||
|
||
**Every credential on an orchestrator host lives in one sops+age encrypted file. Do not put a
|
||
secret anywhere else** — not in a git remote URL, not in a project `.env`, not in a prompt.
|
||
|
||
```
|
||
/secrets/store.yaml the store: sops+age ciphertext, mode 0600
|
||
~/.config/sops/age/keys.txt the age private key — the ONE plaintext secret, mode 0600
|
||
```
|
||
|
||
`/secrets/` is deliberately **not a git repo and has no remote**, so there is no path by
|
||
which a `git add`/`git push` can leak it; the store is ciphertext at rest anyway.
|
||
|
||
Read it with `engine/secrets.py` (stdlib + the `sops` binary, no Python deps):
|
||
|
||
```python
|
||
from secrets import get, get_group
|
||
cookie = get("tangled.cookie") # a single value
|
||
env = get_group("cc_ci_testenv") # a whole group as a dict
|
||
```
|
||
|
||
```sh
|
||
python3 engine/secrets.py list # group/key NAMES only — never prints values
|
||
python3 engine/secrets.py get tangled.cookie # one value on stdout
|
||
python3 engine/secrets.py materialize tangled-session # write a runtime file from the store
|
||
sops /secrets/store.yaml # add/edit: decrypts to $EDITOR, re-encrypts on save
|
||
```
|
||
|
||
**One home per secret — two shapes.**
|
||
|
||
*Values our code reads* live **in the store**; import this module and ask for them. Nothing is
|
||
written to disk (`engine/.tangled-session` is gone — the tangled tools read `tangled.cookie`).
|
||
|
||
*Secrets a third party reads from a fixed path* (ssh keys, a systemd `EnvironmentFile`, nix's
|
||
`authKeyFile`, a TLS keypair) live as **real files in `/secrets/files/`, symlinked from the path
|
||
the consumer expects**:
|
||
|
||
```
|
||
~/.ssh/tangled-ed25519 -> /secrets/files/tangled-ed25519
|
||
/etc/ts-auth-key -> /secrets/files/ts-auth-key
|
||
/srv/cc-ci/.testenv -> /secrets/files/cc-ci.testenv
|
||
```
|
||
|
||
The consumer is unchanged and unaware; the file exists once, in one directory, at 0600. Do **not**
|
||
also copy such a secret into `store.yaml` — that is two sources of truth again.
|
||
|
||
For a one-off where neither shape fits, inject at run time and leave nothing behind:
|
||
|
||
```sh
|
||
python3 engine/secrets.py exec-env <group> -- some-command # group as env vars
|
||
python3 engine/secrets.py with-file <group.key> -- cmd -i {} # 0600 file in a private
|
||
# tmpdir, deleted on exit
|
||
```
|
||
|
||
**The symlink exception: apps that rewrite their own credential file.** An app that refreshes an
|
||
OAuth token by writing `auth.json` atomically (write-temp + rename) **replaces the symlink with a
|
||
regular file**, silently splitting the home again. `~/.local/share/opencode/auth.json` is such a
|
||
file, so it stays where it is and is deliberately *not* centralised. Before symlinking a secret,
|
||
ask whether its owner ever writes it back.
|
||
|
||
**Rules of thumb**
|
||
|
||
- Prefer ssh remotes over `https://user:pass@host/...`. A password in a remote URL is printed by
|
||
`git remote -v`, copied into every clone, and survives in `.git/config` where nobody looks.
|
||
- A private key is `chmod 600`. Check with
|
||
`find . -name '*.key' -o -name 'id_*' ! -name '*.pub' -perm /044`.
|
||
- Anything a project must keep on disk goes in `.gitignore` **and** gets its real home in the store.
|
||
|
||
---
|
||
|
||
## Nix
|
||
|
||
A `flake.nix` provides a reproducible devShell with the runtime deps (`python311` for stdlib
|
||
`tomllib`, plus `tmux` and `git`):
|
||
|
||
```bash
|
||
nix develop # enter the shell
|
||
nix develop -c python3 agents.py selftest # or run one command in it
|
||
nix flake check # evaluate + build the devShell
|
||
```
|
||
|
||
The agent CLIs themselves (`claude`, `opencode`) are **external, non-Nix tools** — install them
|
||
per their own docs and make sure they are on `PATH` before launching live agents. The devShell
|
||
documents this in its banner.
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
The `tests/` directory holds the harness's own test suite. One runner drives everything:
|
||
|
||
```bash
|
||
nix develop -c ./tests/run.sh # unit tests always; live backend smokes when available
|
||
# or just: ./tests/run.sh # (python3 + tmux must be on PATH)
|
||
```
|
||
|
||
What it runs:
|
||
|
||
- **Unit tests** (`tests/test_unit.py`) — pure logic, **no agents spawned, no live tmux sessions**.
|
||
Cover config load + defaults merge, kickoff-template assembly, the phase machine (advance on the
|
||
done marker, idempotent sequence-complete, append-a-phase resumes), usage-limit reset-banner
|
||
parsing, `WAITING-UNTIL` / stall parsing, and the per-backend activity detectors (claude +
|
||
opencode footers). Always run; a failure fails the suite. Run them alone with
|
||
`python3 -m unittest discover -s tests` (or `python3 tests/test_unit.py`).
|
||
- **Live backend smokes** (`tests/smoke_claude.sh`, `tests/smoke_opencode.sh`) — each brings a
|
||
throwaway scratch project up **through `agents.py`** on a real backend, in a fully isolated
|
||
sandbox (its own unique `session_prefix`, a temp `log_dir`, and — for opencode — a dedicated
|
||
server on a non-default port `AOTEST_OC_PORT`, default `4097`), confirms the session attaches and
|
||
`status` reports it RUNNING, then `down`s it and cleans up (no leftover sessions, port freed).
|
||
Each **SKIPs gracefully** (exit 0) when its backend's binary or creds are unavailable. Useful env:
|
||
`CLAUDE_BIN` / `OPENCODE_BIN`, `AOTEST_MODEL`, `AOTEST_OC_PORT`, `AOTEST_OC_CREDS`.
|
||
- **Isolation sanity** — after the live runs, the runner asserts no `aotest-*` tmux sessions leaked
|
||
and reports that any live sessions are untouched.
|
||
|
||
The smokes are safe by construction: a unique per-run session prefix (never `cc-ci-` or any real
|
||
project's), a dedicated opencode port (never `4096`), and a cleanup trap that fires on success,
|
||
failure, and Ctrl+C.
|
||
|
||
---
|
||
|
||
## Adding things
|
||
|
||
- **Add an agent** — add an `[[agent]]` block; `agents.py up <name>`. No code change.
|
||
- **Add a backend** — add a `[backend.<name>]` block (`bin`, `prompt_delivery`, the regexes);
|
||
point an agent at it with `backend = "<name>"`.
|
||
- **Add / append a phase** — add an entry to `[loop].phases`; the watchdog advances into it
|
||
automatically (clearing a stale `SEQUENCE-COMPLETE` if the sequence had finished).
|
||
- **Change a model or backend** — edit the field (or a phase's `models = {}`), then
|
||
`agents.py down <name> && agents.py up <name>`. The watchdog re-reads the file; it won't fight you.
|