docs(examples): add builder-solo — single builder, no adversary (control)
A single Builder that builds AND self-verifies (same DoD rigor), with NO independent Adversary and no claim/review handoff. The control for measuring what the AI adversary costs (its tokens, ~half of a loop-pair run) and buys (independent cold verification vs self-certification). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
27
examples/builder-solo/README.md
Normal file
27
examples/builder-solo/README.md
Normal file
@ -0,0 +1,27 @@
|
||||
# Builder-solo example — no Adversary (self-verification baseline)
|
||||
|
||||
A single **Builder** agent, same task spec as [`../builder-adversary`](../builder-adversary/), but
|
||||
with **no Adversary**: the Builder builds *and* verifies its own work, then self-certifies `## DONE`.
|
||||
No `claim(`/`review(` handoff — there's nothing to hand off to.
|
||||
|
||||
This is the **control** for the AI-as-adversary design. Comparing it against `builder-adversary` on
|
||||
the same task answers two things:
|
||||
|
||||
- **Cost:** how much of a run's tokens is the independent Adversary? (In the loop-pair runs the
|
||||
Adversary is ~45–53% of the total — this variant removes that.)
|
||||
- **Quality:** does an independent cold verifier catch things a self-checking builder misses? Self-
|
||||
certification has an obvious failure mode — the same agent that wrote the bug decides whether it's
|
||||
a bug. This variant measures what you give up by dropping the second pair of eyes.
|
||||
|
||||
The Builder's role prompt keeps the same verification *rigor* (run every DoD check, try to break it,
|
||||
paste observed output, no self-rubber-stamping) — the only thing removed is the **independent**
|
||||
adversary. So the comparison is "independent verification vs self-verification," not "verification vs
|
||||
none."
|
||||
|
||||
```bash
|
||||
python3 ../../agents.py status --config agents.toml
|
||||
python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
|
||||
```
|
||||
|
||||
The `agent-orchestrator-benchmark` repo runs this head-to-head with the other variants on the same
|
||||
multi-phase task and reports tokens + the efficiency ratios.
|
||||
68
examples/builder-solo/agents.toml
Normal file
68
examples/builder-solo/agents.toml
Normal file
@ -0,0 +1,68 @@
|
||||
# examples/builder-solo — a single Builder, NO Adversary (self-verification baseline).
|
||||
#
|
||||
# Same pattern + same task spec as ../builder-adversary, but there is only ONE agent: the Builder
|
||||
# builds AND verifies its own work, then self-certifies "## DONE". This is the control for measuring
|
||||
# what the independent AI Adversary actually costs (its tokens) and buys (independent cold
|
||||
# verification). No claim/review handoff — nothing to hand off to.
|
||||
#
|
||||
# python3 ../../agents.py status --config agents.toml
|
||||
# python3 ../../agents.py up --config agents.toml # needs `claude` on PATH
|
||||
|
||||
[watchdog]
|
||||
signal_interval = 30
|
||||
heavy_interval = 300
|
||||
limit_probe_fallback = 300
|
||||
limit_reset_slack = 45
|
||||
stall_grace = 180
|
||||
|
||||
[defaults]
|
||||
session_prefix = "solo-"
|
||||
log_dir = ".ao-state"
|
||||
backend = "claude" # set to "demo" for a dependency-free mechanics-only run
|
||||
model = "claude-sonnet-4-6"
|
||||
watch = "heal"
|
||||
|
||||
[backend.claude]
|
||||
bin = "claude"
|
||||
flags = "--dangerously-skip-permissions"
|
||||
remote_control = true
|
||||
supports_resume = true
|
||||
prompt_delivery = "arg"
|
||||
process_name = "claude"
|
||||
submit_key = "Enter"
|
||||
stall_idle = 300
|
||||
active_re = "esc to interrupt|Running tool|⠇|⠙|· \\d+"
|
||||
limit_re = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
|
||||
fatal_re = "redacted_thinking|blocks cannot be modified|cannot be modified"
|
||||
|
||||
[backend.demo]
|
||||
bin = "echo '[demo] {session} up (kickoff: {kickoff})'; exec sleep 1000000"
|
||||
prompt_delivery = "exec"
|
||||
|
||||
# The lone builder — builds and self-verifies.
|
||||
[[agent]]
|
||||
name = "builder" # tmux session: solo-builder
|
||||
kind = "loop"
|
||||
role = "builder" # kickoff = prompts/kickoff.md (per phase) + prompts/builder.md
|
||||
dir = "./work"
|
||||
watch = "heal+stall"
|
||||
|
||||
[[service]]
|
||||
name = "cleanlogs"
|
||||
command = "python3 ../../agent-log.py follow-all"
|
||||
dir = "."
|
||||
|
||||
# Phase machine. No handoff (single agent); the watchdog auto-advances when the builder writes
|
||||
# "## DONE" to the phase status file (read from handoff.repo's state_subdir).
|
||||
[loop]
|
||||
state_file = "phase-idx"
|
||||
resume_phase = true
|
||||
auto_advance = true
|
||||
done_marker = "## DONE"
|
||||
kickoff_template = "prompts/kickoff.md"
|
||||
roles_dir = "prompts"
|
||||
handoff = { repo = "./work", state_subdir = "machine-docs" }
|
||||
phases = [
|
||||
{ id = "wc", plan = "plans/wc.md", status = "STATUS-wc.md" },
|
||||
{ id = "json", plan = "plans/json.md", status = "STATUS-json.md" },
|
||||
]
|
||||
0
examples/builder-solo/machine-docs/.gitkeep
Normal file
0
examples/builder-solo/machine-docs/.gitkeep
Normal file
32
examples/builder-solo/plans/json.md
Normal file
32
examples/builder-solo/plans/json.md
Normal file
@ -0,0 +1,32 @@
|
||||
# Phase `json` — machine-readable output
|
||||
|
||||
**Mission.** Extend the `wc.py` from the previous phase with a `--json` mode, without regressing any
|
||||
`wc`-phase behaviour. Single source of truth for this phase.
|
||||
|
||||
(The phase config gives the Builder `claude-opus-4-8` for this phase — an example of a per-phase
|
||||
model override; the Adversary stays on the default model.)
|
||||
|
||||
## Definition of Done
|
||||
|
||||
- **D1 — json output.** `python wc.py --json FILE` prints a single JSON object:
|
||||
`{"lines": N, "words": N, "chars": N, "file": "FILE"}` (valid JSON, parseable by `json.loads`).
|
||||
With stdin (no FILE), `"file"` is `null`.
|
||||
- **D2 — composes with flags.** `--json` honours `-l/-w/-c`: only the requested counts appear as keys
|
||||
(plus `file`). E.g. `wc.py --json -l FILE` → `{"lines": N, "file": "FILE"}`.
|
||||
- **D3 — no regression.** Every `wc`-phase gate (D1–D4 there) still passes unchanged.
|
||||
- **D4 — tests green.** `test_wc.py` is extended for the JSON cases and `pytest -q` is all-green.
|
||||
|
||||
## How the Adversary verifies (cold)
|
||||
|
||||
```bash
|
||||
pytest -q # D4 + D3 regression
|
||||
printf 'a b c\nd e\n' > /tmp/f.txt
|
||||
python wc.py --json /tmp/f.txt | python -c 'import sys,json; d=json.load(sys.stdin); \
|
||||
assert d=={"lines":2,"words":5,"chars":10,"file":"/tmp/f.txt"}, d; print("ok")' # D1
|
||||
python wc.py --json -l /tmp/f.txt # D2: expect {"lines": 2, "file": "/tmp/f.txt"}
|
||||
```
|
||||
|
||||
The Builder restates the exact commands, expected JSON, and commit sha in
|
||||
`machine-docs/STATUS-json.md`. When every DoD item has a fresh PASS in `machine-docs/REVIEW-json.md`
|
||||
and there is no `## VETO`, the Builder writes `## DONE` to `STATUS-json.md` — this is the last phase,
|
||||
so the watchdog then fires the one-shot `reporter` (see `agents.toml` `[loop].on_complete`).
|
||||
43
examples/builder-solo/plans/wc.md
Normal file
43
examples/builder-solo/plans/wc.md
Normal file
@ -0,0 +1,43 @@
|
||||
# Phase `wc` — a word-count CLI
|
||||
|
||||
**Mission.** Build a small, dependency-free `wc` clone in Python: a script `wc.py` in the work repo
|
||||
that counts lines, words, and characters, plus a `pytest` suite. This is the single source of truth
|
||||
for the phase — the Builder builds to the Definition of Done below; the Adversary cold-verifies it.
|
||||
|
||||
This task is deliberately tiny and fully local (no network, no services) so the example exercises the
|
||||
loop-pair *protocol* — claim → cold-verify → PASS/FAIL handshake — not infrastructure.
|
||||
|
||||
## Definition of Done
|
||||
|
||||
Each Dn is an independent gate. The Builder claims it (`claim(Dn): …`); the Adversary records a fresh
|
||||
PASS in `machine-docs/REVIEW-wc.md` after re-running the check from its own clone.
|
||||
|
||||
- **D1 — default output.** `python wc.py FILE` prints exactly `<lines> <words> <chars> <FILE>`
|
||||
(counts whitespace-separated words, `\n`-terminated lines, and bytes for `chars`), matching GNU
|
||||
`wc` on ASCII input.
|
||||
- **D2 — flags.** `-l`, `-w`, `-c` restrict the output to that single count (e.g. `wc.py -l FILE`
|
||||
prints `<lines> <FILE>`). Flags may combine; output order is lines, words, chars.
|
||||
- **D3 — stdin.** With no FILE argument, `wc.py` reads stdin and prints the counts with no filename.
|
||||
- **D4 — tests green.** A `test_wc.py` runs under `pytest -q` with **0 failures**, covering: an empty
|
||||
file (`0 0 0`), a multi-line fixture, the no-trailing-newline case, and each flag.
|
||||
|
||||
## How the Adversary verifies (cold)
|
||||
|
||||
From a fresh clone of the work repo:
|
||||
|
||||
```bash
|
||||
pytest -q # D4: must be all-green
|
||||
printf 'a b c\nd e\n' > /tmp/f.txt
|
||||
python wc.py /tmp/f.txt # D1: expect "2 5 10 /tmp/f.txt"
|
||||
python wc.py -l /tmp/f.txt # D2: expect "2 /tmp/f.txt"
|
||||
printf 'a b c\nd e\n' | python wc.py # D3: expect "2 5 10"
|
||||
```
|
||||
|
||||
Expected outputs are above — the Builder must restate them (and the exact commands, plus the commit
|
||||
sha) in `machine-docs/STATUS-wc.md` so the Adversary can re-run without reading the Builder's
|
||||
reasoning. Any mismatch is a FAIL with repro steps in `machine-docs/REVIEW-wc.md`.
|
||||
|
||||
## Out of scope (defer to a later phase or DEFERRED.md)
|
||||
|
||||
Multibyte/`-m` char counting, `--files0-from`, multiple-file totals, locale handling. JSON output is
|
||||
the next phase (`plans/json.md`).
|
||||
15
examples/builder-solo/prompts/builder.md
Normal file
15
examples/builder-solo/prompts/builder.md
Normal file
@ -0,0 +1,15 @@
|
||||
You are the **Builder** — and the ONLY agent. There is no Adversary. You build to the plan's DoD **and verify your own work** before certifying it done. Read the phase plan (the SSOT) and build to its DoD.
|
||||
|
||||
Loop: run `/loop` (no interval), one unit of work per wake. Liveness (watchdog-enforced): cap every wait at 10 min; before going idle your LAST output line MUST be exactly `WAITING-UNTIL: <ISO-8601 UTC>`; compact at ~80% context.
|
||||
|
||||
Git: `pull --rebase`, smallest change, commit, push; never `--force`. Prefix commits conventionally (`feat/fix/test/status/…`).
|
||||
|
||||
**SELF-VERIFICATION (this replaces the Adversary — do it rigorously; do NOT rubber-stamp yourself):**
|
||||
- For each DoD gate, RUN the exact check the plan specifies (its command + expected output) from a clean state and confirm it passes. Don't assume — execute it and read the actual output.
|
||||
- Actively try to BREAK your own work: edge cases, malformed input, the failure modes the plan names. A gate you can break is not done.
|
||||
- Record it in `machine-docs/{status}` (or STATUS for the phase): per gate, WHAT it is, the exact command, the EXPECTED result, and the OBSERVED result (paste the real output).
|
||||
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
|
||||
|
||||
Done: write "## DONE" to the phase status file ONLY after every DoD gate has a real, observed PASS from your own verification and you have no outstanding self-found defect.
|
||||
|
||||
Begin: read the plan, then enter the loop.
|
||||
7
examples/builder-solo/prompts/kickoff.md
Normal file
7
examples/builder-solo/prompts/kickoff.md
Normal file
@ -0,0 +1,7 @@
|
||||
*** PHASE {phase_id} ***
|
||||
Plan (this phase's single source of truth): {plan} — read it fully now; it defines the mission and the Definition of Done (DoD).
|
||||
You are the ONLY agent — there is no separate Adversary. You BUILD and you VERIFY YOUR OWN WORK.
|
||||
Track state under machine-docs/ (create if missing): {status} and JOURNAL-{phase_id}.md.
|
||||
Done = you write "## DONE" to machine-docs/{status} ONLY after every DoD item passes your own observed verification (run the checks, paste the output).
|
||||
|
||||
=== role ===
|
||||
Reference in New Issue
Block a user