mfowler
27df2c7b55
A standalone repo (engine vendored as a submodule at the examples commit) that runs a head-to-head between the builder-adversary and builder-adversary-min example variants: same task, independent headless runs, both on Sonnet, with token counts. Includes the roman-numeral test problem and run-bench.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
58 lines
2.6 KiB
Markdown
58 lines
2.6 KiB
Markdown
# agent-orchestrator-benchmark
|
|
|
|
Benchmarks for the [`agent-orchestrator`](https://git.autonomic.zone/recipe-maintainers/agent-orchestrator)
|
|
harness — vendored here as the `engine/` submodule, pinned at a ref that ships the example variants
|
|
being compared.
|
|
|
|
## What it measures
|
|
|
|
A head-to-head between two example variants in the engine:
|
|
|
|
- **`builder-adversary`** — the original Builder/Adversary loop-pair prompts.
|
|
- **`builder-adversary-min`** — the same pattern with the role + kickoff prompts compressed to
|
|
minimal tokens.
|
|
|
|
The benchmark confirms each variant **independently succeeds** on the same task (no shared context)
|
|
and **clocks the tokens** each uses.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
git submodule update --init # fetch the vendored engine (first time)
|
|
./run-bench.sh # writes RESULTS.md
|
|
```
|
|
|
|
Needs `claude` on `PATH` and `python`/`timeout`. Both variants run on **Sonnet**
|
|
(`claude-sonnet-4-6`) for Builder and Adversary.
|
|
|
|
## How it works
|
|
|
|
`run-bench.sh` assembles exactly the prompt the harness would send a loop agent (the variant's
|
|
`kickoff.md` with `{phase_id}/{plan}/{status}/{role}` substituted, then the role prompt), then drives
|
|
one **Builder** pass and one **Adversary** pass as separate headless `claude -p` sessions — fresh
|
|
context each, so the two variants (and the two roles) share no context. The Builder builds and
|
|
commits in its own repo; the Adversary cold-verifies from its **own clone**. The script then re-runs
|
|
the task's Definition-of-Done check itself and reads the Adversary's verdict, and tallies tokens from
|
|
`claude -p --output-format json`.
|
|
|
|
The test problem is [`plans/roman.md`](plans/roman.md) — an integer→Roman-numeral CLI with a stdlib
|
|
`unittest` suite (deterministic, fully local, cold-verifiable, and not present in either example).
|
|
|
|
### Caveats
|
|
|
|
- This is a **controlled single pass** per variant (N=1; expect run-to-run variance), not the full
|
|
self-paced watchdog loop. It measures task effectiveness + prompt token cost, **not** the live
|
|
loop / handoff / liveness machinery (that needs a real `engine/agents.py up` run).
|
|
- Each `claude -p` call carries a fixed ~24k-token cached system-prompt/tool overhead, and most
|
|
tokens come from the agentic work itself — so the prompt-size difference is a small slice of the
|
|
total. `RESULTS.md` reports the static prompt size separately so the minimisation is visible.
|
|
|
|
## Layout
|
|
|
|
```
|
|
engine/ agent-orchestrator, vendored as a submodule (the variants live in engine/examples/)
|
|
plans/roman.md the test problem (single source of truth + Definition of Done)
|
|
run-bench.sh the runner
|
|
RESULTS.md generated by run-bench.sh
|
|
```
|