mfowler
b46dca003c
Combined run 1 (orig/min/stateless) + run 2 (lean/stateless). Key result: the SAME stateless variant used 5.96M tokens in run 1 and 9.27M in run 2 (±55%) — nondeterministic iteration count dominates every between-variant gap. So: - prose minimization ~-6% (small, same-invocation) - lean (full per-gate review) ~= stateless (batched): full review is ~free - the earlier "-45% from context hygiene" is NOT reproducible — mostly noise Honest conclusion: need >=5 runs/variant to resolve the context-hygiene effect; log_tokens now makes that easy to collect. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
agent-orchestrator-benchmark
Benchmarks for the agent-orchestrator
harness — vendored here as the engine/ submodule, pinned at a ref that ships the example variants
being compared.
What it measures
A head-to-head between two example variants in the engine:
builder-adversary— the original Builder/Adversary loop-pair prompts.builder-adversary-min— the same pattern with the role + kickoff prompts compressed to minimal tokens.
The benchmark confirms each variant independently succeeds on the same task (no shared context) and clocks the tokens each uses.
Run
git submodule update --init # fetch the vendored engine (first time)
./run-bench.sh # writes RESULTS.md
Needs claude on PATH and python/timeout. Both variants run on Sonnet
(claude-sonnet-4-6) for Builder and Adversary.
How it works
run-bench.sh assembles exactly the prompt the harness would send a loop agent (the variant's
kickoff.md with {phase_id}/{plan}/{status}/{role} substituted, then the role prompt), then drives
one Builder pass and one Adversary pass as separate headless claude -p sessions — fresh
context each, so the two variants (and the two roles) share no context. The Builder builds and
commits in its own repo; the Adversary cold-verifies from its own clone. The script then re-runs
the task's Definition-of-Done check itself and reads the Adversary's verdict, and tallies tokens from
claude -p --output-format json.
The test problem is plans/roman.md — an integer→Roman-numeral CLI with a stdlib
unittest suite (deterministic, fully local, cold-verifiable, and not present in either example).
Caveats
- This is a controlled single pass per variant (N=1; expect run-to-run variance), not the full
self-paced watchdog loop. It measures task effectiveness + prompt token cost, not the live
loop / handoff / liveness machinery (that needs a real
engine/agents.py uprun). - Each
claude -pcall carries a fixed ~24k-token cached system-prompt/tool overhead, and most tokens come from the agentic work itself — so the prompt-size difference is a small slice of the total.RESULTS.mdreports the static prompt size separately so the minimisation is visible.
Layout
engine/ agent-orchestrator, vendored as a submodule (the variants live in engine/examples/)
plans/roman.md the test problem (single source of truth + Definition of Done)
run-bench.sh the runner
RESULTS.md generated by run-bench.sh