121 lines
7.3 KiB
Markdown
121 lines
7.3 KiB
Markdown
# Findings — Builder/Adversary prompt & verification-cadence benchmark
|
||
|
||
A controlled study of what actually drives **token cost** in the agent-orchestrator Builder/Adversary
|
||
loop, on a fixed, well-specified task.
|
||
|
||
- **Task:** build a 3-phase Python calculator (lexer → parser → evaluator), each phase with 4–6
|
||
cold-verifiable Definition-of-Done gates. Deliberately offline and deterministic so it stresses the
|
||
*protocol*, not infrastructure.
|
||
- **How:** each variant run autonomously to `SEQUENCE-COMPLETE` via the real harness (`agents.py up`
|
||
— Builder + Adversary loop pair + watchdog), **5 runs each** (N=5).
|
||
Both loops on **claude-sonnet-4-6**. Tokens summed from each loop's Claude Code session
|
||
transcripts. The deliverable is behaviorally identical across all variants (verified on a
|
||
24-expression probe), so this compares like-for-like.
|
||
- **Full data:** [`RESULTS-campaign.md`](RESULTS-campaign.md) (analysis), `RESULTS-campaign.md.data`
|
||
(raw per-run rows). Every run's git repo is preserved under `/tmp/ao-campaign-*` and `/tmp/ao-solo-*`.
|
||
|
||
## The variants
|
||
|
||
| variant | what changes | engine example |
|
||
|---|---|---|
|
||
| `builder-adversary` (orig) | the original full prompts; Adversary verifies **per phase** | `examples/builder-adversary` |
|
||
| `builder-adversary-min` | prompts compressed to minimal tokens | `examples/builder-adversary-min` |
|
||
| `builder-adversary-stateless` | orig + **context hygiene** (compact per checkpoint, read diffs not trees, lean loads) | `examples/builder-adversary-stateless` |
|
||
| `builder-adversary-lean` | orig + context hygiene + **per-gate** review (one claim/verdict per gate) | `examples/builder-adversary-lean` |
|
||
| `builder-adversary-deferred` | orig; Adversary verifies **once, after the whole build** (a final comprehensive `review` phase) | `examples/builder-adversary-deferred` |
|
||
| `builder-solo` | **no Adversary** — a single Builder that self-certifies | `examples/builder-solo` |
|
||
|
||
(stateless/lean/deferred are all built on the *full original* prompts, so they isolate their one
|
||
change without the minimal-prompt confound.)
|
||
|
||
## Headline results — median tokens (5 runs each)
|
||
|
||
| variant | adversary verifies… | median tokens | vs orig | commits | LOC |
|
||
|---|---|--:|--:|--:|--:|
|
||
| **builder-solo** | never (self-certify) | **2.77M** | −79% | 5 | 426 |
|
||
| **min** | per phase *(minimal prompts)* | 9.77M | −25% | 15 | 367 |
|
||
| **stateless** | per phase *(+context hygiene)* | 10.12M | −22% | 14 | 400 |
|
||
| **orig** | per **phase** | 13.04M | — | 14 | 449 |
|
||
| **deferred** | once, after **whole build** | 12.89M | −1% | 12 | 425 |
|
||
| **lean** | per **gate** | 13.41M | +3% | 28 | 390 |
|
||
|
||
## The two big findings
|
||
|
||
### 1. The adversary's *existence* costs ~4.7× — its *cadence* barely matters.
|
||
Every loop-pair variant lands near **~13M tokens regardless of how the review is chunked** —
|
||
per-gate (`lean`, 28 commits), per-phase (`orig`, 14), or one deferred pass (`deferred`, 10).
|
||
`builder-solo` (no adversary) is **2.77M**. So the dominant cost is **whether an independent cold
|
||
re-verification happens at all**, not how it's scheduled. The verification *work* is roughly
|
||
conserved; chunking it finer or coarser mostly changes the commit/handshake count — which is itself
|
||
nearly token-neutral.
|
||
|
||
### 2. Deferred review was the surprise — and the loser.
|
||
Hypothesis: deferring to one pass would be cheapest (fewest handshakes ≈ solo-build + one review).
|
||
It wasn't — `deferred` ≈ `orig`. Handshakes *did* drop (10 commits), but the **single comprehensive
|
||
review is itself expensive** (the Adversary re-verifies the whole calculator + cross-feature probes
|
||
in one shot), so total tokens stayed put. And it carries the downside that the independent check
|
||
**arrives late** — late-rework risk, plus self-certification drift on the build phases. Worst of both
|
||
for this task.
|
||
|
||
## The levers, ranked
|
||
|
||
1. **Drop the adversary → ~−79%** — but you lose all independent verification. On this clean,
|
||
well-specified task `solo` produced correct calculators, so the adversary bought no *measured*
|
||
quality here — but it is **insurance against self-certification rubber-stamping a bug**, whose
|
||
value shows on ambiguous/underspecified work this benchmark can't stress.
|
||
2. **Context hygiene → −22%** — the **only clean win**: same review effort (same commits/LOC as
|
||
orig), just less context carried and reloaded each turn. (`stateless` vs `orig`.)
|
||
3. **Minimal prompts → −25%, but not free** — ~⅓ of the saving comes from the agents writing **~25%
|
||
fewer tests** (the compressed prompts drop the emphatic "try to break it / paste the output /
|
||
a red test is information" language that drives thorough testing). Same features, thinner test
|
||
suite.
|
||
4. **Review cadence → ~0%** — per-gate / per-phase / per-build are interchangeable on *cost*; choose
|
||
for **quality and latency**, not tokens: finer = earlier defect-catching at slight overhead;
|
||
coarser = late but holistic (better at cross-feature bugs).
|
||
|
||
## Why: cost is *process*, not *product*
|
||
|
||
Pooled across all 28 successful runs:
|
||
|
||
| tokens vs | Pearson r |
|
||
|---|--:|
|
||
| duration | **+0.83** |
|
||
| commits (review rounds) | **+0.79** |
|
||
| LOC (code shipped) | **−0.04** |
|
||
|
||
Token cost tracks how long the loop **runs and verifies**, and is **uncorrelated with how much code
|
||
ships**. The deliverable (LOC, behavior) is near-constant across variants; the cost variance is all
|
||
process intensity.
|
||
|
||
## Methodology notes & caveats
|
||
|
||
- **N matters.** A single full-loop run is wildly nondeterministic: the *same* variant varied **±55%**
|
||
run-to-run early on, which is why this is N=5. (An early single-run claim of "context hygiene halves
|
||
tokens" did **not** reproduce — the real, stable figure is −22%.)
|
||
- **Variance source:** number of review rounds / retries, not output size.
|
||
- **Real failures excluded** (2 of 27 loop-pair runs): a wedge and a usage-limit/timeout collision;
|
||
superseded by clean re-runs. `LIMIT`-flagged runs (a usage-limit *pause* inflates duration without
|
||
adding tokens) are kept for token totals but excluded from `tokens/sec`.
|
||
- **Scope:** one task, one model (Sonnet), one harness. The *relative* findings should generalize;
|
||
absolute numbers are task-specific. The adversary's quality value is **not** measured here (the task
|
||
is too well-specified to stress it).
|
||
|
||
## Practical guidance
|
||
|
||
- **Want to cut tokens without losing the independent check?** Use **context hygiene** (the `stateless`
|
||
pattern). It's the only free lunch.
|
||
- **Don't pay for minimal prompts with test coverage** — keep the emphatic testing language unless you
|
||
genuinely want less testing.
|
||
- **Pick review cadence for the work, not the bill:** per-gate to catch regressions early in long
|
||
phases; per-phase as a sane default; deferred only when features are independent and cheap to fix
|
||
late (it saves nothing and checks late).
|
||
- **`solo` is ~5× cheaper** — reasonable for low-stakes / well-specified work, but you're trusting the
|
||
builder to grade its own homework.
|
||
|
||
---
|
||
_Artifacts in this repo: `FINDINGS.md` (this summary), `RESULTS-campaign.md` (per-variant stats +
|
||
ratios + full raw table), `RESULTS-campaign.md.data` (raw rows), `analyze.py` (regenerates the
|
||
analysis), `run-harness-bench.sh` / `run-solo-bench.sh` (the runners), `plans/calc/` (the task), and
|
||
the six `engine/examples/builder-adversary*` variants. All variants are N=5 (two `deferred` and the
|
||
`min`/`lean` wedge/limit failures excluded; see RESULTS for the raw rows)._
|