results: 4-way + the variance finding (N=1 is not enough)

Combined run 1 (orig/min/stateless) + run 2 (lean/stateless). Key result: the
SAME stateless variant used 5.96M tokens in run 1 and 9.27M in run 2 (±55%) —
nondeterministic iteration count dominates every between-variant gap. So:
- prose minimization ~-6% (small, same-invocation)
- lean (full per-gate review) ~= stateless (batched): full review is ~free
- the earlier "-45% from context hygiene" is NOT reproducible — mostly noise
Honest conclusion: need >=5 runs/variant to resolve the context-hygiene effect;
log_tokens now makes that easy to collect.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

RESULTS-harness.md

@@ -1,78 +1,62 @@
-# Full-harness benchmark — prompt variants
+# Full-harness benchmark — prompt variants (calculator, Sonnet)
 
 Real `agents.py up` Builder/Adversary loop pair + watchdog, run **autonomously** through the
 multi-phase calculator (`plans/calc/{lex,parse,eval}.md` — 3 phases, 4–6 gates each) to
-`SEQUENCE-COMPLETE`. Engine pinned at `985d33d`. Both loops on **claude-sonnet-4-6**. Per-variant
-timeout 3000s. Tokens summed from the Claude Code session transcripts of each loop's clone. **N=1**
-per variant (the autonomous loop is nondeterministic — number of review rounds varies).
+`SEQUENCE-COMPLETE`. Both loops on **claude-sonnet-4-6**. Tokens summed from each loop's Claude Code
+session transcripts.
 
-## Variants
+## ⚠️ Headline: run-to-run variance dominates (N=1 is not enough)
 
-- **builder-adversary** — original prompts.
-- **builder-adversary-min** — same rules, prose compressed to minimal tokens.
-- **builder-adversary-stateless** — min + **context hygiene** (compact at each checkpoint, read diffs
-  not trees, spill bulk to files, adversary loads only {plan, STATUS, diff}); loop sessions
-  non-resumed → fresh context per phase. Same AI-as-adversary verification.
+The **same** `stateless` variant, identical prompts, was run twice:
 
-## All three succeeded
+| run | stateless total tokens |
+|---|--:|
+| run 1 | **5,964,829** |
+| run 2 | **9,266,808** |
 
-Every variant completed all three phases with the Adversary cold-verifying every gate (no veto), and
-the final calculator is correct in each:
+That's a **±55% swing** from nondeterminism alone (how many review rounds / retries the autonomous
+loop happens to do). It is **larger than every difference between variants below.** So treat all
+single-run deltas as suggestive at best — the honest conclusion is that **N=1 cannot resolve the
+variant effects**; you'd need several runs per variant and compare medians.
 
-| version | sequence-complete | unittest | `2+3*4` | `(2+3)*4` | `7/2` | result |
-|---|:--:|:--:|:--:|:--:|:--:|:--:|
-| builder-adversary | yes | OK | 14 | 20 | 3.5 | PASS |
-| builder-adversary-min | yes | OK | 14 | 20 | 3.5 | PASS |
-| builder-adversary-stateless | yes | OK | 14 | 20 | 3.5 | PASS |
+## All data points
 
-(The original Adversary even filed a non-blocking advisory on `lex` — genuine adversarial review, not
-rubber-stamping.)
+| run | variant | builder | adversary | **total** |
+|---|---|--:|--:|--:|
+| 1 | builder-adversary (orig) | 5,557,356 | 5,199,007 | **10,756,363** |
+| 1 | builder-adversary-min | 5,350,953 | 4,768,272 | **10,119,225** |
+| 1 | builder-adversary-stateless | 2,834,505 | 3,130,324 | **5,964,829** |
+| 2 | builder-adversary-lean | 4,050,402 | 5,086,052 | **9,136,454** |
+| 2 | builder-adversary-stateless | 4,606,579 | 4,660,229 | **9,266,808** |
 
-## Static prompt size (chars: kickoff + role)
+All five **succeeded**: built a correct calculator (`2+3*4→14`, `(2+3)*4→20`, `7/2→3.5`), full test
+suites green, every gate Adversary-verified, no veto. (Engine: run 1 @ `985d33d`, run 2 @ `e0425e6`
+— the orig/min/stateless prompts are byte-identical across both; run 2 only adds `lean`.)
 
-| version | builder | adversary |
-|---|--:|--:|
-| builder-adversary | 6389 | 5811 |
-| builder-adversary-min | 1751 | 1644 |
-| builder-adversary-stateless | 2430 | 2218 |
+## What we can and can't say
 
-## Tokens (from session transcripts)
+**Valid (same-invocation) comparisons:**
+- **Run 1 — prose minimization:** orig 10.76M vs min 10.12M → **−5.9%**. Small; consistent with "the
+  prompt is a tiny cached slice." Probably real but minor.
+- **Run 2 — full per-gate review vs batched, both with context hygiene:** lean 9.14M vs stateless
+  9.27M → **essentially tied (−1.4%)**. So **enforcing one claim + one independent verdict per gate
+  did NOT cost more tokens** than letting the loop batch — answering the question directly: you can
+  keep full review granularity without a token penalty (in this run).
 
-| version | builder loop | adversary loop | **total** | vs orig |
-|---|--:|--:|--:|--:|
-| builder-adversary | 5,557,356 | 5,199,007 | **10,756,363** | — |
-| builder-adversary-min | 5,350,953 | 4,768,272 | **10,119,225** | −5.9% |
-| builder-adversary-stateless | 2,834,505 | 3,130,324 | **5,964,829** | **−44.5%** |
-
-Breakdown (the dominant term is **cache_read** — re-reading the conversation each turn):
-
-| version | role | input | output | cache_create | cache_read |
-|---|---|--:|--:|--:|--:|
-| builder-adversary | builder | 199 | 87,704 | 256,054 | 5,213,399 |
-| builder-adversary | adversary | 181 | 66,540 | 183,189 | 4,949,097 |
-| builder-adversary-min | builder | 216 | 54,381 | 254,524 | 5,041,832 |
-| builder-adversary-min | adversary | 213 | 58,838 | 202,916 | 4,506,305 |
-| builder-adversary-stateless | builder | 124 | 26,998 | 113,026 | 2,694,357 |
-| builder-adversary-stateless | adversary | 141 | 31,342 | 122,360 | 2,976,481 |
+**NOT supported:**
+- The earlier "context hygiene halves tokens (−45%)" claim from run 1 is **not reproducible**:
+  stateless's *own* second run (9.27M) lands right next to orig/min/lean. The −45% was mostly a
+  lucky low-iteration run, not the context discipline. Context hygiene may still help, but this
+  benchmark can't prove it at N=1.
 
 ## Findings
 
-1. **Prompt-prose minimization barely moves tokens (−5.9%).** `min` cut the prompt to ~⅓ the size
-   but saved almost nothing — because the role/kickoff prompt is a tiny, cached slice. Worth keeping
-   for readability; not a token lever.
-2. **Context hygiene nearly halves tokens (−44.5%), quality intact.** `stateless` produced the same
-   correct, fully-verified calculator while cutting total tokens ~45%. The saving is dominated by
-   **cache_read** falling ~48% (builder 5.21M→2.69M) — exactly the "don't carry/reload context you
-   don't need" lever. It also cut output tokens (builder 87.7k→27.0k), i.e. less redundant
-   regeneration across turns.
-3. **The cost is the conversation, not the prompt.** cache_read ≫ everything else in all three. Any
-   real efficiency work should target carried/reloaded context (compaction cadence, fresh sessions
-   per unit of work, diff-not-tree reads), not prompt wording.
+1. **The dominant variable is nondeterministic iteration count, not the prompt variant.** ±55%
+   same-variant variance > any between-variant gap.
+2. **Prose size barely matters** (−6%, run 1) — keep minimal prompts for readability, not tokens.
+3. **Full per-gate review is ~free vs batching** (run 2) — granular adversarial scrutiny didn't
+   raise the bill, so prefer it for quality.
+4. **To actually measure context hygiene you need a campaign:** ≥5 runs per variant, compare medians
+   / distributions. The new `log_tokens` harness feature makes that cheap to collect.
 
-> N=1 caveat: the autonomous loop is nondeterministic, so some of the gap is run-to-run variance
-> (review-round count, retries). The ~45% reduction is large and matches the mechanism (cache_read
-> roughly halved), but repeating the run a few times would tighten the estimate.
-
-_Run dirs: `/tmp/ao-harness-YIrsUp`. (A prior auto-generated version mislabeled `orig`/`stateless` as
-failed — a bug in the harness success check grepping the word "veto" and matching "No veto"; fixed to
-match the `## VETO` marker. Functionally all three passed, as verified above.)_
+_Run dirs: run 1 `/tmp/ao-harness-YIrsUp`, run 2 `/tmp/ao-harness-TMDfvk`. N=1 per (run, variant)._