recipe-maintainers/agent-orchestrator-benchmark
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: FINDINGS.md — benchmark synthesis; track raw results data

Browse Source 
Capstone summary of the Builder/Adversary prompt + verification-cadence study:
- adversary EXISTENCE costs ~4.7x (solo 2.8M vs ~13M); cadence is ~token-neutral
- context hygiene is the one clean -22% win; minimal prompts -25% but test less
- deferred review saves nothing (the one comprehensive pass is expensive) + late
- cost is process not product (tokens~duration 0.83, ~commits 0.79, ~LOC -0.04)
All results now in-repo: FINDINGS.md + RESULTS-campaign.md + raw .data + runners.
(deferred N=3, finalizing to N=5.)
This commit is contained in:
notplants
2026-06-16 01:53:34 +00:00
parent 819000417b
commit 3bf3316572
4 changed files with 164 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RESULTS-campaign.md
+13 -7
View File
@@ -1,6 +1,6 @@
# Full-harness benchmark — campaign analysis
Real `agents.py up` Builder/Adversary loop pair + watchdog through the 3-phase calculator to SEQUENCE-COMPLETE. Both loops on Sonnet. Tokens summed from each loop's session transcript; commits = work-repo commit count; LOC = non-blank `calc/*.py` lines (code + tests). 25 successful runs of 27 total.
Real `agents.py up` Builder/Adversary loop pair + watchdog through the 3-phase calculator to SEQUENCE-COMPLETE. Both loops on Sonnet. Tokens summed from each loop's session transcript; commits = work-repo commit count; LOC = non-blank `calc/*.py` lines (code + tests). 28 successful runs of 30 total.
## Per-variant total tokens (successful runs)
@@ -11,6 +11,7 @@ Real `agents.py up` Builder/Adversary loop pair + watchdog through the 3-phase c
| builder-adversary-stateless | 5/5 | 10,122,375 | 10,735,401 | 9,992,834 | 13,009,792 | 1.30x |
| builder-adversary-lean | 5/6 | 13,409,349 | 13,216,582 | 12,101,355 | 13,815,595 | 1.14x |
| builder-solo | 5/5 | 2,773,634 | 2,744,840 | 2,417,528 | 2,948,467 | 1.22x |
| builder-adversary-deferred | 3/3 | 13,366,800 | 13,863,041 | 12,888,082 | 15,334,242 | 1.19x |
## Efficiency ratios — min / median / max (successful runs)
@@ -18,12 +19,13 @@ tokens/sec excludes runs flagged `LIMIT` (a usage-limit pause inflates duration
| variant | tokens / LOC | tokens / sec | tokens / commit |
|---|--:|--:|--:|
| builder-adversary | 23,857 / 30,391 / 32,665 | 8,083 / 11,670 / 13,852 | 793,540 / 935,026 / 1,044,586 |
| builder-adversary | 23,857 / 30,391 / 32,665 | 11,581 / 12,226 / 13,852 | 793,540 / 935,026 / 1,044,586 |
| builder-adversary-min | 23,527 / 25,252 / 32,814 | 8,173 / 14,807 / 15,814 | 582,292 / 669,789 / 712,415 |
| builder-adversary-stateless | 21,802 / 27,799 / 30,861 | 10,544 / 11,620 / 12,755 | 697,172 / 765,282 / 778,644 |
| builder-adversary-lean | 28,077 / 33,238 / 38,966 | 12,416 / 13,523 / 14,403 | 432,191 / 478,905 / 575,650 |
| builder-solo | 6,029 / 6,611 / 6,969 | 6,542 / 6,715 / 7,020 | 392,494 / 483,506 / 737,117 |
| **all** | 6,029 / 28,077 / 38,966 | 6,542 / 11,712 / 15,814 | 392,494 / 697,172 / 1,044,586 |
| builder-adversary-deferred | 31,451 / 33,827 / 34,537 | 12,170 / 13,105 / 15,343 | 1,277,854 / 1,336,680 / 1,841,155 |
| **all** | 6,029 / 29,024 / 38,966 | 6,542 / 12,170 / 15,814 | 392,494 / 705,228 / 1,841,155 |
## Per-variant medians (commits / LOC / duration)
@@ -34,21 +36,22 @@ tokens/sec excludes runs flagged `LIMIT` (a usage-limit pause inflates duration
| builder-adversary-stateless | 14 | 400 | 900 |
| builder-adversary-lean | 28 | 390 | 960 |
| builder-solo | 5 | 426 | 420 |
| builder-adversary-deferred | 10 | 425 | 1020 |
## Correlations with total tokens (pooled, n=25)
## Correlations with total tokens (pooled, n=28)
| tokens vs | Pearson r |
|---|--:|
| duration | +0.83 |
| commits | +0.79 |
| LOC | -0.04 |
| commits | +0.65 |
| LOC | +0.01 |
## All runs (raw)
| variant | rep | ok | limit | total | dur(s) | commits | LOC | tok/LOC | tok/sec | tok/commit |
|---|:--:|:--:|:--:|--:|--:|--:|--:|--:|--:|--:|
| builder-adversary | 1 | YES | | 11,117,474 | 960 | 14 | 466 | 23,857 | 11,581 | 794,105 |
| builder-adversary | 2 | YES | | 13,579,616 | 1680 | 13 | 449 | 30,244 | 8,083 | 1,044,586 |
| builder-adversary | 2 | YES | LIMIT | 13,579,616 | 1680 | 13 | 449 | 30,244 | 8,083 | 1,044,586 |
| builder-adversary | 3 | YES | | 14,960,414 | 1080 | 16 | 458 | 32,665 | 13,852 | 935,026 |
| builder-adversary | 4 | YES | | 13,037,683 | 1020 | 13 | 429 | 30,391 | 12,782 | 1,002,899 |
| builder-adversary | 5 | YES | | 11,903,098 | 1020 | 15 | 381 | 31,242 | 11,670 | 793,540 |
@@ -74,5 +77,8 @@ tokens/sec excludes runs flagged `LIMIT` (a usage-limit pause inflates duration
| builder-solo | 3 | YES | | 2,837,115 | 420 | 6 | 426 | 6,660 | 6,755 | 472,852 |
| builder-solo | 4 | YES | | 2,773,634 | 420 | 5 | 398 | 6,969 | 6,604 | 554,727 |
| builder-solo | 5 | YES | | 2,948,467 | 420 | 4 | 446 | 6,611 | 7,020 | 737,117 |
| builder-adversary-deferred | 1 | YES | | 13,366,800 | 1020 | 10 | 425 | 31,451 | 13,105 | 1,336,680 |
| builder-adversary-deferred | 2 | YES | | 12,888,082 | 840 | 7 | 381 | 33,827 | 15,343 | 1,841,155 |
| builder-adversary-deferred | 3 | YES | | 15,334,242 | 1260 | 12 | 444 | 34,537 | 12,170 | 1,277,854 |
_Stats over successful runs. `LIMIT` = the run hit a usage-limit pause (duration/tok-sec distorted, token total fine). Repos kept under the run root for analysis._