From 143f83a710a48ba6cbd6f053705a53836dade474 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Mon, 1 Jun 2026 03:23:27 +0000 Subject: [PATCH] review(5 V2): flag stale rerun verdict race FAIL --- machine-docs/BACKLOG-5.md | 28 ++++++++++++++++++++++++++++ machine-docs/BUILDER-INBOX.md | 15 +++++++++++++++ machine-docs/REVIEW-5.md | 31 +++++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+) create mode 100644 machine-docs/BUILDER-INBOX.md diff --git a/machine-docs/BACKLOG-5.md b/machine-docs/BACKLOG-5.md index b3d025f..41b3c35 100644 --- a/machine-docs/BACKLOG-5.md +++ b/machine-docs/BACKLOG-5.md @@ -24,6 +24,34 @@ Single-writer: `## Build backlog` = Builder-only; `## Adversary findings` = Adve ## Adversary findings +### [adversary] A5-3 — `POST=1 testme-on-pr.sh` can return a stale prior GREEN on re-runs +**Status:** OPEN — found 2026-06-01T03:22:00Z. + +The helper currently posts a fresh `!testme`, then polls the recipe PR head's combined commit status. +If that PR head SHA already has a previous successful `cc-ci/testme` status and the bridge has not yet +processed the new comment, the helper exits immediately with the **old** GREEN/build URL instead of a +fresh `PENDING` or the new run's URL. + +This is a real Phase-5/V2 correctness bug because re-commenting `!testme` on the same PR head is a +supported path, and the helper is meant to report the verdict for the run it just triggered. + +**Cold repro:** +1. Use an open PR whose current head SHA already has `cc-ci/testme: success` from an earlier run. +2. Record the PR comment count. +3. Run: + `POST=1 MAX_WAIT=40 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5` +4. Observe: + - the PR comment count increases by exactly one (`3 -> 4` in the reproducer), so one fresh `!testme` + was posted; + - the helper returns `VERDICT=GREEN` with the **old** build URL + `https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37`; + - later, the live system shows a new run was actually triggered and reflected on the PR as build + `#41` (`cc-ci/testme pending -> success`, target URL `/41`). + +**Likely fix direction:** after `POST=1`, do not trust a pre-existing terminal status on the same SHA. +Poll for evidence that belongs to the newly-triggered run (e.g. a newer status timestamp, a pending +status after the new comment, or a changed build URL/context generation marker) before returning. + ### [adversary] A5-2 — CRITICAL: testme-on-pr.sh cannot read verdicts (commit status vs comment mismatch) **Status:** CLOSED — re-tested 2026-05-31T19:41:12Z; see `REVIEW-5.md` follow-up entry. diff --git a/machine-docs/BUILDER-INBOX.md b/machine-docs/BUILDER-INBOX.md new file mode 100644 index 0000000..23da156 --- /dev/null +++ b/machine-docs/BUILDER-INBOX.md @@ -0,0 +1,15 @@ +## 2026-06-01T03:22:00Z — A5-3 heads-up + +Fresh adversary probe on `custom-html-tiny` PR `#5` found a V2 rerun bug and I recorded it formally in +`REVIEW-5.md` / `BACKLOG-5.md` as **A5-3**. + +Summary: +- `POST=1 MAX_WAIT=40 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5` + posted exactly one new `!testme` comment. +- The helper then returned the stale old GREEN/build URL for build `#37` immediately. +- About 40s later, the live PR/statuses showed the newly-triggered run was actually build `#41` + (`pending -> success`, PR comment updated to `/41`). + +Interpretation: on a rerun against the same PR head SHA, the helper can read a pre-existing terminal +`cc-ci/testme` status before the bridge posts the new pending status, so `POST=1` does not reliably +report the run it just triggered. diff --git a/machine-docs/REVIEW-5.md b/machine-docs/REVIEW-5.md index b77d72f..3516410 100644 --- a/machine-docs/REVIEW-5.md +++ b/machine-docs/REVIEW-5.md @@ -149,3 +149,34 @@ I did **not** read `JOURNAL-5.md` before this verdict update. it that way posted a fresh `!testme` comment (`#13818`) and kicked off build `#35`. - This is a STATUS/HOW issue, not a new code defect. I notified the Builder via `BUILDER-INBOX.md` so the verification instructions can be corrected before the next claim. + +--- + +## Cold-verify finding — 2026-06-01T03:22:00Z + +No `Gate: CLAIMED` was pending in `STATUS-5.md`, so I used the idle slot for a fresh V2 rerun +probe. I did **not** read `JOURNAL-5.md` before forming this verdict. + +### A5-3: `POST=1` can return a stale prior GREEN on a re-run of the same PR head +- Probe target: `recipe-maintainers/custom-html-tiny` PR `#5`, head + `4bd8416a209f8521fdd804139c578156961633d3`. +- Before invoking the helper, the PR had `BEFORE_COMMENTS=3` and the head SHA already carried an older + successful `cc-ci/testme` status pointing at build `#37`. +- Cold-shell invocation: + `POST=1 MAX_WAIT=40 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5` +- Observed immediately from that single command: + - exactly one fresh trigger comment was posted (`AFTER_COMMENTS=4`); + - the helper returned: + `VERDICT=GREEN` + `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37` + - That build URL was stale: it belonged to the previous successful run on the same SHA, not the run + just triggered by this new `!testme`. +- Follow-up check ~40s later showed the live system had in fact started and reflected a new run for the + same SHA: + - `STATUS cc-ci/testme pending .../41 2026-06-01T03:21:30Z` + - `STATUS cc-ci/testme success .../41 2026-06-01T03:22:00Z` + - The PR result comment was updated to build `#41`. + +**Verdict:** FAIL for this V2 edge. Re-triggering `!testme` on an unchanged PR head can race against an +older terminal commit status, causing `POST=1` to report the wrong run/result. Filed as +`BACKLOG-5.md` item **A5-3**.