review(5): flag matrix-synapse stale-test status gap
This commit is contained in:
@@ -25,6 +25,32 @@ Single-writer: `## Build backlog` = Builder-only; `## Adversary findings` = Adve
|
|||||||
|
|
||||||
## Adversary findings
|
## Adversary findings
|
||||||
|
|
||||||
|
### [adversary] A5-4 — `matrix-synapse` stale-test/default path leaves no recipe commit status
|
||||||
|
**Status:** OPEN — found 2026-06-01T14:16:00Z; see `REVIEW-5.md`.
|
||||||
|
|
||||||
|
On the live V5 stale-test candidate `recipe-maintainers/matrix-synapse` PR `#1`, the PR comments show a
|
||||||
|
terminal failed `!testme` result for build `#53` plus the default-mode explanatory stale-test comment,
|
||||||
|
but the recipe PR head has **no** `cc-ci/testme` commit status at all. As a result, the helper cannot
|
||||||
|
read the verdict back from the PR and poll-only returns `PENDING` even though the PR already shows the
|
||||||
|
terminal outcome.
|
||||||
|
|
||||||
|
**Cold repro:**
|
||||||
|
1. Use `recipe-maintainers/matrix-synapse` PR `#1`, head
|
||||||
|
`21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0`.
|
||||||
|
2. Confirm PR comments include:
|
||||||
|
- failure result comment for build `#53` (`#13872`), and
|
||||||
|
- explanatory stale-test comment (`#13877`).
|
||||||
|
3. Run:
|
||||||
|
`POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
|
||||||
|
4. Observe:
|
||||||
|
- helper returns `VERDICT=PENDING` and `BUILD=?`;
|
||||||
|
- `GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0/status`
|
||||||
|
returns `{"state":"","total_count":0,"statuses":null}`.
|
||||||
|
|
||||||
|
**Impact:** this breaks the Phase-5 requirement that the upgrade tooling read the verdict back from the
|
||||||
|
PR on the live stale-test/default path. The comment surface says the run is terminal; the status surface
|
||||||
|
still says nothing.
|
||||||
|
|
||||||
### [adversary] A5-3 — `POST=1 testme-on-pr.sh` can return a stale prior GREEN on re-runs
|
### [adversary] A5-3 — `POST=1 testme-on-pr.sh` can return a stale prior GREEN on re-runs
|
||||||
**Status:** CLOSED — re-tested 2026-06-01T03:31:30Z; see `REVIEW-5.md` follow-up entry.
|
**Status:** CLOSED — re-tested 2026-06-01T03:31:30Z; see `REVIEW-5.md` follow-up entry.
|
||||||
|
|
||||||
|
|||||||
@@ -227,3 +227,43 @@ poll-only probe against the Builder's current V5/V6 sandbox candidate. I did **n
|
|||||||
showed `cc-ci/testme status=success` with target URL `/47`.
|
showed `cc-ci/testme status=success` with target URL `/47`.
|
||||||
|
|
||||||
**Verdict:** V2's poll-only path still holds on the live `n8n` sandbox PR. No new defect found.
|
**Verdict:** V2's poll-only path still holds on the live `n8n` sandbox PR. No new defect found.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cold-verify finding — 2026-06-01T14:16:00Z
|
||||||
|
|
||||||
|
No `Gate: <Mn> CLAIMED` was pending in `STATUS-5.md`, so I used the idle slot for a fresh cold probe of
|
||||||
|
the Builder's current V5 stale-test candidate plus the newly-fixed `lasuite-meet` enrollment. I did
|
||||||
|
**not** read `JOURNAL-5.md` before forming this verdict.
|
||||||
|
|
||||||
|
### Control probe: `lasuite-meet` enrollment fix still holds
|
||||||
|
- Cold-shell invocation:
|
||||||
|
`POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh lasuite-meet 2`
|
||||||
|
- The helper returned:
|
||||||
|
`VERDICT=GREEN`
|
||||||
|
`BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/58`
|
||||||
|
- PR comment count stayed unchanged across that call (`4 -> 4`), confirming `POST=0` still polls without
|
||||||
|
re-triggering.
|
||||||
|
- `GET /repos/recipe-maintainers/lasuite-meet/commits/2d0c70779e7a87dfc240b69606c7bcff2472d720/status`
|
||||||
|
still shows `cc-ci/testme status=success` with target URL `/58`.
|
||||||
|
|
||||||
|
### A5-4: stale-test/default path on `matrix-synapse` leaves no recipe commit status, so poll-only reports `PENDING`
|
||||||
|
- Probe target: `recipe-maintainers/matrix-synapse` PR `#1`, head
|
||||||
|
`21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0`.
|
||||||
|
- Cold-shell invocation:
|
||||||
|
`POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
|
||||||
|
- The helper returned:
|
||||||
|
`VERDICT=PENDING`
|
||||||
|
`BUILD=?`
|
||||||
|
- Live PR comments at verify time show the run has already reached a terminal outcome on the PR:
|
||||||
|
- `#13872` (`2026-06-01T13:48:21Z`):
|
||||||
|
`cc-ci: run for matrix-synapse @ 21e5d844 ❌ failure -> .../53`
|
||||||
|
- `#13877` (`2026-06-01T14:03:04Z`): explanatory stale-test/default-mode comment telling the operator
|
||||||
|
to re-run `/recipe-upgrade matrix-synapse --with-tests`.
|
||||||
|
- But the recipe head's combined status endpoint is empty:
|
||||||
|
`GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0/status`
|
||||||
|
returned `{"state":"","total_count":0,"statuses":null}`.
|
||||||
|
|
||||||
|
**Verdict:** FAIL for this live V5/V2 intersection. The PR comment surface reflects the terminal
|
||||||
|
stale-test result, but the commit-status surface is absent, so `testme-on-pr.sh` cannot read the verdict
|
||||||
|
back from the PR and incorrectly reports `PENDING`. Filed as `BACKLOG-5.md` item **A5-4**.
|
||||||
|
|||||||
Reference in New Issue
Block a user