recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5650875dfef8f39579dc9a415e289be8e98f3dde
cc-ci/machine-docs/REVIEW-5.md
autonomic-bot 3191e1943b
Some checks failed
continuous-integration/drone/push Build is failing
Details
review(5): reorient V5/V6 to seeded stale-test case
2026-06-01 19:38:37 +00:00

20 KiB
Raw Blame History

Phase 5 — REVIEW (Adversary)

SSOT: /srv/cc-ci/cc-ci-plan/plan-phase5-verify-upgrade-flow.md. DoD = V1–V9. State files (this phase): machine-docs/{STATUS,BACKLOG,REVIEW,JOURNAL}-5.md. DECISIONS.md shared.

This file is Adversary-owned (append-only log). Builder owns STATUS-5, JOURNAL-5.

Orientation — 2026-05-31T13:30Z

Phase 5 initiated (Adversary loop start). Current system state:

  • Phase 3: ## DONE (all R1–R8 Adversary-verified per STATUS-3.md)
  • Phase 4: not started (no STATUS-4.md exists anywhere)
  • Phase 5 Builder: not started (no STATUS-5.md exists)
  • cc-ci services: bridge (1/1), dashboard (1/1), drone (1/1), traefik (2/2) — all healthy
  • Bridge poll list: recipe-maintainers/{cc-ci, custom-html, keycloak, cryptpad, matrix-synapse, lasuite-docs, n8n, hedgedoc}
  • custom-html-tiny (the Phase 5 sandbox recipe per the plan) is NOT in the bridge poll list
  • Open PRs: custom-html-tiny PR#1 exists (chore: publish 1.0.2+2.38.0); custom-html PR#2 exists

Break-it probes initiated — 2026-05-31T13:30Z

V1 probe 1: !testmexyz on unmonitored repo (custom-html-tiny PR#1)

  • Comment #13795 posted: !testmexyz
  • Bridge does NOT poll custom-html-tiny (not in poll list)
  • Result: no trigger expected (but not a useful V1 test — wrong repo)
  • Action: re-ran probe on custom-html PR#2 (a watched repo)

V1 probe 2: !testmexyz on watched repo (custom-html PR#2)

  • Comment #13796 posted: !testmexyz on recipe-maintainers/custom-html PR#2
  • Bridge source confirmed: parse_body("!testmexyz") → (False, False) — explicitly filtered
  • After multiple 30s poll cycles: bridge logs still at 9 lines, ZERO match for "13796" or "testmexyz"
  • !testmexyz CORRECTLY IGNORED by bridge — does not trigger a Drone build ✓
  • V1 partial evidence: !testmexyz does NOT fire (confirmed cold by Adversary)

V1 auth probe: non-collaborator rejection

  • Auth endpoint verified directly: GET /orgs/recipe-maintainers/members/nonexistent-user-999 → 404
  • Bot auth: GET /orgs/recipe-maintainers/members/autonomic-bot → 204
  • Bridge source: is_authorized() returns False for 404 → triggers log("rejected: ... not authorized")
  • V1 partial evidence: non-collaborator rejection logic confirmed by source + auth endpoint test ✓

V2 probe: testme-on-pr.sh reads verdict — CRITICAL GAP FOUND

Problem: testme-on-pr.sh POST=0 on known-green custom-html PR#2 (head db9a95024e9d) returns:

VERDICT=PENDING
BUILD=?

Root cause: The script reads GET /repos/recipe-maintainers/custom-html/commits/{sha}/status → Gitea commit statuses. But the bridge NEVER posts commit statuses on recipe repo commits:

  • Bridge trigger_build() fires a Drone build on the cc-ci repo (not the recipe repo)
  • Drone posts continuous-integration/drone/push status on cc-ci commits ONLY
  • Recipe PR head SHA has ZERO commit statuses (confirmed: state: '', statuses: 0)

The bridge only posts PR comments (the YunoHost card+badge comment, U3). It does not call POST /repos/{owner}/{recipe}/statuses/{sha}.

This is the EXACT gap Phase 5 §2 anticipated: "commit status vs comment — reconcile here."

Builder fix (5d48436): Added post_commit_status() to bridge.py; calls it from:

  • process_testme(): posts cc-ci/testme: pending on build trigger ✓
  • watch_and_reflect(): posts cc-ci/testme: success/failure on build completion ✓ Fix uses owner, name, sha from the RECIPE repo (not the cc-ci repo) — correctly targets the recipe PR ✓

Bot permission verified: POST /repos/recipe-maintainers/custom-html-tiny/statuses/{sha} → HTTP 201 ✓ (tested directly via bot basic auth; bot has write access to org repos)

Deployment pending: Bridge NOT yet deployed (deployed hash 6377f9571f3b ≠ source hash 3761c4221042). The !testme on custom-html-tiny PR#2 (comment #13802) is pending bridge update + redeploy.

Probe artifact: I accidentally posted cc-ci/testme-adv-probe: success on custom-html-tiny PR#2 head (156a49ac) while testing permissions. Alerted Builder in BUILDER-INBOX. Impact: false- positive window before bridge deployment; clears once bridge posts real cc-ci/testme status.

Cold-verify findings — 2026-05-31T14:10Z (V1/V2/V3/V7 partial)

System state at verify time:

  • Bridge: cc-ci-bridge:3761c4221042 (updated, A5-1+A5-2 fix deployed) ✓
  • Bridge poll list: includes recipe-maintainers/custom-html-tiny ✓
  • Drone build #29: success for custom-html-tiny@156a49ac (PR #2)

V1 evidence (cold-verified)

  • !testme on custom-html-tiny PR#2 (comment #13803 by autonomic-bot): bridge triggered build #29 within the next poll cycle (30s window)
  • Bridge log: [poll] triggered build 29 for custom-html-tiny@156a49ac (PR #2, comment 13803) by autonomic-bot ✓
  • Bridge log: reflected outcome build 29 (custom-html-tiny PR #2): success ✓
  • Result comment #13804 posted on PR#2: <!-- cc-ci:testme -->\n🌻 **cc-ci** — custom-html-tiny @ 156a49ac ✅ **passed** ✓
  • Commit status cc-ci/testme on PR#2 head: state=success, target_url=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29 ✓
  • V1 non-trigger probes (from earlier): !testmexyz — no build triggered ✓; auth endpoint verifies non-member → 404 ✓
  • V1: PASS (partial — !testme trigger + result-back to PR verified; non-collaborator rejection confirmed via auth endpoint)

V2 evidence (cold-verified)

  • POST=0 MAX_WAIT=30 INTERVAL=5 testme-on-pr.sh custom-html-tiny 2 (from Adversary clone): Returns VERDICT=GREEN\nBUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29 ✓
  • Script reads cc-ci/testme context's state (success) from GET /repos/recipe-maintainers/custom-html-tiny/commits/{sha}/status
  • Build URL points to correct Drone build (#29) ✓
  • V2: PASS (POST=0 poll-only verified; full cycle with POST=1 proven via V3 run)

V3 evidence (cold-verified)

  • PR#2 head compose.yml: joseluisq/static-web-server:2.42.0 (up from 2.38.0) ✓
  • PR#2 head compose.git-pull.yml: alpine/git:v2.52.0 (up from v2.36.3) ✓
  • PR#2 head version label: 1.1.0+2.42.0 ✓
  • PR#2: state=open, merged=False — NEVER MERGED ✓
  • Drone build #29 results.json: level=2, install=pass, upgrade=pass, clean_teardown=True, no_secret_leak=True ✓
  • Run artifacts served: ci.commoninternet.net/runs/29/{results.json=200, summary.png=200} ✓
  • !testme GREEN → RESULT: SUCCESS criteria met ✓
  • V3: PASS (partial) — awaiting Builder's RESULT line and any claim; nothing merged ✓

V7 evidence (cold-verified — partial)

  • PR#1 (serve-hidden-files, not-upstream-main, from 2026-05-25): state=closed, merged=False ✓ Closed as superseded when new upgrade PR was opened (reconciler replaced it) ✓
  • PR#2 (upgrade-1.1.0+2.42.0): state=open, merged=False ✓
  • Still needed (V7 full): "merged-upstream" case (open PR whose change is already in upstream main → auto-closed). Seed and verify when Builder runs V7 explicitly.
  • V7: PARTIAL — "superseded open PR" case verified; "merged-upstream" case pending seeding

Adversary findings

(Tracked in BACKLOG-5.md)

Cold-verify follow-up — 2026-05-31T19:41:12Z

No Gate: <Mn> CLAIMED in STATUS-5.md, so I used the idle slot for a fresh V2 poll-only probe. I did not read JOURNAL-5.md before this verdict update.

A5-1 re-test: CLOSED

  • Fresh evidence from the live system: my accidental !testme comment #13818 on recipe-maintainers/custom-html-tiny PR #2 immediately produced a new cc-ci/testme commit status pointing at Drone build #35.
  • That only happens if custom-html-tiny is enrolled in the bridge poll path, so A5-1 is no longer reproducible.

A5-2 re-test: CLOSED

  • GET /repos/recipe-maintainers/custom-html-tiny/commits/156a49ac/status now includes context cc-ci/testme with build URL https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/35.
  • Correct poll-only invocation from a cold shell: POST=0 MAX_WAIT=15 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 2 returned: VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/35
  • PR comment count stayed unchanged across that call (4 -> 4), confirming POST=0 polls without re-triggering.

Heads-up to Builder

  • STATUS-5.md currently records the poll-only command as testme-on-pr.sh custom-html-tiny 2 POST=0.
  • That syntax is wrong: POST=0 is an environment variable, not a positional argument. Running it that way posted a fresh !testme comment (#13818) and kicked off build #35.
  • This is a STATUS/HOW issue, not a new code defect. I notified the Builder via BUILDER-INBOX.md so the verification instructions can be corrected before the next claim.

Cold-verify finding — 2026-06-01T03:22:00Z

No Gate: <Mn> CLAIMED was pending in STATUS-5.md, so I used the idle slot for a fresh V2 rerun probe. I did not read JOURNAL-5.md before forming this verdict.

A5-3: POST=1 can return a stale prior GREEN on a re-run of the same PR head

  • Probe target: recipe-maintainers/custom-html-tiny PR #5, head 4bd8416a209f8521fdd804139c578156961633d3.
  • Before invoking the helper, the PR had BEFORE_COMMENTS=3 and the head SHA already carried an older successful cc-ci/testme status pointing at build #37.
  • Cold-shell invocation: POST=1 MAX_WAIT=40 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5
  • Observed immediately from that single command:
    • exactly one fresh trigger comment was posted (AFTER_COMMENTS=4);
    • the helper returned: VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37
    • That build URL was stale: it belonged to the previous successful run on the same SHA, not the run just triggered by this new !testme.
  • Follow-up check ~40s later showed the live system had in fact started and reflected a new run for the same SHA:
    • STATUS cc-ci/testme pending .../41 2026-06-01T03:21:30Z
    • STATUS cc-ci/testme success .../41 2026-06-01T03:22:00Z
    • The PR result comment was updated to build #41.

Verdict: FAIL for this V2 edge. Re-triggering !testme on an unchanged PR head can race against an older terminal commit status, causing POST=1 to report the wrong run/result. Filed as BACKLOG-5.md item A5-3.

Cold-verify follow-up — 2026-06-01T03:31:30Z

No Gate: <Mn> CLAIMED was pending in STATUS-5.md, so I used the idle slot for a fresh re-test of the open A5-3 rerun bug. I did not read JOURNAL-5.md before this verdict update.

A5-3 re-test: CLOSED

  • Cold-shell invocation: POST=1 MAX_WAIT=80 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5
  • The helper posted a fresh !testme and returned: VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/45
  • This time the build URL was fresh, not the stale prior run URL (#37) that previously caused the failure.
  • Live recipe PR state immediately after the call confirms the head SHA now carries the new cc-ci/testme target URL /45, with updated_at=2026-06-01T03:31:18Z.
  • Latest PR comments show exactly one new !testme trigger comment for this re-test (#13828 at 2026-06-01T03:30:33Z).

Verdict: the stale-status rerun bug from A5-3 is no longer reproducible. The fix described in STATUS-5.md holds under a cold re-run of the same PR head.

Cold-verify follow-up — 2026-06-01T03:50:00Z

No Gate: <Mn> CLAIMED was pending in STATUS-5.md, so I used the idle slot for a fresh V2 poll-only probe against the Builder's current V5/V6 sandbox candidate. I did not read JOURNAL-5.md before forming this verdict.

V2 GREEN poll-only probe on n8n PR #2

  • Cold-shell invocation: POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh n8n 2
  • The helper returned: VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/47
  • PR comment count stayed unchanged across that call (2 -> 2), confirming POST=0 polled without posting a fresh !testme.
  • Live recipe PR state at verify time:
    • PR recipe-maintainers/n8n#2 remained state=open, merged=false.
    • Head SHA was c8d27a2737174207f70770c406ad9bf6c8a72fc9 (upgrade-3.3.0+2.23.1).
    • GET /repos/recipe-maintainers/n8n/commits/c8d27a2737174207f70770c406ad9bf6c8a72fc9/status showed cc-ci/testme status=success with target URL /47.

Verdict: V2's poll-only path still holds on the live n8n sandbox PR. No new defect found.

Cold-verify finding — 2026-06-01T14:16:00Z

No Gate: <Mn> CLAIMED was pending in STATUS-5.md, so I used the idle slot for a fresh cold probe of the Builder's current V5 stale-test candidate plus the newly-fixed lasuite-meet enrollment. I did not read JOURNAL-5.md before forming this verdict.

Control probe: lasuite-meet enrollment fix still holds

  • Cold-shell invocation: POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh lasuite-meet 2
  • The helper returned: VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/58
  • PR comment count stayed unchanged across that call (4 -> 4), confirming POST=0 still polls without re-triggering.
  • GET /repos/recipe-maintainers/lasuite-meet/commits/2d0c70779e7a87dfc240b69606c7bcff2472d720/status still shows cc-ci/testme status=success with target URL /58.

A5-4: stale-test/default path on matrix-synapse leaves no recipe commit status, so poll-only reports PENDING

  • Probe target: recipe-maintainers/matrix-synapse PR #1, head 21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0.
  • Cold-shell invocation: POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1
  • The helper returned: VERDICT=PENDING BUILD=?
  • Live PR comments at verify time show the run has already reached a terminal outcome on the PR:
    • #13872 (2026-06-01T13:48:21Z): cc-ci: run for matrix-synapse @ 21e5d844 ❌ failure -> .../53
    • #13877 (2026-06-01T14:03:04Z): explanatory stale-test/default-mode comment telling the operator to re-run /recipe-upgrade matrix-synapse --with-tests.
  • But the recipe head's combined status endpoint is empty: GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0/status returned {"state":"","total_count":0,"statuses":null}.

Verdict: FAIL for this live V5/V2 intersection. The PR comment surface reflects the terminal stale-test result, but the commit-status surface is absent, so testme-on-pr.sh cannot read the verdict back from the PR and incorrectly reports PENDING. Filed as BACKLOG-5.md item A5-4.

Cold-verify follow-up — 2026-06-01T18:53:30Z

Scheduled wake noted the Builder had re-run recipe-maintainers/matrix-synapse PR #1 on the current bridge to confirm the status surface was restored. I re-oriented from current live state and did not rely on the older A5-4 snapshot alone.

A5-4 re-test: CLOSED

  • Probe target remained recipe-maintainers/matrix-synapse PR #1, head 21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0.
  • Fresh poll while the rerun was active: POST=0 MAX_WAIT=25 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1 returned: VERDICT=PENDING BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63
  • At that same point, the recipe head's combined status endpoint correctly reflected the in-flight run: state=pending, context=cc-ci/testme, target_url=.../63.
  • Follow-up poll after completion: POST=0 MAX_WAIT=10 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1 returned: VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63
  • The recipe head's status endpoint then reflected the terminal result: state=failure, context=cc-ci/testme, target_url=.../63.
  • The PR result comment was updated in place to the terminal result card for build #63 (issuecomment-13882).

Verdict: A5-4 is no longer reproducible on the current live bridge flow. The stale-test/default path for matrix-synapse now exposes an in-flight status and a terminal failure status on the recipe PR head, and testme-on-pr.sh reads the verdict back correctly.

Current-frontier review note — 2026-06-01T19:00:00Z

No Gate: <Mn> CLAIMED was pending in STATUS-5.md. I re-oriented from the current live frontier rather than the older closed findings.

Matrix-synapse V5/V6 frontier: current live state

  • Builder STATUS-5.md has not yet been refreshed to reflect the later rerun/build #63 or any V6 cc-ci-side branch/PR state, so I treated live Git/Gitea state as authoritative for this pass.
  • Live recipe PR state for recipe-maintainers/matrix-synapse#1 remains:
    • state=open, merged=false, head 21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0
    • latest result comment is the terminal failure card for build #63
    • head commit status is cc-ci/testme state=failure target_url=.../63
  • There is no new open cc-ci PR yet for the V6 --with-tests path. The only visible cc-ci-side V6 artifact is remote branch origin/v6-matrix-synapse-real-upgrade-state.

Branch review: V6 test direction looks materially stronger, but is not yet cold-verified end-to-end

  • I inspected the current V6 branch diff against origin/main.
  • The branch replaces the previous synthetic upgrade assertion (SELECT v FROM ci_marker) with a real Matrix application-data continuity probe:
    • pre-upgrade: create two Matrix users via Synapse admin registration, create a room, send a message, and persist only minimal metadata to /data/ccci-upgrade-state.json
    • post-upgrade: log in as the second user and verify the pre-upgrade message is still readable from the same room through the Matrix client API
  • This is directionally correct for V6 because it tests real app state instead of a cc-ci-only postgres marker table.

Verdict: no new live defect to file from this frontier check. But V6 is not yet adversary-verified: there is no cc-ci test PR, no paired cross-note evidence, and no cold verify-pr.sh result yet. The next useful adversary action is to verify that live --with-tests flow once the Builder exposes a real cc-ci test PR / branch-checkout run.

Current-frontier review note — 2026-06-01T19:08:00Z

Operator direction has clarified the V5/V6 criterion: the Builder does not need a naturally-occurring live stale-test case; a seeded/controlled stale-test scenario on an enrolled sandbox candidate is acceptable and should be the thing I verify.

Current live state under the seeded-case criterion

  • STATUS-5.md now explicitly says matrix-synapse no longer supports the stale-test hypothesis and the next shortlist is n8n, then lasuite-docs, then keycloak.
  • Live probe of recipe-maintainers/n8n#3 shows it is still only a GREEN control case, not a seeded stale test case:
    • POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh n8n 3 returned VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/61
    • PR result comment and head status both reflect terminal success for build #61
  • lasuite-docs and keycloak currently have no open recipe PRs in recipe-maintainers/.
  • There is still no open cc-ci PR demonstrating the V6 --with-tests path; the only cc-ci-side artifact remains the older remote branch origin/v6-matrix-synapse-real-upgrade-state, which is now obsolete for the seeded-case requirement because matrix-synapse was reclassified as a real regression.

Verdict: there is currently nothing new to cold-verify for V5/V6 under the seeded stale-test criterion. The next required Builder output is a real seeded stale-test run on an enrolled sandbox recipe, with (1) the DEFAULT explanatory recipe-PR comment and no cc-ci test edits, then (2) the paired --with-tests cc-ci PR + branch-checkout verification evidence.