fix(tests): track custom-html /var docroot

machine-docs/BACKLOG-5.md

@@ -26,7 +26,7 @@ Single-writer: `## Build backlog` = Builder-only; `## Adversary findings` = Adve
 ## Adversary findings
 
 ### [adversary] A5-4 — `matrix-synapse` stale-test/default path leaves no recipe commit status
-**Status:** OPEN — found 2026-06-01T14:16:00Z; see `REVIEW-5.md`.
+**Status:** CLOSED — re-tested 2026-06-01T18:53:30Z; see `REVIEW-5.md` follow-up entry.
 
 On the live V5 stale-test candidate `recipe-maintainers/matrix-synapse` PR `#1`, the PR comments show a
 terminal failed `!testme` result for build `#53` plus the default-mode explanatory stale-test comment,
@@ -51,6 +51,10 @@ terminal outcome.
 PR on the live stale-test/default path. The comment surface says the run is terminal; the status surface
 still says nothing.
 
+**Re-test result:** no longer reproducible on rerun build `#63`. The recipe PR head now shows
+`cc-ci/testme` `pending -> failure` with target URL `.../63`, and poll-only returns
+`VERDICT=PENDING BUILD=.../63` while in flight, then `VERDICT=RED BUILD=.../63` after completion.
+
 ### [adversary] A5-3 — `POST=1 testme-on-pr.sh` can return a stale prior GREEN on re-runs
 **Status:** CLOSED — re-tested 2026-06-01T03:31:30Z; see `REVIEW-5.md` follow-up entry.
 

machine-docs/JOURNAL-5.md

@@ -361,3 +361,63 @@ Default-mode Phase-5 action taken:
   `/recipe-upgrade matrix-synapse --with-tests` to get a verified cc-ci test PR.
 
 Next: treat `matrix-synapse` as the V6 candidate and prepare the dedicated cc-ci test-branch fix.
+
+## 2026-06-01 — A5-4 cleared; matrix-synapse V6 branch invalidated
+
+Adversary finding A5-4 was real and caused by timing around the temporary old bridge image during the
+host-recovery rollout, not by the current live bridge behavior.
+
+Live re-test on the current bridge:
+- `POST=1 MAX_WAIT=90 INTERVAL=5 /srv/cc-ci-orch/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
+  -> `VERDICT=PENDING`
+  -> `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`
+- `POST=0 MAX_WAIT=360 INTERVAL=10 /srv/cc-ci-orch/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
+  -> `VERDICT=RED`
+  -> `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`
+- `GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0/status`
+  now shows context `cc-ci/testme state=failure target_url=.../63`.
+
+Conclusion for A5-4:
+- cleared on current live behavior; the helper can again read the verdict back from the PR via commit
+  status on this stale-test/default-path candidate.
+
+V6 branch-checkout work on matrix-synapse:
+- Created dedicated clone `/tmp/opencode/cc-ci-v6`, branch
+  `v6-matrix-synapse-real-upgrade-state`.
+- Implemented a real app-data upgrade assertion there:
+  - `tests/matrix-synapse/ops.py` now seeds two Matrix users, a room, and a message before upgrade and
+    persists only `{user_b,password,room_id,marker}` to `/data/ccci-upgrade-state.json`.
+  - `tests/matrix-synapse/test_upgrade.py` now logs back in after upgrade and asserts the pre-upgrade
+    message is still readable from the same room.
+- Branch commit: `5edcf8d fix(tests): use real matrix data for upgrade state`
+- Pushed remote branch: `origin/v6-matrix-synapse-real-upgrade-state`
+
+While verifying that branch I found and fixed a helper bug in the V6 path itself:
+- `ci-test-review/verify-pr.sh` previously passed a branch name like
+  `upgrade-7.2.0+v1.153.0` straight through as `REF`, but the generic upgrade assertion expects the PR
+  head COMMIT SHA there (same shape `!testme` uses). That made branch-checkout verification falsely RED
+  at HC1 with `head_ref='upgrade-7.2...'` vs `chaos-version='21e5d844'`.
+- Patched `verify-pr.sh` to resolve non-SHA refs to their branch head commit via the Gitea API before
+  invoking `runner/run_recipe_ci.py`.
+
+Dedicated host checkout for verification:
+- materialized `/root/cc-ci-v6-verify` on `cc-ci` from the dedicated branch clone
+- marked it safe for git on the host:
+  - `git config --global --add safe.directory /root/cc-ci-v6-verify`
+
+Verification results:
+- First branch-verify run (before the helper fix) hit the HC1 false-red and also showed the new overlay
+  login failure.
+- Second branch-verify run (after the helper fix):
+  - `REMOTE_ROOT=/root/cc-ci-v6-verify RECIPE=matrix-synapse REF=upgrade-7.2.0+v1.153.0 /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh`
+  - helper now resolves `REF_SHA=21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0`
+  - generic upgrade tier PASSed
+  - but the new real-data overlay still FAILED:
+    `login upgradeb53398657 HTTP 403: {'errcode': 'M_FORBIDDEN', 'error': 'Invalid username or password'}`
+
+Conclusion:
+- `matrix-synapse` is NOT a V6 stale-test branch after all.
+- Once the synthetic marker was replaced with a real Matrix data-survival assertion, the upgrade still
+  failed. This points to a true recipe upgrade regression, not a stale cc-ci test.
+
+Next: move to the next enrolled V5/V6 candidate (`n8n`, then `lasuite-docs`, then `keycloak`).

machine-docs/REVIEW-5.md

@@ -267,3 +267,94 @@ the Builder's current V5 stale-test candidate plus the newly-fixed `lasuite-meet
 **Verdict:** FAIL for this live V5/V2 intersection. The PR comment surface reflects the terminal
 stale-test result, but the commit-status surface is absent, so `testme-on-pr.sh` cannot read the verdict
 back from the PR and incorrectly reports `PENDING`. Filed as `BACKLOG-5.md` item **A5-4**.
+
+---
+
+## Cold-verify follow-up — 2026-06-01T18:53:30Z
+
+Scheduled wake noted the Builder had re-run `recipe-maintainers/matrix-synapse` PR `#1` on the current
+bridge to confirm the status surface was restored. I re-oriented from current live state and did **not**
+rely on the older A5-4 snapshot alone.
+
+### A5-4 re-test: CLOSED
+- Probe target remained `recipe-maintainers/matrix-synapse` PR `#1`, head
+  `21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0`.
+- Fresh poll while the rerun was active:
+  `POST=0 MAX_WAIT=25 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
+  returned:
+  `VERDICT=PENDING`
+  `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`
+- At that same point, the recipe head's combined status endpoint correctly reflected the in-flight run:
+  `state=pending`, `context=cc-ci/testme`, `target_url=.../63`.
+- Follow-up poll after completion:
+  `POST=0 MAX_WAIT=10 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh matrix-synapse 1`
+  returned:
+  `VERDICT=RED`
+  `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`
+- The recipe head's status endpoint then reflected the terminal result:
+  `state=failure`, `context=cc-ci/testme`, `target_url=.../63`.
+- The PR result comment was updated in place to the terminal result card for build `#63`
+  (`issuecomment-13882`).
+
+**Verdict:** A5-4 is no longer reproducible on the current live bridge flow. The stale-test/default path
+for `matrix-synapse` now exposes an in-flight status and a terminal failure status on the recipe PR head,
+and `testme-on-pr.sh` reads the verdict back correctly.
+
+---
+
+## Current-frontier review note — 2026-06-01T19:00:00Z
+
+No `Gate: <Mn> CLAIMED` was pending in `STATUS-5.md`. I re-oriented from the current live frontier rather
+than the older closed findings.
+
+### Matrix-synapse V5/V6 frontier: current live state
+- Builder `STATUS-5.md` has **not** yet been refreshed to reflect the later rerun/build `#63` or any V6
+  cc-ci-side branch/PR state, so I treated live Git/Gitea state as authoritative for this pass.
+- Live recipe PR state for `recipe-maintainers/matrix-synapse#1` remains:
+  - `state=open`, `merged=false`, head `21e5d84430bdc52f8fa8aa9a40fa5bda8adf06c0`
+  - latest result comment is the terminal failure card for build `#63`
+  - head commit status is `cc-ci/testme state=failure target_url=.../63`
+- There is **no** new open cc-ci PR yet for the V6 `--with-tests` path. The only visible cc-ci-side V6
+  artifact is remote branch `origin/v6-matrix-synapse-real-upgrade-state`.
+
+### Branch review: V6 test direction looks materially stronger, but is not yet cold-verified end-to-end
+- I inspected the current V6 branch diff against `origin/main`.
+- The branch replaces the previous synthetic upgrade assertion (`SELECT v FROM ci_marker`) with a real
+  Matrix application-data continuity probe:
+  - pre-upgrade: create two Matrix users via Synapse admin registration, create a room, send a message,
+    and persist only minimal metadata to `/data/ccci-upgrade-state.json`
+  - post-upgrade: log in as the second user and verify the pre-upgrade message is still readable from the
+    same room through the Matrix client API
+- This is directionally correct for V6 because it tests real app state instead of a cc-ci-only postgres
+  marker table.
+
+**Verdict:** no new live defect to file from this frontier check. But V6 is **not yet adversary-verified**:
+there is no cc-ci test PR, no paired cross-note evidence, and no cold `verify-pr.sh` result yet. The next
+useful adversary action is to verify that live `--with-tests` flow once the Builder exposes a real cc-ci
+test PR / branch-checkout run.
+
+---
+
+## Current-frontier review note — 2026-06-01T19:08:00Z
+
+Operator direction has clarified the V5/V6 criterion: the Builder does **not** need a naturally-occurring
+live stale-test case; a **seeded/controlled** stale-test scenario on an enrolled sandbox candidate is
+acceptable and should be the thing I verify.
+
+### Current live state under the seeded-case criterion
+- `STATUS-5.md` now explicitly says `matrix-synapse` no longer supports the stale-test hypothesis and the
+  next shortlist is `n8n`, then `lasuite-docs`, then `keycloak`.
+- Live probe of `recipe-maintainers/n8n#3` shows it is still only a GREEN control case, not a seeded stale
+  test case:
+  - `POST=0 MAX_WAIT=20 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh n8n 3`
+    returned `VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/61`
+  - PR result comment and head status both reflect terminal success for build `#61`
+- `lasuite-docs` and `keycloak` currently have no open recipe PRs in `recipe-maintainers/`.
+- There is still no open cc-ci PR demonstrating the V6 `--with-tests` path; the only cc-ci-side artifact
+  remains the older remote branch `origin/v6-matrix-synapse-real-upgrade-state`, which is now obsolete for
+  the seeded-case requirement because `matrix-synapse` was reclassified as a real regression.
+
+**Verdict:** there is currently **nothing new to cold-verify for V5/V6** under the seeded stale-test
+criterion. The next required Builder output is a real seeded stale-test run on an enrolled sandbox recipe,
+with (1) the DEFAULT explanatory recipe-PR comment and no cc-ci test edits, then (2) the paired
+`--with-tests` cc-ci PR + branch-checkout verification evidence.

machine-docs/STATUS-5.md

@@ -74,8 +74,8 @@ preferred, `/root/cc-ci` fallback) instead of hard-coding `/root/cc-ci`.
 | V2 — testme-on-pr.sh reads verdict | DONE | GREEN ✓ (build #29/#35); RED ✓ (build #34); rerun fix ✓ (build #43) |
 | V3 — /recipe-upgrade sandbox GREEN | DONE | custom-html-tiny PR#2; build #29 SUCCESS |
 | V4 — 3-iter regression loop | DONE | custom-html-tiny PR#5; build #34 RED, build #37 GREEN |
-| V5 — stale-test DEFAULT = comment | IN PROGRESS | matrix-synapse PR#1 comment posted; no test edits; awaiting full evidence bundle |
-| V6 — --with-tests opens+verifies cc-ci test PR | TODO | |
+| V5 — stale-test DEFAULT = comment | IN PROGRESS | matrix-synapse default-mode comment posted, but later invalidated as a likely real regression; next candidate pending |
+| V6 — --with-tests opens+verifies cc-ci test PR | TODO | matrix-synapse branch invalidated by real regression; next candidate pending |
 | V7 — mirror reconciliation | DONE | PR#1 superseded, PR#4 merged-upstream, main=upstream ✓ |
 | V8 — /upgrade-all DEFAULT run | TODO | |
 | V8a — cc-ci-upgrader agent | TODO | |
@@ -123,12 +123,22 @@ preferred, `/root/cc-ci` fallback) instead of hard-coding `/root/cc-ci`.
   Default-mode explanatory PR comment posted with no test edit:
   `https://git.autonomic.zone/recipe-maintainers/matrix-synapse/pulls/1#issuecomment-13877`
   telling the operator to re-run `/recipe-upgrade matrix-synapse --with-tests` for a test-update PR.
+- Adversary finding A5-4 is now cleared on current live behavior: re-`!testme` on the same PR head
+  produced build `#63`; `POST=0 ... testme-on-pr.sh matrix-synapse 1` returned
+  `VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`; and
+  `GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d844.../status` now shows
+  `cc-ci/testme state=failure target_url=.../63`.
+- V6 branch verification on `matrix-synapse` no longer supports the stale-test hypothesis. In a
+  dedicated cc-ci branch checkout with a real Matrix data-survival upgrade assertion, the helper path
+  now resolves the recipe branch to its head SHA correctly, generic upgrade PASSes, but the upgraded
+  app still fails the real post-upgrade assertion: the pre-upgrade Matrix user cannot log in after the
+  upgrade (`HTTP 403 Invalid username or password`). That points to a true recipe upgrade regression,
+  not a stale test.
 
 ## Verification next step
 
-- Advance this `matrix-synapse` stale-test candidate into V6 by preparing a dedicated cc-ci test branch,
-  updating the synthetic upgrade assertion to a real app-data-preservation check, and verifying the
-  recipe upgrade with that test change applied.
+- Move to the next enrolled candidate for V5/V6. Current shortlist: `n8n` first, then `lasuite-docs`,
+  then `keycloak`.
 
 ## Phase 5 gates
 

tests/custom-html/functional/test_content_roundtrip.py

@@ -23,11 +23,11 @@ def test_content_roundtrip(live_app):
     exact bytes round-trip. Non-vacuous: a stale page or misrouted backend would not return our
     randomly-generated content."""
     marker = f"ccci-roundtrip-{uuid.uuid4().hex}"
-    # written into the served volume; nginx routes /<filename> to /usr/share/nginx/html/<filename>
+    # written into the served volume; nginx routes /<filename> to /var/www/html/<filename>
     filename = f"ccci-roundtrip-{uuid.uuid4().hex[:12]}.txt"
     lifecycle.exec_in_app(
         live_app,
-        ["sh", "-c", f"printf %s {marker} > /usr/share/nginx/html/{filename}"],
+        ["sh", "-c", f"mkdir -p /var/www/html && printf %s {marker} > /var/www/html/{filename}"],
     )
 
     url = f"https://{live_app}/{filename}"

tests/custom-html/functional/test_content_type_header.py

@@ -40,7 +40,7 @@ def test_content_type_html_and_txt(live_app):
     body = "hello"
     for name in (html_name, txt_name):
         lifecycle.exec_in_app(
-            live_app, ["sh", "-c", f"printf %s {body} > /usr/share/nginx/html/{name}"]
+            live_app, ["sh", "-c", f"mkdir -p /var/www/html && printf %s {body} > /var/www/html/{name}"]
         )
 
     s_html, h_html = _head(f"https://{live_app}/{html_name}")

tests/custom-html/ops.py

@@ -1,7 +1,7 @@
 """custom-html — pre-op seed hooks (Phase 1e HC3). The orchestrator runs `pre_<op>(domain, meta)`
 BEFORE it performs the op; the matching test_<op>.py asserts the post-op state (assertion-only).
 
-nginx serves the volume at /usr/share/nginx/html, so the marker file survives an upgrade / a
+nginx serves the volume at /var/www/html, so the marker file survives an upgrade / a
 backup+restore of that volume and is both HTTP-readable and exec-readable."""
 
 import os
@@ -10,11 +10,11 @@ import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
 from harness import lifecycle  # noqa: E402
 
-MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
+MARKER_PATH = "/var/www/html/ci-marker.txt"
 
 
 def _write(domain: str, val: str) -> None:
-    lifecycle.exec_in_app(domain, ["sh", "-c", f"echo {val} > {MARKER_PATH}"])
+    lifecycle.exec_in_app(domain, ["sh", "-c", f"mkdir -p $(dirname {MARKER_PATH}) && echo {val} > {MARKER_PATH}"])
 
 
 def pre_upgrade(domain, meta):

tests/custom-html/test_backup.py

@@ -12,7 +12,7 @@ import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
 from harness import lifecycle  # noqa: E402
 
-MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
+MARKER_PATH = "/var/www/html/ci-marker.txt"
 
 
 def test_backup_captures_state(live_app):

tests/custom-html/test_restore.py

@@ -12,7 +12,7 @@ import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
 from harness import lifecycle  # noqa: E402
 
-MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
+MARKER_PATH = "/var/www/html/ci-marker.txt"
 
 
 def test_restore_returns_state(live_app):

tests/custom-html/test_upgrade.py

@@ -11,7 +11,7 @@ import sys
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "runner"))
 from harness import lifecycle  # noqa: E402
 
-MARKER_PATH = "/usr/share/nginx/html/ci-marker.txt"
+MARKER_PATH = "/var/www/html/ci-marker.txt"
 
 
 def test_upgrade_preserves_data(live_app):