# STATUS — cc-ci Phase 5 Builder

**Phase:** 5 — Verify `/recipe-upgrade` + `testme-on-pr.sh` end-to-end flow
**SSOT:** `/srv/cc-ci/cc-ci-plan/plan-phase5-verify-upgrade-flow.md`
**Started:** 2026-05-31

## Current focus

V3, V5, V6 FULL PASS (Adversary-verified). V7 DONE.
V8 live run: upgrader running uptime-kuma upgrade. Build #91 IN PROGRESS.
- `!testme` comment #13903 posted by upgrader at 21:57:51Z
- Bridge triggered build #91 for uptime-kuma@72861889 (PR #1, comment 13903)
- Commit status: cc-ci/testme state=pending on uptime-kuma PR#1 head
- Monitoring for build completion (10-20 min estimated)
V8a lifecycle: partial (start-idle → fresh, start-busy → leave-alone verified)

## Fix A5-6: uptime-kuma bridge enrollment

**A5-6 FIX:** `nix/modules/bridge.nix` commit `51ba205`: added `recipe-maintainers/uptime-kuma`
to POLL_REPOS. Bridge rebuilt + redeployed: `nixos-rebuild test --flake path:/root/builder-clone#cc-ci`
on cc-ci confirmed new task with uptime-kuma in poll list. Upgrader restarted.
Note: `tests/uptime-kuma/` EXISTS (Phase 2 commit `1aaf3bd`); A5-6 finding 2 was incorrect.

## Fixes applied (A5-1, A5-2, related)

**A5-2 FIX:** `bridge/bridge.py` commit `5d48436`: `post_commit_status()` added. Bridge POSTs
Gitea commit status on recipe PR's head SHA (pending→trigger, success/failure→finish).

**A5-1 FIX:** `nix/modules/bridge.nix` commit `5d48436`: `recipe-maintainers/custom-html-tiny`
added to POLL_REPOS. Bridge rebuilt: `cc-ci-bridge:3761c4221042` (via `nixos-rebuild build
--flake path:/root/builder-clone#cc-ci` on cc-ci + `cc-ci-reconcile-bridge`).

**open-recipe-pr.sh FIX (orchestrator repo):** `0df57c6` — replaced python3 with jq (cc-ci
has jq, not python3).

**testme-on-pr.sh FIX (orchestrator repo):** `6910b19` — reads cc-ci/testme context URL
instead of first-status URL (fixes wrong BUILD URL when multiple statuses exist).

**A5-3 FIX (orchestrator repo, uncommitted):** `testme-on-pr.sh` now ignores a pre-existing
`cc-ci/testme` status on the same PR head after `POST=1` until the status tuple changes, so a
fresh re-`!testme` no longer returns a stale prior GREEN/build URL.

**ci-test-review helper FIX (orchestrator repo, uncommitted):** `verify-pr.sh` and
`run-all-recipes.sh` now resolve the live host checkout dynamically (`/root/builder-clone`
preferred, `/root/cc-ci` fallback) instead of hard-coding `/root/cc-ci`.

## V3 — COMPLETE: /recipe-upgrade custom-html-tiny END-TO-END GREEN

**Upgrade PR:** `https://git.autonomic.zone/recipe-maintainers/custom-html-tiny/pulls/2`
- Branch: `upgrade-1.1.0+2.42.0`, head sha `156a49ac`
- Changes: compose.yml sws 2.38.0→2.42.0; compose.git-pull.yml alpine/git v2.36.3→v2.52.0; version 1.0.1+2.38.0→1.1.0+2.42.0
- !testme posted → Drone build #29 triggered → SUCCESS (install PASS, upgrade PASS, backup N/A)
- Commit status: `cc-ci/testme state=success target=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29`
- `POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 2` → `VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29`
- PR comment updated by bridge with 🌻 result

## V7 — COMPLETE: mirror reconciliation

- PR #1 (`serve-hidden-files`) auto-closed as superseded when PR #2 opened.
- PR #4 (`already-in-upstream-v7`) auto-closed as merged-upstream.
- Mirror `main` force-synced to upstream `main` (`435df8fc`).

**V1/V2 partial evidence:**
- V1: !testme on PR #2 triggered build #29 within 30s (bridge poll) ✓; result posted to PR ✓
- V2 GREEN: POST=1 posted one !testme; POST=0 polled and returned VERDICT=GREEN BUILD=<drone-url> ✓
- V2 RED: poll-only on PR #5 returned VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/34 ✓
- V2 rerun edge: `POST=1 MAX_WAIT=80 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5`
  now returns the fresh rerun build `#43` (not the stale prior `#37`); PR comments `4 -> 5` ✓

## V4 — COMPLETE: 2-run regression loop (within the 3-run budget)

**Regression PR:** `https://git.autonomic.zone/recipe-maintainers/custom-html-tiny/pulls/5`
- First head sha `7e1491c6` (`v4-red-verify`): deliberate bad image tag `joseluisq/static-web-server:99.0.0-bad-tag`
- `POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5` → `VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/34`
- Build #34 result: install PASS, upgrade FAIL, clean_teardown=true, no_secret_leak=true
- Fix pushed on the same PR branch: head sha `4bd8416a`, restoring the known-good upgrade files from `upgrade-1.1.0+2.42.0`
- Re-`!testme` on PR #5 → Drone build #37 → `VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37`
- PR remains open and unmerged; both RED and GREEN results are recorded on the PR

## Verification item status

| Item | Status | Evidence |
|---|---|---|
| V1 — !testme trigger + result-back | PARTIAL | build #29 triggered in <30s; commit status + PR comment posted ✓ |
| V2 — testme-on-pr.sh reads verdict | DONE | GREEN ✓ (build #29/#35); RED ✓ (build #34); rerun fix ✓ (build #43) |
| V3 — /recipe-upgrade sandbox GREEN | DONE | custom-html-tiny PR#2; build #29 SUCCESS |
| V4 — 3-iter regression loop | DONE | custom-html-tiny PR#5; build #34 RED, build #37 GREEN |
| V5 — stale-test DEFAULT = comment | PASS (Adversary) | A5-5 CLOSED 21:49Z; build #81; comment #13900; RESULT log @ /srv/cc-ci/.cc-ci-logs/upgrades/custom-html-upgrade-2026-06-01.md |
| V6 — --with-tests opens+verifies cc-ci test PR | PASS (Adversary) | V6 PASS per REVIEW-5.md 21:38Z; cc-ci PR#3; verify-pr.sh GREEN |
| V7 — mirror reconciliation | DONE | PR#1 superseded, PR#4 merged-upstream, main=upstream ✓ |
| V8 — /upgrade-all DEFAULT run | TODO | |
| V8a — cc-ci-upgrader agent | TODO | |
| V9 — cleanup | TODO | |

## V5/V6 groundwork in progress

- Added orchestration helpers in `/srv/cc-ci-orch/.claude/skills/`:
  - `recipe-upgrade/post-pr-comment.sh` — post explanatory/cross-link PR comments via Gitea API
  - `ci-test-review/open-cc-ci-pr.sh` — open/update `recipe-maintainers/cc-ci` PRs from a dedicated branch
- Live candidate check: `ssh cc-ci "abra recipe upgrade n8n -m -n"` shows a real n8n upgrade path
  (`n8nio/n8n 2.20.6 -> 2.23.1`, `pgautoupgrade 17-alpine -> 18-alpine`).
- Live recipe PR proof: `https://git.autonomic.zone/recipe-maintainers/n8n/pulls/2`
  (`upgrade-3.3.0+2.23.1`, head `c8d27a2`). `!testme` build #47 returned
  `VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/47`.
- Conclusion: `n8n` is a good sandbox for V5/V6, but this real upgrade did **not** naturally surface the
  stale-test path. Next step is to seed the stale-test case explicitly on a sandbox/scratch branch per
  Phase 5 §2, then exercise DEFAULT comment-only and `--with-tests` flows against that seeded case.
- Second live candidate check: `cryptpad` app image `version-2026.2.0 -> version-2026.5.1` plus
  `nginx 1.29 -> 1.31` on PR `https://git.autonomic.zone/recipe-maintainers/cryptpad/pulls/3`
  (`upgrade-0.5.5+v2026.5.1`, head `9db61d3`) also went GREEN on `!testme` build `#50`.
- Additional live finding: `lasuite-meet` has a real upgrade path (`v1.16.0 -> v1.17.0`), but its PR
  `https://git.autonomic.zone/recipe-maintainers/lasuite-meet/pulls/2` stayed `VERDICT=PENDING BUILD=?`
  across repeated `POST=0` polls because `recipe-maintainers/lasuite-meet` is not in the bridge's
  enrolled poll list. That makes it unusable for V5/V6 until explicitly enrolled.
- Enrollment fix authored and pushed: `f28a2a3 fix(bridge): enroll lasuite-meet for !testme` adds
  `recipe-maintainers/lasuite-meet` to `nix/modules/bridge.nix` `POLL_REPOS`.
- Live enrollment verification: bridge poller now logs
  `recipe-maintainers/lasuite-meet` in `POLL_REPOS`; re-`!testme` on PR #2 triggered build `#55`.
- Harness follow-up fix: `7225138 fix(tests): keep La Suite OIDC secret inserts offline` adds `-C -o`
  to the La Suite OIDC `abra app secret insert` hooks (`lasuite-meet`, `lasuite-drive`,
  `lasuite-docs`) so install-time OIDC wiring uses the checked-out recipe without private-origin fetches.
- Result: `POST=1 ... testme-on-pr.sh lasuite-meet 2` now returns `VERDICT=GREEN`
  `BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/58`.
- V5 live candidate: `matrix-synapse` PR `https://git.autonomic.zone/recipe-maintainers/matrix-synapse/pulls/1`
  (`upgrade-7.2.0+v1.153.0`, head `21e5d844`) triggered build `#53` and returned RED.
  Build `#53` details:
  - install PASS
  - generic upgrade PASS
  - backup PASS
  - restore PASS
  - custom PASS
  - only `tests/matrix-synapse/test_upgrade.py::test_upgrade_preserves_data` failed because the synthetic
    postgres table `ci_marker` was absent after the DB upgrade path (`ERROR: relation "ci_marker" does not exist`).
  Default-mode explanatory PR comment posted with no test edit:
  `https://git.autonomic.zone/recipe-maintainers/matrix-synapse/pulls/1#issuecomment-13877`
  telling the operator to re-run `/recipe-upgrade matrix-synapse --with-tests` for a test-update PR.
- Adversary finding A5-4 is now cleared on current live behavior: re-`!testme` on the same PR head
  produced build `#63`; `POST=0 ... testme-on-pr.sh matrix-synapse 1` returned
  `VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`; and
  `GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d844.../status` now shows
  `cc-ci/testme state=failure target_url=.../63`.
- V6 branch verification on `matrix-synapse` no longer supports the stale-test hypothesis. In a
  dedicated cc-ci branch checkout with a real Matrix data-survival upgrade assertion, the helper path
  now resolves the recipe branch to its head SHA correctly, generic upgrade PASSes, but the upgraded
  app still fails the real post-upgrade assertion: the pre-upgrade Matrix user cannot log in after the
  upgrade (`HTTP 403 Invalid username or password`). That points to a true recipe upgrade regression,
  not a stale test.
- Seeded Phase-5 sandbox stale-test case (operator-directed simulation):
  - Recipe PR: `https://git.autonomic.zone/recipe-maintainers/custom-html/pulls/3`
    - branch: `v5-stale-docroot`, head `71e7326a`
    - seeded behavior: `.txt` files are intentionally served as `application/octet-stream` while the
      app remains externally healthy and lifecycle tiers still pass.
  - DEFAULT/V5 evidence:
    - `POST=1 ... testme-on-pr.sh custom-html 3` -> build `#75`
    - `POST=0 ... testme-on-pr.sh custom-html 3` ->
      `VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/75`
    - build `#75` summary: install PASS, upgrade PASS, backup PASS, restore PASS, only custom FAIL
    - exact failing stale assertion: `tests/custom-html/functional/test_content_type_header.py`
      expected `.txt` `Content-Type` to start with `text/plain`, but got `application/octet-stream`
    - explanatory recipe-PR comment with no cc-ci test edit:
      `https://git.autonomic.zone/recipe-maintainers/custom-html/pulls/3#issuecomment-13883`
  - `--with-tests`/V6 evidence:
    - paired cc-ci branch: `origin/v6-custom-html-mime` @ `826daec`
    - paired cc-ci PR: `https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/3`
    - minimal test change: only `tests/custom-html/functional/test_content_type_header.py` updated so
      the seeded sandbox `.txt` response expects `application/octet-stream`
    - cold branch-checkout verification on cc-ci:
      `REMOTE_ROOT=/root/cc-ci-v6-custom-mime RECIPE=custom-html REF=v5-stale-docroot /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh`
    - expected/observed result:
      `VERDICT: GREEN — custom-html PR (REF=v5-stale-docroot) passed cold full-suite x1. Ready for operator merge (NOT merged).`
      Host log: `cc-ci:/root/cc-ci-review-logs/verify-custom-html-20260601T200544Z.1.log`
    - cross-link comments posted:
      - recipe PR note: `https://git.autonomic.zone/recipe-maintainers/custom-html/pulls/3#issuecomment-13894`
      - cc-ci PR note: `https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/3#issuecomment-13896`

## Verification next step

- With seeded V5/V6 now exercised, move on to V8 `/upgrade-all`, V8a `cc-ci-upgrader`, and V9 cleanup.

## Phase 5 gates

(None claimed yet.)

## Blocked

(none)