178 lines
12 KiB
Markdown
178 lines
12 KiB
Markdown
# STATUS — cc-ci Phase 5 Builder
|
|
|
|
**Phase:** 5 — Verify `/recipe-upgrade` + `testme-on-pr.sh` end-to-end flow
|
|
**SSOT:** `/srv/cc-ci/cc-ci-plan/plan-phase5-verify-upgrade-flow.md`
|
|
**Started:** 2026-05-31
|
|
|
|
## Current focus
|
|
|
|
A5-5 FIXED: ran full `/recipe-upgrade custom-html` DEFAULT skill against seeded PR#3. Fresh build #81
|
|
confirms same MIME-type failure; accurate explanatory comment posted (#13900, referencing build #81);
|
|
RESULT log written at `/srv/cc-ci/.cc-ci-logs/upgrades/custom-html-upgrade-2026-06-01.md`.
|
|
V6 PASS confirmed by Adversary (cold-verified: cc-ci PR#3, verify-pr.sh GREEN).
|
|
**Active work: V8 `/upgrade-all` + V8a `cc-ci-upgrader` agent lifecycle.**
|
|
|
|
## Fixes applied (A5-1, A5-2, related)
|
|
|
|
**A5-2 FIX:** `bridge/bridge.py` commit `5d48436`: `post_commit_status()` added. Bridge POSTs
|
|
Gitea commit status on recipe PR's head SHA (pending→trigger, success/failure→finish).
|
|
|
|
**A5-1 FIX:** `nix/modules/bridge.nix` commit `5d48436`: `recipe-maintainers/custom-html-tiny`
|
|
added to POLL_REPOS. Bridge rebuilt: `cc-ci-bridge:3761c4221042` (via `nixos-rebuild build
|
|
--flake path:/root/builder-clone#cc-ci` on cc-ci + `cc-ci-reconcile-bridge`).
|
|
|
|
**open-recipe-pr.sh FIX (orchestrator repo):** `0df57c6` — replaced python3 with jq (cc-ci
|
|
has jq, not python3).
|
|
|
|
**testme-on-pr.sh FIX (orchestrator repo):** `6910b19` — reads cc-ci/testme context URL
|
|
instead of first-status URL (fixes wrong BUILD URL when multiple statuses exist).
|
|
|
|
**A5-3 FIX (orchestrator repo, uncommitted):** `testme-on-pr.sh` now ignores a pre-existing
|
|
`cc-ci/testme` status on the same PR head after `POST=1` until the status tuple changes, so a
|
|
fresh re-`!testme` no longer returns a stale prior GREEN/build URL.
|
|
|
|
**ci-test-review helper FIX (orchestrator repo, uncommitted):** `verify-pr.sh` and
|
|
`run-all-recipes.sh` now resolve the live host checkout dynamically (`/root/builder-clone`
|
|
preferred, `/root/cc-ci` fallback) instead of hard-coding `/root/cc-ci`.
|
|
|
|
## V3 — COMPLETE: /recipe-upgrade custom-html-tiny END-TO-END GREEN
|
|
|
|
**Upgrade PR:** `https://git.autonomic.zone/recipe-maintainers/custom-html-tiny/pulls/2`
|
|
- Branch: `upgrade-1.1.0+2.42.0`, head sha `156a49ac`
|
|
- Changes: compose.yml sws 2.38.0→2.42.0; compose.git-pull.yml alpine/git v2.36.3→v2.52.0; version 1.0.1+2.38.0→1.1.0+2.42.0
|
|
- !testme posted → Drone build #29 triggered → SUCCESS (install PASS, upgrade PASS, backup N/A)
|
|
- Commit status: `cc-ci/testme state=success target=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29`
|
|
- `POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 2` → `VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/29`
|
|
- PR comment updated by bridge with 🌻 result
|
|
|
|
## V7 — COMPLETE: mirror reconciliation
|
|
|
|
- PR #1 (`serve-hidden-files`) auto-closed as superseded when PR #2 opened.
|
|
- PR #4 (`already-in-upstream-v7`) auto-closed as merged-upstream.
|
|
- Mirror `main` force-synced to upstream `main` (`435df8fc`).
|
|
|
|
**V1/V2 partial evidence:**
|
|
- V1: !testme on PR #2 triggered build #29 within 30s (bridge poll) ✓; result posted to PR ✓
|
|
- V2 GREEN: POST=1 posted one !testme; POST=0 polled and returned VERDICT=GREEN BUILD=<drone-url> ✓
|
|
- V2 RED: poll-only on PR #5 returned VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/34 ✓
|
|
- V2 rerun edge: `POST=1 MAX_WAIT=80 INTERVAL=5 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5`
|
|
now returns the fresh rerun build `#43` (not the stale prior `#37`); PR comments `4 -> 5` ✓
|
|
|
|
## V4 — COMPLETE: 2-run regression loop (within the 3-run budget)
|
|
|
|
**Regression PR:** `https://git.autonomic.zone/recipe-maintainers/custom-html-tiny/pulls/5`
|
|
- First head sha `7e1491c6` (`v4-red-verify`): deliberate bad image tag `joseluisq/static-web-server:99.0.0-bad-tag`
|
|
- `POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh custom-html-tiny 5` → `VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/34`
|
|
- Build #34 result: install PASS, upgrade FAIL, clean_teardown=true, no_secret_leak=true
|
|
- Fix pushed on the same PR branch: head sha `4bd8416a`, restoring the known-good upgrade files from `upgrade-1.1.0+2.42.0`
|
|
- Re-`!testme` on PR #5 → Drone build #37 → `VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/37`
|
|
- PR remains open and unmerged; both RED and GREEN results are recorded on the PR
|
|
|
|
## Verification item status
|
|
|
|
| Item | Status | Evidence |
|
|
|---|---|---|
|
|
| V1 — !testme trigger + result-back | PARTIAL | build #29 triggered in <30s; commit status + PR comment posted ✓ |
|
|
| V2 — testme-on-pr.sh reads verdict | DONE | GREEN ✓ (build #29/#35); RED ✓ (build #34); rerun fix ✓ (build #43) |
|
|
| V3 — /recipe-upgrade sandbox GREEN | DONE | custom-html-tiny PR#2; build #29 SUCCESS |
|
|
| V4 — 3-iter regression loop | DONE | custom-html-tiny PR#5; build #34 RED, build #37 GREEN |
|
|
| V5 — stale-test DEFAULT = comment | DONE | seeded custom-html PR#3; full skill run: build #81 RED (MIME only); accurate comment #13900; RESULT log written; no test edits |
|
|
| V6 — --with-tests opens+verifies cc-ci test PR | DONE | paired cc-ci PR#3 opened; cold branch-checkout verify GREEN against custom-html PR#3 |
|
|
| V7 — mirror reconciliation | DONE | PR#1 superseded, PR#4 merged-upstream, main=upstream ✓ |
|
|
| V8 — /upgrade-all DEFAULT run | TODO | |
|
|
| V8a — cc-ci-upgrader agent | TODO | |
|
|
| V9 — cleanup | TODO | |
|
|
|
|
## V5/V6 groundwork in progress
|
|
|
|
- Added orchestration helpers in `/srv/cc-ci-orch/.claude/skills/`:
|
|
- `recipe-upgrade/post-pr-comment.sh` — post explanatory/cross-link PR comments via Gitea API
|
|
- `ci-test-review/open-cc-ci-pr.sh` — open/update `recipe-maintainers/cc-ci` PRs from a dedicated branch
|
|
- Live candidate check: `ssh cc-ci "abra recipe upgrade n8n -m -n"` shows a real n8n upgrade path
|
|
(`n8nio/n8n 2.20.6 -> 2.23.1`, `pgautoupgrade 17-alpine -> 18-alpine`).
|
|
- Live recipe PR proof: `https://git.autonomic.zone/recipe-maintainers/n8n/pulls/2`
|
|
(`upgrade-3.3.0+2.23.1`, head `c8d27a2`). `!testme` build #47 returned
|
|
`VERDICT=GREEN BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/47`.
|
|
- Conclusion: `n8n` is a good sandbox for V5/V6, but this real upgrade did **not** naturally surface the
|
|
stale-test path. Next step is to seed the stale-test case explicitly on a sandbox/scratch branch per
|
|
Phase 5 §2, then exercise DEFAULT comment-only and `--with-tests` flows against that seeded case.
|
|
- Second live candidate check: `cryptpad` app image `version-2026.2.0 -> version-2026.5.1` plus
|
|
`nginx 1.29 -> 1.31` on PR `https://git.autonomic.zone/recipe-maintainers/cryptpad/pulls/3`
|
|
(`upgrade-0.5.5+v2026.5.1`, head `9db61d3`) also went GREEN on `!testme` build `#50`.
|
|
- Additional live finding: `lasuite-meet` has a real upgrade path (`v1.16.0 -> v1.17.0`), but its PR
|
|
`https://git.autonomic.zone/recipe-maintainers/lasuite-meet/pulls/2` stayed `VERDICT=PENDING BUILD=?`
|
|
across repeated `POST=0` polls because `recipe-maintainers/lasuite-meet` is not in the bridge's
|
|
enrolled poll list. That makes it unusable for V5/V6 until explicitly enrolled.
|
|
- Enrollment fix authored and pushed: `f28a2a3 fix(bridge): enroll lasuite-meet for !testme` adds
|
|
`recipe-maintainers/lasuite-meet` to `nix/modules/bridge.nix` `POLL_REPOS`.
|
|
- Live enrollment verification: bridge poller now logs
|
|
`recipe-maintainers/lasuite-meet` in `POLL_REPOS`; re-`!testme` on PR #2 triggered build `#55`.
|
|
- Harness follow-up fix: `7225138 fix(tests): keep La Suite OIDC secret inserts offline` adds `-C -o`
|
|
to the La Suite OIDC `abra app secret insert` hooks (`lasuite-meet`, `lasuite-drive`,
|
|
`lasuite-docs`) so install-time OIDC wiring uses the checked-out recipe without private-origin fetches.
|
|
- Result: `POST=1 ... testme-on-pr.sh lasuite-meet 2` now returns `VERDICT=GREEN`
|
|
`BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/58`.
|
|
- V5 live candidate: `matrix-synapse` PR `https://git.autonomic.zone/recipe-maintainers/matrix-synapse/pulls/1`
|
|
(`upgrade-7.2.0+v1.153.0`, head `21e5d844`) triggered build `#53` and returned RED.
|
|
Build `#53` details:
|
|
- install PASS
|
|
- generic upgrade PASS
|
|
- backup PASS
|
|
- restore PASS
|
|
- custom PASS
|
|
- only `tests/matrix-synapse/test_upgrade.py::test_upgrade_preserves_data` failed because the synthetic
|
|
postgres table `ci_marker` was absent after the DB upgrade path (`ERROR: relation "ci_marker" does not exist`).
|
|
Default-mode explanatory PR comment posted with no test edit:
|
|
`https://git.autonomic.zone/recipe-maintainers/matrix-synapse/pulls/1#issuecomment-13877`
|
|
telling the operator to re-run `/recipe-upgrade matrix-synapse --with-tests` for a test-update PR.
|
|
- Adversary finding A5-4 is now cleared on current live behavior: re-`!testme` on the same PR head
|
|
produced build `#63`; `POST=0 ... testme-on-pr.sh matrix-synapse 1` returned
|
|
`VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/63`; and
|
|
`GET /repos/recipe-maintainers/matrix-synapse/commits/21e5d844.../status` now shows
|
|
`cc-ci/testme state=failure target_url=.../63`.
|
|
- V6 branch verification on `matrix-synapse` no longer supports the stale-test hypothesis. In a
|
|
dedicated cc-ci branch checkout with a real Matrix data-survival upgrade assertion, the helper path
|
|
now resolves the recipe branch to its head SHA correctly, generic upgrade PASSes, but the upgraded
|
|
app still fails the real post-upgrade assertion: the pre-upgrade Matrix user cannot log in after the
|
|
upgrade (`HTTP 403 Invalid username or password`). That points to a true recipe upgrade regression,
|
|
not a stale test.
|
|
- Seeded Phase-5 sandbox stale-test case (operator-directed simulation):
|
|
- Recipe PR: `https://git.autonomic.zone/recipe-maintainers/custom-html/pulls/3`
|
|
- branch: `v5-stale-docroot`, head `71e7326a`
|
|
- seeded behavior: `.txt` files are intentionally served as `application/octet-stream` while the
|
|
app remains externally healthy and lifecycle tiers still pass.
|
|
- DEFAULT/V5 evidence:
|
|
- `POST=1 ... testme-on-pr.sh custom-html 3` -> build `#75`
|
|
- `POST=0 ... testme-on-pr.sh custom-html 3` ->
|
|
`VERDICT=RED BUILD=https://drone.ci.commoninternet.net/recipe-maintainers/cc-ci/75`
|
|
- build `#75` summary: install PASS, upgrade PASS, backup PASS, restore PASS, only custom FAIL
|
|
- exact failing stale assertion: `tests/custom-html/functional/test_content_type_header.py`
|
|
expected `.txt` `Content-Type` to start with `text/plain`, but got `application/octet-stream`
|
|
- explanatory recipe-PR comment with no cc-ci test edit:
|
|
`https://git.autonomic.zone/recipe-maintainers/custom-html/pulls/3#issuecomment-13883`
|
|
- `--with-tests`/V6 evidence:
|
|
- paired cc-ci branch: `origin/v6-custom-html-mime` @ `826daec`
|
|
- paired cc-ci PR: `https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/3`
|
|
- minimal test change: only `tests/custom-html/functional/test_content_type_header.py` updated so
|
|
the seeded sandbox `.txt` response expects `application/octet-stream`
|
|
- cold branch-checkout verification on cc-ci:
|
|
`REMOTE_ROOT=/root/cc-ci-v6-custom-mime RECIPE=custom-html REF=v5-stale-docroot /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh`
|
|
- expected/observed result:
|
|
`VERDICT: GREEN — custom-html PR (REF=v5-stale-docroot) passed cold full-suite x1. Ready for operator merge (NOT merged).`
|
|
Host log: `cc-ci:/root/cc-ci-review-logs/verify-custom-html-20260601T200544Z.1.log`
|
|
- cross-link comments posted:
|
|
- recipe PR note: `https://git.autonomic.zone/recipe-maintainers/custom-html/pulls/3#issuecomment-13894`
|
|
- cc-ci PR note: `https://git.autonomic.zone/recipe-maintainers/cc-ci/pulls/3#issuecomment-13896`
|
|
|
|
## Verification next step
|
|
|
|
- With seeded V5/V6 now exercised, move on to V8 `/upgrade-all`, V8a `cc-ci-upgrader`, and V9 cleanup.
|
|
|
|
## Phase 5 gates
|
|
|
|
(None claimed yet.)
|
|
|
|
## Blocked
|
|
|
|
(none)
|