cc-ci-orchestrator/cc-ci-plan/plan-phase5-verify-upgrade-flow.md

cc-ci Phase 5 — verify the /recipe-upgrade + testme-on-pr.sh end-to-end flow

Status: QUEUED — the FINAL phase, after Phase 4. Transition: auto (last in the launcher sequence); the watchdog STOPS after it (build complete). Owner: Builder + Adversary loops (Adversary cold-verifies); the orchestrator may also drive it. This file: /srv/cc-ci/cc-ci-plan/plan-phase5-verify-upgrade-flow.md Phase order: … 3 → 4 → 5.

---

0. Why this is LAST

The /recipe-upgrade and /upgrade-all skills live in the orchestration repo (/srv/cc-ci/.claude/skills/) and drive cc-ci by posting !testme on a recipe PR (testme-on-pr.sh) and reading the verdict back from the PR. That flow only works once cc-ci is fully built — specifically it depends on Phase 3 (results UX): the run's result must be posted back to the PR (a Gitea commit status and/or result comment on the PR head) for testme-on-pr.sh to read it. So this verification can only run after the CI server is complete. It proves the upgrade tooling actually works end-to-end before the weekly /upgrade-all cron is activated (cc-ci-upgrade-all-cron — activate only after the build is done).

Use a sandbox recipe for the live runs (e.g. custom-html-tiny or a dedicated throwaway recipe), and close every PR opened during verification afterward — do not pollute real recipe mirrors, and never merge anything.

1. Definition of Done (Adversary cold-verifies → machine-docs/REVIEW-5.md)

• V1 — !testme on a recipe PR is the real gate. Posting !testme (as the bot, a collaborator) on a recipe mirror PR triggers bridge → Drone → harness, runs the suite, and posts the run + its result back to the PR (commit status on the PR head and/or a result comment, per Phase 3). A !testme from a non-collaborator is rejected. A bad trigger (!testmexyz) does NOT fire. (Re-uses the existing !testme D-gate evidence.)
• V2 — testme-on-pr.sh reads the verdict correctly. POST=1 posts exactly one !testme; POST=0 polls WITHOUT re-triggering; the helper returns VERDICT=GREEN on a passing PR, VERDICT=RED on a failing PR, VERDICT=PENDING while in flight, and a BUILD=<url> that points at the Drone build. (Drive a known-green and a known-red PR head.)
• V3 — /recipe-upgrade <sandbox-recipe> (DEFAULT mode) end-to-end GREEN. On a recipe with a real available upgrade: it reconciles the mirror, opens a recipe PR with the bump, !testme runs and results appear in the PR, GREEN → RESULT: SUCCESS. Nothing merged.
• V4 — 3-iteration regression loop. Seed a real upgrade regression (e.g. a deliberately bad image tag) → !testme RED → fix on the recipe branch → re-!testme → GREEN, all within ≤3 !testme runs, each recorded in the PR. With a persistent regression, after 3 runs it leaves the PR open and reports RESULT: FAILED — upgrade not green after 3 !testme runs (does not weaken anything to force green).
• V5 — stale-test DEFAULT = comment, no test edit. With a genuinely stale test for the new version, default mode (no --with-tests) leaves an explanatory comment on the recipe PR (upgrade looks correct; which test is stale + why; "re-run --with-tests"), modifies no test, and reports RESULT: SUCCESS-PENDING-TESTS.
• V6 — --with-tests opens + verifies a cc-ci test PR. The same stale-test case under --with-tests opens a cc-ci test-update PR (dedicated branch, separate clone — never main, never the loops' clones), verifies the recipe upgrade WITH the test change applied via the branch-checkout harness run (verify-pr.sh, since !testme uses prod tests), pairs the two PRs with cross-notes, and reports RESULT: SUCCESS+TESTPR. Nothing merged.
• V7 — mirror reconciliation. After a run, the recipe-maintainers/<recipe> mirror main is identical to upstream main; an open PR whose change is already in upstream main is auto-closed (merged-upstream); a superseded open PR is closed and replaced by the new one. (Seed each case and confirm.)
• V8 — /upgrade-all DEFAULT run. --dry-run lists candidates correctly; a live run over a small set (1–2 sandbox recipes) opens recipe PRs using default mode (never --with-tests), surfaces "tests look stale" PRs in its own summary section, runs sequentially with teardown between recipes, and the summary leads with the PR list. Confirms the weekly cron never auto-edits tests.
• V8a — the cc-ci-upgrader agent. cc-ci-plan/launch-upgrader.sh start spins up a remote-control cc-ci-upgrader session (viewable at claude.ai/code) that runs /upgrade-all (DEFAULT) to completion, then STOPS and stays idle (does NOT self-terminate) with the summary visible. A second start while a run is in flight (busy) leaves it alone; a start against a finished/idle (or wedged) session kills it and runs fresh. (Use UPGRADER_ARGS=--dry-run for a cheap check, then a real small-set run.) This is exactly what the weekly cron invokes — verify the cron-equivalent path end-to-end.
• V9 — cleanup. Every PR opened during verification (recipe + any cc-ci test PR) is closed and any sandbox deploy is torn down; the verification cc-ci-upgrader session is stopped (launch-upgrader.sh stop); the box is left clean.

2. Method / notes

• Sandbox first. Prefer custom-html-tiny / a throwaway recipe for V3–V8 so real recipe mirrors aren't spammed. For the seeded-regression (V4) and stale-test (V5/V6) cases, construct the seed on a sandbox recipe / a scratch branch, not a real upstream recipe.
• Real path throughout — real !testme → bridge → Drone → harness → results-in-PR; real abra.
• The skills are at /srv/cc-ci/.claude/skills/{recipe-upgrade,upgrade-all,ci-test-review}/; read their SKILL.md as the spec for expected behavior.
• If Phase 3's PR-result surface differs from what testme-on-pr.sh polls (commit status vs comment), reconcile them here — either adjust the helper to read the real surface, or file a CI finding — and record the resolution in DECISIONS.md. (This is the most likely real gap to find.)

3. Guardrails

• NEVER merge any PR; never weaken a test to make a run pass.
• Close verification PRs + tear down deploys when done (V9) — leave no junk.
• Single-writer: cc-ci test PRs on dedicated branches + separate clones; never push main or touch the loops' working clones; shared Swarm is stateful — serialize + tear down.
• Bounded: this phase VERIFIES the flow; it does not redesign the skills. A real defect → fix the skill/helper (small) or file it; a bigger idea → IDEAS.md.

4. On phase completion — install the weekly cron (+1h) and verify it fires

Once V1–V9 pass (the upgrade flow + the cc-ci-upgrader agent are proven) AND this is the final phase (the whole cc-ci build is complete), the orchestrator installs the recurring weekly upgrade cron. This SUPERSEDES the earlier fixed "Sat 03:00 UTC" placeholder — the schedule is now anchored to actual completion (cc-ci-upgrade-all-cron).

1. Schedule = completion + 1 hour, weekly thereafter. Compute T0 = now + 1h; install a weekly job at T0's day-of-week + HH:MM (cron MM HH * * DOW) so the FIRST run fires ~1h after the build finishes, then every 7 days at that same slot.
2. What it runs: /srv/cc-ci/cc-ci-plan/launch-upgrader.sh start (→ spins up the cc-ci-upgrader agent → /upgrade-all DEFAULT → stops idle/viewable). The cron environment must have the claude CLI on PATH, tmux access, and the box logged into the claude.ai account (remote-control) — a bare cron env is minimal, so set PATH/HOME explicitly (mirror how cc-ci-loops.service launches). A user crontab / systemd timer / Claude scheduled task are all acceptable mechanisms; pick one that gives that environment.
3. VERIFY the first kickoff (the point of this step).
   • Cheap pre-check first: UPGRADER_ARGS=--dry-run cc-ci-plan/launch-upgrader.sh fresh (or schedule a one-off test fire a couple of minutes out) to confirm the launcher path works from the chosen cron mechanism's environment.
   • Then confirm the real first fire at T0 (~1h later) actually kicked it off: launch-upgrader.sh status shows cc-ci-upgrader RUNNING, it ran /upgrade-all, and produced a summary. If it did NOT fire/launch (PATH, login, mechanism), fix and re-verify.
   • Record the installed schedule + the verified-kickoff result in DECISIONS.md.