--- name: recipe-upgrade description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), then VERIFIES the change green on cc-ci (full suite, cold, against the PR head) and opens a recipe PR. If the upgrade is correct but a cc-ci TEST is now stale, it also updates the test, verifies that, and opens a second PR to the cc-ci repo. NEVER merges — leaves verified, ready-to-merge PRs. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade . --- # recipe-upgrade Autonomous, end-to-end upgrade of **one** recipe, **gated by the cc-ci CI server**. This is the cc-ci analogue of recipe-maintainer's `/recipe-upgrade-full`, but the test gate is **cc-ci** (the real recipe CI), and it inherits `ci-test-review`'s discipline: classify recipe-vs-CI, create + **verify** fix PRs, **never merge**, never weaken a test. It drives cc-ci over `ssh cc-ci` (cc-ci has abra, the recipe checkouts at `~/.abra/recipes/`, the harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's `/srv/cc-ci/.testenv` creds. Plans/reports go to `/srv/cc-ci/.cc-ci-logs/upgrades/`. **Boundaries (same as ci-test-review):** the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). **Create + verify, NEVER merge** (operator merges). **Never weaken a test** to make a PR green. ## Procedure ### 1. Plan the upgrade (research — follow recipe-upgrade-plan methodology) On cc-ci, refresh the recipe and see what's available: ``` ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \ git -C ~/.abra/recipes/ status --short; \ abra recipe fetch --force; \ git -C ~/.abra/recipes/ fetch origin main; \ abra recipe versions -m; \ abra recipe upgrade -m -n' ``` - **Dirty worktree on cc-ci → abort this recipe** (status `SKIPPED — dirty-worktree`); don't prompt. - **No upgrades available → stop** (status `SKIPPED — up-to-date`). - Check `git log HEAD..origin/main` and upstream PRs (`git.coopcloud.tech/coop-cloud//pulls`) — if someone already started the bump, **re-plan from the tip of `origin/main`**, not from scratch. - For each service with an upgrade, fetch upstream **release notes** (WebFetch) between current and target versions and call out **breaking changes / required migrations / new-or-renamed config / dependency bumps** in an "Operator Action Required" section. - Write the plan to `/srv/cc-ci/.cc-ci-logs/upgrades/-upgrade-.md`: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.) ### 2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci) On cc-ci's `~/.abra/recipes/`: - `abra recipe upgrade -n` for the tags it can bump; hand-edit `compose.yml` for any tags it won't, plus the config/env/volume/label changes the plan calls for. - Bump the `coop-cloud.${STACK_NAME}.version` label to the new version string from the plan. - `abra recipe lint -C` — fix obvious lint errors; if unfixable, record it and continue. - Commit on a branch: `git commit -m "chore: upgrade to "` (the commit message drives the PR branch name `upgrade-`). Do **not** push to upstream; do **not** tag-push. ### 3. Open the recipe PR (never merge) ``` set -a; . /srv/cc-ci/.testenv; set +a ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s " \ < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh ``` This pushes the upgrade branch to `recipe-maintainers/` (force-syncing mirror `main` from upstream so the diff is exactly the upgrade) and opens the PR. Capture the `PR_URL`. Optionally export `RECIPE_PR_BODY` first (image-tag table + the planned operator-action notes). ### 4. VERIFY the upgrade on the CI server (the gate) Run the cc-ci full suite **cold** against the PR head — the dogfood gate from `ci-test-review`: ``` RECIPE= REF=upgrade- /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh ``` (`SRC` defaults to `recipe-maintainers/`.) **GREEN** ⇔ harness exits 0 → the recipe PR is verified, ready for operator merge. **General bar = one cold green**; use `REPEAT=3` only for a recipe already known to be FLAKY (e.g. lasuite-drive). Record the run summary in the report. ### 5. If verification is RED → diagnose (recipe vs cc-ci TEST), per ci-test-review Read the harness log and classify the failure: - **The UPGRADE itself is broken** (real regression: bad tag, missing migration, config gap) → fix it on the recipe branch (bounded — ≤3 iterations), re-push, re-verify. The recipe PR is only "working" once cc-ci is green. If still red after the budget, leave the PR open and report `FAILED — upgrade not green`. - **The upgrade is correct but a cc-ci TEST is now stale/wrong** for the new version (asserts old behavior, an overlay needs updating, a readiness gate changed) → this is a **CI-server fix**. Make it the `ci-test-review` way: 1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`, never touch the build loops' `/cc-ci` `/cc-ci-adv` clones), update the test/overlay. 2. **Verify** the recipe upgrade **with the updated test applied**: check the cc-ci branch out on cc-ci, rebuild if needed, re-run `verify-pr.sh` for the recipe + a small regression sample. 3. Open the **cc-ci test PR** via the Gitea API (`/srv/cc-ci/.testenv` creds). Capture its URL. 4. The recipe PR + the cc-ci test PR are a **pair** — note in each that it depends on the other. - **FLAKY** (passes on a re-run) → note it; don't author a fix for a flake. Never weaken a test to turn a red upgrade green. ### 6. Report — single parseable RESULT line + summary Write the report to `/srv/cc-ci/.cc-ci-logs/upgrades/-upgrade-.md` and print, as the **last line**, one of these exact prefixes (so `/upgrade-all` can collect it): - `RESULT: SUCCESS — → , cc-ci GREEN, recipe PR: ` - `RESULT: SUCCESS+TESTPR — → , cc-ci GREEN; recipe PR: ; cc-ci test PR: ` - `RESULT: FAILED — at : ` (e.g. upgrade not green after 3 tries) - `RESULT: SKIPPED — : ` Always state explicitly that **nothing was merged** — the PR(s) await operator review. ## Guardrails - **cc-ci is the gate** — deterministic harness decides pass/fail; AI plans, implements, diagnoses. - **Create + verify, NEVER merge.** Recipe PR (and any cc-ci test PR) are operator-merged after review. - **Prefer a recipe-only PR.** Only open a cc-ci test PR when the upgrade is correct but a test is genuinely stale — not to paper over a real upgrade regression. - **Never weaken a test**; **bounded** changes (the upgrade + minimal test update, not rewrites). - **Real abra path** throughout (`abra recipe upgrade` / `abra app deploy` via the harness). - **Single-writer / coordination:** dedicated branches + separate clones; never push `main` or disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). See `ci-test-review` for the shared rules.