--- name: recipe-upgrade description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), opens a recipe PR, and verifies it by posting `!testme` on the PR (real CI; results visible in the PR; iterates up to 3×). DEFAULT: recipe PR only, using existing tests — if a test fails because it is genuinely stale, it leaves an explanatory COMMENT on the PR for the operator (does NOT touch tests). With `--with-tests`: also opens + verifies a PR to update the stale cc-ci test. NEVER merges. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade [--with-tests]. --- # recipe-upgrade Autonomous, end-to-end upgrade of **one** recipe, **gated by the cc-ci CI server**. This is the cc-ci analogue of recipe-maintainer's `/recipe-upgrade-full`, but the test gate is **cc-ci** (the real recipe CI), and it inherits `ci-test-review`'s discipline: classify recipe-vs-CI, create + **verify** fix PRs, **never merge**, never weaken a test. It drives cc-ci over `ssh cc-ci` (cc-ci has abra, the recipe checkouts at `~/.abra/recipes/`, the harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's `/srv/cc-ci/.testenv` creds. Plans/reports go to `/srv/cc-ci/.cc-ci-logs/upgrades/`. **Boundaries (same as ci-test-review):** the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). **Create + verify, NEVER merge** (operator merges). **Never weaken a test** to make a PR green. ## Arguments - `` — the recipe to upgrade (required). - `--with-tests` (optional) — **opt in to updating cc-ci tests.** Without it (the DEFAULT, and what the weekly `/upgrade-all` cron uses), this skill **only ever opens a recipe PR using the existing tests**; if a test fails and is diagnosed as genuinely stale, it **leaves a comment** on the recipe PR explaining why and stops — it does **not** modify any test. With `--with-tests`, a stale-test failure additionally gets a **verified cc-ci test-update PR** (step 5b). A real upgrade regression is fixed + re-tested in **both** modes. ## Procedure > ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`) > `abra` needs a TTY. Over plain `ssh cc-ci 'abra …'` there is none, so many abra commands die with > **`FATA inappropriate ioctl for device`** / `the input device is not a TTY`. This is how the working > harness avoids it (`runner/harness/abra.py`): wrap every abra call in a pseudo-TTY via util-linux > `script`, and pass `-n` (`--no-input`) so it never prompts. **Always invoke abra like this:** > ``` > ssh cc-ci 'script -qec "abra -n" /dev/null' > ``` > (`git`/other commands do NOT need the wrapper — only `abra`.) Do this for every abra command below. ### 1. Plan the upgrade (research — follow recipe-upgrade-plan methodology) On cc-ci, refresh the recipe and see what's available (each abra call wrapped per the box above): ``` ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \ git -C ~/.abra/recipes/ status --short; \ script -qec "abra recipe fetch --force -n" /dev/null; \ git -C ~/.abra/recipes/ fetch origin main; \ script -qec "abra recipe versions -m -n" /dev/null; \ script -qec "abra recipe upgrade -m -n" /dev/null' ``` - **Dirty worktree on cc-ci → abort this recipe** (status `SKIPPED — dirty-worktree`); don't prompt. - **Reconcile the mirror first (always, even if up to date).** Run: ``` set -a; . /srv/cc-ci/.testenv; set +a ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s --reconcile-only" \ < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh ``` This force-syncs the `recipe-maintainers/` mirror `main` to be **identical to true upstream main**, and **closes any open mirror PR whose changes are already in upstream main** (merged upstream — the mirror just never reflected it). Do this before the up-to-date check so a stale already-merged PR gets cleaned up even when there's nothing new to upgrade. - **No upgrades available → stop** (status `SKIPPED — up-to-date`) — after the reconcile above. - Check `git log HEAD..origin/main` and upstream PRs (`git.coopcloud.tech/coop-cloud//pulls`) — if someone already started the bump, **re-plan from the tip of `origin/main`**, not from scratch. - For each service with an upgrade, fetch upstream **release notes** (WebFetch) between current and target versions and call out **breaking changes / required migrations / new-or-renamed config / dependency bumps** in an "Operator Action Required" section. - Write the plan to `/srv/cc-ci/.cc-ci-logs/upgrades/-upgrade-.md`: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.) ### 2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci) On cc-ci's `~/.abra/recipes/` (wrap every abra call per the pseudo-TTY box above — `ssh cc-ci 'script -qec "abra … -n" /dev/null'`): - `script -qec "abra recipe upgrade -n" /dev/null` for the tags it can bump; hand-edit `compose.yml` for any tags it won't, plus the config/env/volume/label changes the plan calls for. - Bump the `coop-cloud.${STACK_NAME}.version` label to the new version string from the plan. - `script -qec "abra recipe lint -C -n" /dev/null` — fix obvious lint errors; if unfixable, record it and continue. - Commit on a branch: `git commit -m "chore: upgrade to "` (the commit message drives the PR branch name `upgrade-`). Do **not** push to upstream; do **not** tag-push. ### 3. Open the recipe PR (never merge) ``` set -a; . /srv/cc-ci/.testenv; set +a ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s " \ < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh ``` This (a) re-syncs mirror `main` to true upstream main, (b) **closes any open PR already merged upstream** (merging it would be a no-op), then handles the upgrade PR: **if an open upgrade PR already exists for this recipe (branch `upgrade-*`), the new work is pushed onto THAT PR's branch and the PR is updated + re-tested** — one evolving upgrade PR per recipe, not a second parallel one. Only if none exists is a fresh `upgrade-` PR opened. Unrelated open PRs (e.g. a backup fix) are **NOT** closed or touched — the operator decides on those. The PR's diff is exactly this upgrade against the real current upstream main. Capture the `PR_URL`. Optionally export `RECIPE_PR_BODY` first (image-tag table + the planned operator-action notes). Re-running with the same target version updates the existing same-branch PR rather than duplicating it. ### 4. VERIFY by running `!testme` ON THE PR (results visible in the PR; iterate ≤3×) Trigger the **real** CI on the recipe PR by posting `!testme` — the bridge runs the harness on cc-ci and posts the run + its results back to the PR, so the operator sees them in the PR. Use the PR index from step 3's `PR_URL` (`.../pulls/`): ``` /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh # POST=1: posts !testme + polls # if it prints VERDICT=PENDING, poll again WITHOUT re-triggering: POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh ``` - **GREEN** → the recipe PR is verified on cc-ci, results in the PR, ready for operator merge. Done. - **RED** → go to step 5. You get **up to 3 `!testme` runs total** on this PR (initial + 2 retries): use them to fix and re-test a real upgrade regression (step 5a). Each retry is a fresh `!testme` after you push a fix, so every attempt's result is recorded in the PR. ### 5. If `!testme` is RED → diagnose from the run (recipe regression vs stale cc-ci TEST) Read the failing run (linked from the PR / the harness log on cc-ci) and classify: **5a. The UPGRADE itself is broken** (real regression: bad tag, missing migration, config gap) — fix it on the recipe branch, push, and re-run `!testme` (within the 3-run budget). The recipe PR is only "working" once `!testme` is GREEN. If still red after 3 runs, leave the PR open (its red `!testme` results stand as the evidence) and report `FAILED — upgrade not green after 3 !testme runs`. **5b. The upgrade is correct but a cc-ci TEST is genuinely stale/wrong** for the new version (asserts old behavior, an overlay needs updating, a readiness gate changed). `!testme` can't go green without a test change, and a test change is **gated by `--with-tests`**: - **DEFAULT (no `--with-tests`) — comment, don't touch tests.** Post a **comment on the recipe PR** that: states the upgrade itself looks correct; identifies the specific test that failed and **why it appears stale** (what the new version changed vs what the test asserts); and tells the operator they can **re-run `/recipe-upgrade --with-tests`** to also open + verify a cc-ci test-update PR. Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a stale test; operator to decide). - **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way: 1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`, never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay. 2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the *deployed/main* cc-ci tests, so it can't see the unmerged test change — verify this combo by running the harness with the cc-ci branch checked out on cc-ci: `RECIPE= REF=upgrade- /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh` (after checking out the cc-ci test branch on cc-ci + rebuilding if needed), plus a small regression sample. Green ⇔ the upgrade passes under the corrected test. 3. Open the **cc-ci test PR** via the Gitea API. Note in BOTH PRs that they are a dependent pair (the recipe PR's `!testme` goes green only once the test PR is merged). Report `SUCCESS+TESTPR`. **FLAKY** (passes on a re-`!testme`) → note it; don't author a fix for a flake. Never weaken a test to turn a red upgrade green. ### 6. Report — single parseable RESULT line + summary Write the report to `/srv/cc-ci/.cc-ci-logs/upgrades/-upgrade-.md` and print, as the **last line**, one of these exact prefixes (so `/upgrade-all` can collect it): - `RESULT: SUCCESS — → , !testme GREEN, recipe PR: ` - `RESULT: SUCCESS-PENDING-TESTS — → , recipe PR: ; !testme RED on a stale test (commented; re-run --with-tests to update tests)` - `RESULT: SUCCESS+TESTPR — → ; recipe PR: ; cc-ci test PR: (verified with the test change; pair)` - `RESULT: FAILED — at : ` (e.g. upgrade not green after 3 !testme runs) - `RESULT: SKIPPED — : ` Always state explicitly that **nothing was merged** — the PR(s) await operator review. ## Guardrails - **cc-ci is the gate, via `!testme` on the PR** — the harness decides pass/fail and the results live in the PR (operator-visible); AI plans, implements, diagnoses. Up to 3 `!testme` runs per PR. - **Create + verify, NEVER merge.** Recipe PR (and any cc-ci test PR) are operator-merged after review. - **Tests are opt-in.** Default = recipe PR only, existing tests; a stale-test failure gets an explanatory PR **comment**, never a test edit. Only `--with-tests` opens a cc-ci test PR. The weekly `/upgrade-all` cron always runs the **default** — it never auto-updates tests. - **Mirror reflects reality.** Each run force-syncs the `recipe-maintainers/` mirror `main` to true upstream main and closes open PRs already merged upstream. It does NOT close superseded/unrelated open PRs. **An existing open upgrade PR is extended (new work pushed onto its branch + re-tested) rather than spawning a second upgrade PR** — one evolving upgrade PR per recipe. - **Prefer a recipe-only PR.** Only open a cc-ci test PR (under `--with-tests`) when the upgrade is correct but a test is genuinely stale — never to paper over a real upgrade regression. - **Never weaken a test**; **bounded** changes (the upgrade + minimal test update, not rewrites). - **Real abra path** throughout (`abra recipe upgrade` / `abra app deploy` via the harness). - **Single-writer / coordination:** dedicated branches + separate clones; never push `main` or disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). See `ci-test-review` for the shared rules.