cc-ci-orchestrator/.claude/skills/recipe-upgrade/SKILL.md

---

name: recipe-upgrade description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), opens a recipe PR, and verifies it by posting !testme on the PR (real CI; results visible in the PR; iterates up to 3×). DEFAULT: recipe PR only, using existing tests — if a test fails because it is genuinely stale, it leaves an explanatory COMMENT on the PR for the operator (does NOT touch tests). With --with-tests: also opens + verifies a PR to update the stale cc-ci test. NEVER merges. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade [--with-tests].

recipe-upgrade

Autonomous, end-to-end upgrade of one recipe, gated by the cc-ci CI server. This is the cc-ci analogue of recipe-maintainer's /recipe-upgrade-full, but the test gate is cc-ci (the real recipe CI), and it inherits ci-test-review's discipline: classify recipe-vs-CI, create + verify fix PRs, never merge, never weaken a test.

It drives cc-ci over ssh cc-ci (cc-ci has abra, the recipe checkouts at ~/.abra/recipes/, the harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's /srv/cc-ci/.testenv creds. Plans/reports go to /srv/cc-ci/.cc-ci-logs/upgrades/.

Boundaries (same as ci-test-review): the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). Create + verify, NEVER merge (operator merges). Never weaken a test to make a PR green.

Arguments

• <recipe> — the recipe to upgrade (required).
• --with-tests (optional) — opt in to updating cc-ci tests. Without it (the DEFAULT, and what the weekly /upgrade-all cron uses), this skill only ever opens a recipe PR using the existing tests; if a test fails and is diagnosed as genuinely stale, it leaves a comment on the recipe PR explaining why and stops — it does not modify any test. With --with-tests, a stale-test failure additionally gets a verified cc-ci test-update PR (step 5b). A real upgrade regression is fixed + re-tested in both modes.

Procedure

1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)

On cc-ci, refresh the recipe and see what's available:

ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \
  git -C ~/.abra/recipes/<recipe> status --short; \
  abra recipe fetch <recipe> --force; \
  git -C ~/.abra/recipes/<recipe> fetch origin main; \
  abra recipe versions <recipe> -m; \
  abra recipe upgrade <recipe> -m -n'

• Dirty worktree on cc-ci → abort this recipe (status SKIPPED — dirty-worktree); don't prompt.
• Reconcile the mirror first (always, even if up to date). Run:

  set -a; . /srv/cc-ci/.testenv; set +a
  ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
     < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
  

  This force-syncs the recipe-maintainers/<recipe> mirror main to be identical to true upstream main, and closes any open mirror PR whose changes are already in upstream main (merged upstream — the mirror just never reflected it). Do this before the up-to-date check so a stale already-merged PR gets cleaned up even when there's nothing new to upgrade.
• No upgrades available → stop (status SKIPPED — up-to-date) — after the reconcile above.
• Check git log HEAD..origin/main and upstream PRs (git.coopcloud.tech/coop-cloud/<recipe>/pulls) — if someone already started the bump, re-plan from the tip of origin/main, not from scratch.
• For each service with an upgrade, fetch upstream release notes (WebFetch) between current and target versions and call out breaking changes / required migrations / new-or-renamed config / dependency bumps in an "Operator Action Required" section.
• Write the plan to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.)

2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)

On cc-ci's ~/.abra/recipes/<recipe>:

• abra recipe upgrade <recipe> -n for the tags it can bump; hand-edit compose.yml for any tags it won't, plus the config/env/volume/label changes the plan calls for.
• Bump the coop-cloud.${STACK_NAME}.version label to the new version string from the plan.
• abra recipe lint <recipe> -C — fix obvious lint errors; if unfixable, record it and continue.
• Commit on a branch: git commit -m "chore: upgrade to <new-version>" (the commit message drives the PR branch name upgrade-<new-version>). Do not push to upstream; do not tag-push.

3. Open the recipe PR (never merge)

set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh

This (a) re-syncs mirror main to true upstream main, (b) closes any open PR already merged upstream, (c) closes any other still-open PR for this recipe as superseded, then pushes the upgrade branch and opens the new PR in its place (so the diff is exactly this upgrade, against the real current upstream main). Capture the PR_URL. Optionally export RECIPE_PR_BODY first (image-tag table + the planned operator-action notes). Re-running with the same target version updates the existing same-branch PR rather than duplicating it.

4. VERIFY by running !testme ON THE PR (results visible in the PR; iterate ≤3×)

Trigger the real CI on the recipe PR by posting !testme — the bridge runs the harness on cc-ci and posts the run + its results back to the PR, so the operator sees them in the PR. Use the PR index from step 3's PR_URL (.../pulls/<N>):

/srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>     # POST=1: posts !testme + polls
# if it prints VERDICT=PENDING, poll again WITHOUT re-triggering:
POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>

• GREEN → the recipe PR is verified on cc-ci, results in the PR, ready for operator merge. Done.
• RED → go to step 5. You get up to 3 !testme runs total on this PR (initial + 2 retries): use them to fix and re-test a real upgrade regression (step 5a). Each retry is a fresh !testme after you push a fix, so every attempt's result is recorded in the PR.

5. If !testme is RED → diagnose from the run (recipe regression vs stale cc-ci TEST)

Read the failing run (linked from the PR / the harness log on cc-ci) and classify:

5a. The UPGRADE itself is broken (real regression: bad tag, missing migration, config gap) — fix it on the recipe branch, push, and re-run !testme (within the 3-run budget). The recipe PR is only "working" once !testme is GREEN. If still red after 3 runs, leave the PR open (its red !testme results stand as the evidence) and report FAILED — upgrade not green after 3 !testme runs.

5b. The upgrade is correct but a cc-ci TEST is genuinely stale/wrong for the new version (asserts old behavior, an overlay needs updating, a readiness gate changed). !testme can't go green without a test change, and a test change is gated by --with-tests:

• DEFAULT (no --with-tests) — comment, don't touch tests. Post a comment on the recipe PR that: states the upgrade itself looks correct; identifies the specific test that failed and why it appears stale (what the new version changed vs what the test asserts); and tells the operator they can re-run /recipe-upgrade <recipe> --with-tests to also open + verify a cc-ci test-update PR. Do NOT modify any test. Report SUCCESS-PENDING-TESTS (recipe PR open; !testme red on a stale test; operator to decide).
• --with-tests — open + verify a cc-ci test PR. Make it the ci-test-review way:
  1. Branch recipe-maintainers/cc-ci in a separate clone (single-writer: never push main, never touch the build loops' /cc-ci /cc-ci-adv clones); update the test/overlay.
  2. Verify the recipe upgrade WITH the updated test applied. !testme on the recipe PR uses the deployed/main cc-ci tests, so it can't see the unmerged test change — verify this combo by running the harness with the cc-ci branch checked out on cc-ci: RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh (after checking out the cc-ci test branch on cc-ci + rebuilding if needed), plus a small regression sample. Green ⇔ the upgrade passes under the corrected test.
  3. Open the cc-ci test PR via the Gitea API. Note in BOTH PRs that they are a dependent pair (the recipe PR's !testme goes green only once the test PR is merged). Report SUCCESS+TESTPR.

FLAKY (passes on a re-!testme) → note it; don't author a fix for a flake.

Never weaken a test to turn a red upgrade green.

6. Report — single parseable RESULT line + summary

Write the report to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md and print, as the last line, one of these exact prefixes (so /upgrade-all can collect it):

• RESULT: SUCCESS — <recipe> <old> → <new>, !testme GREEN, recipe PR: <url>
• RESULT: SUCCESS-PENDING-TESTS — <recipe> <old> → <new>, recipe PR: <url>; !testme RED on a stale test (commented; re-run --with-tests to update tests)
• RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>; recipe PR: <url>; cc-ci test PR: <url> (verified with the test change; pair)
• RESULT: FAILED — <recipe> at <step>: <one-line reason> (e.g. upgrade not green after 3 !testme runs)
• RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>

Always state explicitly that nothing was merged — the PR(s) await operator review.

Guardrails

• cc-ci is the gate, via !testme on the PR — the harness decides pass/fail and the results live in the PR (operator-visible); AI plans, implements, diagnoses. Up to 3 !testme runs per PR.
• Create + verify, NEVER merge. Recipe PR (and any cc-ci test PR) are operator-merged after review.
• Tests are opt-in. Default = recipe PR only, existing tests; a stale-test failure gets an explanatory PR comment, never a test edit. Only --with-tests opens a cc-ci test PR. The weekly /upgrade-all cron always runs the default — it never auto-updates tests.
• Mirror reflects reality. Each run force-syncs the recipe-maintainers/<recipe> mirror main to true upstream main, closes open PRs already merged upstream, and replaces any superseded open PR with the new one — so an open mirror PR always means "genuinely still open against current upstream main".
• Prefer a recipe-only PR. Only open a cc-ci test PR (under --with-tests) when the upgrade is correct but a test is genuinely stale — never to paper over a real upgrade regression.
• Never weaken a test; bounded changes (the upgrade + minimal test update, not rewrites).
• Real abra path throughout (abra recipe upgrade / abra app deploy via the harness).
• Single-writer / coordination: dedicated branches + separate clones; never push main or disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). See ci-test-review for the shared rules.