Per-recipe and fleet-wide upgrade skills modelled on recipe-maintainer's
recipe-upgrade-full / recipe-upgrade-cron-all, but gated by the cc-ci CI server
and inheriting ci-test-review's create+verify+never-merge discipline.
- recipe-upgrade/: plan (release notes, breaking changes) -> implement (abra
recipe upgrade + version bump + config, lint) -> open the recipe PR -> VERIFY
green on cc-ci (full suite cold against the PR head via verify-pr.sh). If the
upgrade is correct but a cc-ci TEST went stale, also update the test, verify
it, and open a second PR to recipe-maintainers/cc-ci. Never merges; never
weakens a test; prefers a recipe-only PR. Emits a parseable RESULT line.
+ open-recipe-pr.sh: adapted recipe-create-pr; runs on cc-ci (has the recipe
checkout + bot token), creds passed from the orchestrator .testenv;
force-syncs the mirror main so the PR diff is exactly the upgrade.
- upgrade-all/: weekly fan-out — enumerate enrolled recipes, survey upgrades,
run /recipe-upgrade per upgradeable recipe via subagent (sequential default,
--parallel / --dry-run), collect into one PR-list summary. Coordination +
single-writer + shared-Swarm-teardown guardrails; built for a weekly cron.
- ci-test-review/verify-pr.sh: pass SRC (recipe-maintainers/<recipe>) alongside
REF so the harness clones the mirror PR head correctly (its real contract).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
7.5 KiB
name, description
| name | description |
|---|---|
| recipe-upgrade | Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), then VERIFIES the change green on cc-ci (full suite, cold, against the PR head) and opens a recipe PR. If the upgrade is correct but a cc-ci TEST is now stale, it also updates the test, verifies that, and opens a second PR to the cc-ci repo. NEVER merges — leaves verified, ready-to-merge PRs. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade <recipe>. |
recipe-upgrade
Autonomous, end-to-end upgrade of one recipe, gated by the cc-ci CI server. This is the
cc-ci analogue of recipe-maintainer's /recipe-upgrade-full, but the test gate is cc-ci (the real
recipe CI), and it inherits ci-test-review's discipline: classify recipe-vs-CI, create + verify
fix PRs, never merge, never weaken a test.
It drives cc-ci over ssh cc-ci (cc-ci has abra, the recipe checkouts at ~/.abra/recipes/, the
harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's
/srv/cc-ci/.testenv creds. Plans/reports go to /srv/cc-ci/.cc-ci-logs/upgrades/.
Boundaries (same as ci-test-review): the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). Create + verify, NEVER merge (operator merges). Never weaken a test to make a PR green.
Procedure
1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)
On cc-ci, refresh the recipe and see what's available:
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \
git -C ~/.abra/recipes/<recipe> status --short; \
abra recipe fetch <recipe> --force; \
git -C ~/.abra/recipes/<recipe> fetch origin main; \
abra recipe versions <recipe> -m; \
abra recipe upgrade <recipe> -m -n'
- Dirty worktree on cc-ci → abort this recipe (status
SKIPPED — dirty-worktree); don't prompt. - No upgrades available → stop (status
SKIPPED — up-to-date). - Check
git log HEAD..origin/mainand upstream PRs (git.coopcloud.tech/coop-cloud/<recipe>/pulls) — if someone already started the bump, re-plan from the tip oforigin/main, not from scratch. - For each service with an upgrade, fetch upstream release notes (WebFetch) between current and target versions and call out breaking changes / required migrations / new-or-renamed config / dependency bumps in an "Operator Action Required" section.
- Write the plan to
/srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.)
2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)
On cc-ci's ~/.abra/recipes/<recipe>:
abra recipe upgrade <recipe> -nfor the tags it can bump; hand-editcompose.ymlfor any tags it won't, plus the config/env/volume/label changes the plan calls for.- Bump the
coop-cloud.${STACK_NAME}.versionlabel to the new version string from the plan. abra recipe lint <recipe> -C— fix obvious lint errors; if unfixable, record it and continue.- Commit on a branch:
git commit -m "chore: upgrade to <new-version>"(the commit message drives the PR branch nameupgrade-<new-version>). Do not push to upstream; do not tag-push.
3. Open the recipe PR (never merge)
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
This pushes the upgrade branch to recipe-maintainers/<recipe> (force-syncing mirror main from
upstream so the diff is exactly the upgrade) and opens the PR. Capture the PR_URL. Optionally export
RECIPE_PR_BODY first (image-tag table + the planned operator-action notes).
4. VERIFY the upgrade on the CI server (the gate)
Run the cc-ci full suite cold against the PR head — the dogfood gate from ci-test-review:
RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh
(SRC defaults to recipe-maintainers/<recipe>.) GREEN ⇔ harness exits 0 → the recipe PR is
verified, ready for operator merge. General bar = one cold green; use REPEAT=3 only for a recipe
already known to be FLAKY (e.g. lasuite-drive). Record the run summary in the report.
5. If verification is RED → diagnose (recipe vs cc-ci TEST), per ci-test-review
Read the harness log and classify the failure:
- The UPGRADE itself is broken (real regression: bad tag, missing migration, config gap) → fix it
on the recipe branch (bounded — ≤3 iterations), re-push, re-verify. The recipe PR is only "working"
once cc-ci is green. If still red after the budget, leave the PR open and report
FAILED — upgrade not green. - The upgrade is correct but a cc-ci TEST is now stale/wrong for the new version (asserts old
behavior, an overlay needs updating, a readiness gate changed) → this is a CI-server fix. Make
it the
ci-test-reviewway:- Branch
recipe-maintainers/cc-ciin a separate clone (single-writer: never pushmain, never touch the build loops'/cc-ci/cc-ci-advclones), update the test/overlay. - Verify the recipe upgrade with the updated test applied: check the cc-ci branch out on
cc-ci, rebuild if needed, re-run
verify-pr.shfor the recipe + a small regression sample. - Open the cc-ci test PR via the Gitea API (
/srv/cc-ci/.testenvcreds). Capture its URL. - The recipe PR + the cc-ci test PR are a pair — note in each that it depends on the other.
- Branch
- FLAKY (passes on a re-run) → note it; don't author a fix for a flake.
Never weaken a test to turn a red upgrade green.
6. Report — single parseable RESULT line + summary
Write the report to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md and print, as
the last line, one of these exact prefixes (so /upgrade-all can collect it):
RESULT: SUCCESS — <recipe> <old> → <new>, cc-ci GREEN, recipe PR: <url>RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>, cc-ci GREEN; recipe PR: <url>; cc-ci test PR: <url>RESULT: FAILED — <recipe> at <step>: <one-line reason>(e.g. upgrade not green after 3 tries)RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>
Always state explicitly that nothing was merged — the PR(s) await operator review.
Guardrails
- cc-ci is the gate — deterministic harness decides pass/fail; AI plans, implements, diagnoses.
- Create + verify, NEVER merge. Recipe PR (and any cc-ci test PR) are operator-merged after review.
- Prefer a recipe-only PR. Only open a cc-ci test PR when the upgrade is correct but a test is genuinely stale — not to paper over a real upgrade regression.
- Never weaken a test; bounded changes (the upgrade + minimal test update, not rewrites).
- Real abra path throughout (
abra recipe upgrade/abra app deployvia the harness). - Single-writer / coordination: dedicated branches + separate clones; never push
mainor disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). Seeci-test-reviewfor the shared rules.