Per operator: an open mirror PR must mean "genuinely still open against true current upstream main". On every run the recipe-upgrade flow now: - force-syncs the recipe-maintainers/<recipe> mirror `main` to be IDENTICAL to upstream main (origin/main of the abra checkout = coopcloud); - closes any open mirror PR whose changes are already in upstream main (merged upstream, no-op merge detected via `git merge-tree` vs main's tree) — even when the recipe is up to date (new `--reconcile-only` mode, run in step 1); - when opening a new upgrade PR, closes any other still-open PR for that recipe (superseded) and opens the new one IN ITS PLACE; same-version re-runs just update the existing same-branch PR. open-recipe-pr.sh gains the --reconcile-only mode + the close logic (with an auto-close comment naming the reason). upgrade-all reconciles every candidate's mirror during the survey so merged PRs are closed fleet-wide. Still never merges. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8.8 KiB
name, description
| name | description |
|---|---|
| recipe-upgrade | Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), then VERIFIES the change green on cc-ci (full suite, cold, against the PR head) and opens a recipe PR. If the upgrade is correct but a cc-ci TEST is now stale, it also updates the test, verifies that, and opens a second PR to the cc-ci repo. NEVER merges — leaves verified, ready-to-merge PRs. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade <recipe>. |
recipe-upgrade
Autonomous, end-to-end upgrade of one recipe, gated by the cc-ci CI server. This is the
cc-ci analogue of recipe-maintainer's /recipe-upgrade-full, but the test gate is cc-ci (the real
recipe CI), and it inherits ci-test-review's discipline: classify recipe-vs-CI, create + verify
fix PRs, never merge, never weaken a test.
It drives cc-ci over ssh cc-ci (cc-ci has abra, the recipe checkouts at ~/.abra/recipes/, the
harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's
/srv/cc-ci/.testenv creds. Plans/reports go to /srv/cc-ci/.cc-ci-logs/upgrades/.
Boundaries (same as ci-test-review): the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). Create + verify, NEVER merge (operator merges). Never weaken a test to make a PR green.
Procedure
1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)
On cc-ci, refresh the recipe and see what's available:
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \
git -C ~/.abra/recipes/<recipe> status --short; \
abra recipe fetch <recipe> --force; \
git -C ~/.abra/recipes/<recipe> fetch origin main; \
abra recipe versions <recipe> -m; \
abra recipe upgrade <recipe> -m -n'
- Dirty worktree on cc-ci → abort this recipe (status
SKIPPED — dirty-worktree); don't prompt. - Reconcile the mirror first (always, even if up to date). Run:
This force-syncs the
set -a; . /srv/cc-ci/.testenv; set +a ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \ < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.shrecipe-maintainers/<recipe>mirrormainto be identical to true upstream main, and closes any open mirror PR whose changes are already in upstream main (merged upstream — the mirror just never reflected it). Do this before the up-to-date check so a stale already-merged PR gets cleaned up even when there's nothing new to upgrade. - No upgrades available → stop (status
SKIPPED — up-to-date) — after the reconcile above. - Check
git log HEAD..origin/mainand upstream PRs (git.coopcloud.tech/coop-cloud/<recipe>/pulls) — if someone already started the bump, re-plan from the tip oforigin/main, not from scratch. - For each service with an upgrade, fetch upstream release notes (WebFetch) between current and target versions and call out breaking changes / required migrations / new-or-renamed config / dependency bumps in an "Operator Action Required" section.
- Write the plan to
/srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.)
2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)
On cc-ci's ~/.abra/recipes/<recipe>:
abra recipe upgrade <recipe> -nfor the tags it can bump; hand-editcompose.ymlfor any tags it won't, plus the config/env/volume/label changes the plan calls for.- Bump the
coop-cloud.${STACK_NAME}.versionlabel to the new version string from the plan. abra recipe lint <recipe> -C— fix obvious lint errors; if unfixable, record it and continue.- Commit on a branch:
git commit -m "chore: upgrade to <new-version>"(the commit message drives the PR branch nameupgrade-<new-version>). Do not push to upstream; do not tag-push.
3. Open the recipe PR (never merge)
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
This (a) re-syncs mirror main to true upstream main, (b) closes any open PR already merged
upstream, (c) closes any other still-open PR for this recipe as superseded, then pushes the
upgrade branch and opens the new PR in its place (so the diff is exactly this upgrade, against the
real current upstream main). Capture the PR_URL. Optionally export RECIPE_PR_BODY first (image-tag
table + the planned operator-action notes). Re-running with the same target version updates the
existing same-branch PR rather than duplicating it.
4. VERIFY the upgrade on the CI server (the gate)
Run the cc-ci full suite cold against the PR head — the dogfood gate from ci-test-review:
RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh
(SRC defaults to recipe-maintainers/<recipe>.) GREEN ⇔ harness exits 0 → the recipe PR is
verified, ready for operator merge. General bar = one cold green; use REPEAT=3 only for a recipe
already known to be FLAKY (e.g. lasuite-drive). Record the run summary in the report.
5. If verification is RED → diagnose (recipe vs cc-ci TEST), per ci-test-review
Read the harness log and classify the failure:
- The UPGRADE itself is broken (real regression: bad tag, missing migration, config gap) → fix it
on the recipe branch (bounded — ≤3 iterations), re-push, re-verify. The recipe PR is only "working"
once cc-ci is green. If still red after the budget, leave the PR open and report
FAILED — upgrade not green. - The upgrade is correct but a cc-ci TEST is now stale/wrong for the new version (asserts old
behavior, an overlay needs updating, a readiness gate changed) → this is a CI-server fix. Make
it the
ci-test-reviewway:- Branch
recipe-maintainers/cc-ciin a separate clone (single-writer: never pushmain, never touch the build loops'/cc-ci/cc-ci-advclones), update the test/overlay. - Verify the recipe upgrade with the updated test applied: check the cc-ci branch out on
cc-ci, rebuild if needed, re-run
verify-pr.shfor the recipe + a small regression sample. - Open the cc-ci test PR via the Gitea API (
/srv/cc-ci/.testenvcreds). Capture its URL. - The recipe PR + the cc-ci test PR are a pair — note in each that it depends on the other.
- Branch
- FLAKY (passes on a re-run) → note it; don't author a fix for a flake.
Never weaken a test to turn a red upgrade green.
6. Report — single parseable RESULT line + summary
Write the report to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md and print, as
the last line, one of these exact prefixes (so /upgrade-all can collect it):
RESULT: SUCCESS — <recipe> <old> → <new>, cc-ci GREEN, recipe PR: <url>RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>, cc-ci GREEN; recipe PR: <url>; cc-ci test PR: <url>RESULT: FAILED — <recipe> at <step>: <one-line reason>(e.g. upgrade not green after 3 tries)RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>
Always state explicitly that nothing was merged — the PR(s) await operator review.
Guardrails
- cc-ci is the gate — deterministic harness decides pass/fail; AI plans, implements, diagnoses.
- Create + verify, NEVER merge. Recipe PR (and any cc-ci test PR) are operator-merged after review.
- Mirror reflects reality. Each run force-syncs the
recipe-maintainers/<recipe>mirrormainto true upstream main, closes open PRs already merged upstream, and replaces any superseded open PR with the new one — so an open mirror PR always means "genuinely still open against current upstream main". - Prefer a recipe-only PR. Only open a cc-ci test PR when the upgrade is correct but a test is genuinely stale — not to paper over a real upgrade regression.
- Never weaken a test; bounded changes (the upgrade + minimal test update, not rewrites).
- Real abra path throughout (
abra recipe upgrade/abra app deployvia the harness). - Single-writer / coordination: dedicated branches + separate clones; never push
mainor disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). Seeci-test-reviewfor the shared rules.