cc-ci-orchestrator/.claude/skills/recipe-upgrade/SKILL.md

---

name: recipe-upgrade description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades — fetching each image's release notes via a persisted per-recipe registry (cc-ci-plan/upstream/.md) + the recipe README — plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), opens a recipe PR, and verifies it by posting !testme on the PR (real CI; results visible in the PR; iterates up to 3×). DEFAULT: recipe PR only, using existing tests — if a test fails because it is genuinely stale, it leaves an explanatory COMMENT on the PR for the operator (does NOT touch tests). With --with-tests: also opens + verifies a PR to update the stale cc-ci test. NEVER merges. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade [--with-tests].

recipe-upgrade

Autonomous, end-to-end upgrade of one recipe, gated by the cc-ci CI server. This is the cc-ci analogue of recipe-maintainer's /recipe-upgrade-full, but the test gate is cc-ci (the real recipe CI), and it inherits ci-test-review's discipline: classify recipe-vs-CI, create + verify fix PRs, never merge, never weaken a test.

It drives cc-ci over ssh cc-ci (cc-ci has abra, the recipe checkouts at ~/.abra/recipes/, the harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's /srv/cc-ci/.testenv creds. Plans/reports go to /srv/cc-ci/.cc-ci-logs/upgrades/.

Boundaries (same as ci-test-review): the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). Create + verify, NEVER merge (operator merges). Never weaken a test to make a PR green.

Arguments

• <recipe> — the recipe to upgrade (required).
• --with-tests (optional) — opt in to updating cc-ci tests. Without it (the DEFAULT, and what the weekly /upgrade-all cron uses), this skill only ever opens a recipe PR using the existing tests; if a test fails and is diagnosed as genuinely stale, it leaves a comment on the recipe PR explaining why and stops — it does not modify any test. With --with-tests, a stale-test failure additionally gets a verified cc-ci test-update PR (step 5b). A real upgrade regression is fixed + re-tested in both modes.

Procedure

⚠️ Run abra over a pseudo-TTY (or it FATAs inappropriate ioctl for device)

abra needs a TTY. Over plain ssh cc-ci 'abra …' there is none, so many abra commands die with FATA inappropriate ioctl for device / the input device is not a TTY. This is how the working harness avoids it (runner/harness/abra.py): wrap every abra call in a pseudo-TTY via util-linux script, and pass -n (--no-input) so it never prompts. Always invoke abra like this:

ssh cc-ci 'script -qec "abra <args> -n" /dev/null'

(git/other commands do NOT need the wrapper — only abra.) Do this for every abra command below.

1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)

⚠️ Two gotchas that silently skip recipes — handle BOTH before the version check

1. go-git auth to the private mirror. Recipes hosted on git.autonomic.zone (custom-html(-tiny), discourse, ghost, immich, lasuite-docs, plausible, uptime-kuma) fail with FATA unable to fetch tags … authentication required: Unauthorized because abra's embedded go-git reads remote.origin.url from .git/config literally and ignores git url.insteadOf / credential helpers. Fix: bake the bot creds into the origin URL first (idempotent): git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git" (only when origin is on git.autonomic.zone).
2. Dirty worktree is usually just the untracked cc-ci overlay (compose.ccci.yml) — don't skip on that. git stash -u before the checks and stash pop after.

Sync the mirror from TRUE upstream FIRST, then check for upgrades against that — so the check sees the real current recipe, not a stale mirror.

• (a) Reconcile the mirror from coopcloud (always — do this BEFORE the check). Run:

  set -a; . /srv/cc-ci/.testenv; set +a
  ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
     < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
  

  This fetches true upstream from coopcloud (git.coopcloud.tech/coop-cloud/<recipe>, default branch main OR master — every recipe has a coopcloud correspondent and none are forked) and force-syncs the recipe-maintainers/<recipe> mirror's main + tags to be identical to it. (The checkout's origin is the mirror, so the reconcile fetches coopcloud explicitly via an upstream remote — it does NOT sync the mirror to itself.) It also closes any open mirror PR already in upstream main.

• (b) Refresh the local checkout + check for upgrades (creds + stash per the box; each abra call pty-wrapped) — the checkout pulls from the now-coopcloud-synced mirror:

  ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
    url=$(git -C ~/.abra/recipes/$R config --get remote.origin.url); \
    case "$url" in https://git.autonomic.zone/*) \
      git -C ~/.abra/recipes/$R remote set-url origin "https://'"$GITEA_USERNAME"':'"$GITEA_PASSWORD"'@${url#https://}";; esac; \
    git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
    script -qec "abra recipe fetch $R --force -n" /dev/null; \
    git -C ~/.abra/recipes/$R fetch origin main; \
    script -qec "abra recipe versions $R -m -n" /dev/null; \
    script -qec "abra recipe upgrade $R -m -n" /dev/null; \
    git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
  

  Skip a recipe as SKIPPED — dirty-worktree only if it is still dirty with TRACKED changes after the stash (real local edits), not for the untracked overlay.

• No upgrades available → stop (status SKIPPED — up-to-date) — after the reconcile above.

• Check git log HEAD..origin/main and upstream PRs (git.coopcloud.tech/coop-cloud/<recipe>/pulls) — if someone already started the bump, re-plan from the tip of origin/main, not from scratch.

• Release-notes registry (discover once, reuse — parity with recipe-maintainer's upstream.md). Read cc-ci-plan/upstream/<recipe>.md — a per-image map of source repo + releases/changelog URL. If it's missing, or an image in compose.yml isn't covered (new/renamed service), discover the upstream project + releases page for each uncovered image (WebSearch the image → its GitHub/source repo → releases/changelog), write/update cc-ci-plan/upstream/<recipe>.md and commit it (git -C /srv/cc-ci-orch add cc-ci-plan/upstream/<recipe>.md && git -C /srv/cc-ci-orch commit -m "upstream(<recipe>): release-notes sources" && git -C /srv/cc-ci-orch push). Format + how it's used: cc-ci-plan/upstream/README.md.

• For each service with an upgrade, fetch the release notes from the registry URLs (WebFetch) between current → target versions, and call out breaking changes / required migrations / new-or- renamed config / dependency version requirements in an "Operator Action Required" section.

• Read the recipe's README (~/.abra/recipes/<recipe>/README.md, if present) for upgrade-specific instructions, migration steps, or breaking-change notes shipped with the recipe itself.

• Write the plan to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.)

2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)

On cc-ci's ~/.abra/recipes/<recipe> (wrap every abra call per the pseudo-TTY box above — ssh cc-ci 'script -qec "abra … -n" /dev/null'):

• script -qec "abra recipe upgrade <recipe> -n" /dev/null for the tags it can bump; hand-edit compose.yml for any tags it won't, plus the config/env/volume/label changes the plan calls for.
• tag+digest pins (image:tag@sha256:…) — abra can't bump these (it FATAs tag and digest … not supported), so do it by hand, and KEEP the digest pin: decide the new tag from the direct upstream-tag check (see /upgrade-all box item 4 — list the registry tags, pick the newest one the app version supports; do not blindly take the max). Then re-resolve the digest for the new tag and re-append it so the pin style is preserved (a digest is a reproducibility/security pin — never silently drop it): docker buildx imagetools inspect <image>:<newtag> --format '{{.Manifest.Digest}}' (or docker manifest inspect), then write image: <image>:<newtag>@<newdigest>. If the app and its digest-pinned sidecars must move together (e.g. immich-server ↔ its postgres:…-vectorchord… DB), bump them as a matched, compatibility-checked set per the plan.
• Bump the coop-cloud.${STACK_NAME}.version label to the new version string from the plan.
• script -qec "abra recipe lint <recipe> -C -n" /dev/null — fix obvious lint errors; if unfixable, record it and continue.
• Commit on a branch: git commit -m "chore: upgrade to <new-version>" (the commit message drives the PR branch name upgrade-<new-version>). Do not push to upstream; do not tag-push.

2b. Direct deploy + inspect on cc-ci — live feedback BEFORE CI (recipe-maintainer style)

Before opening the PR / running !testme, deploy the WIP recipe directly on the cc-ci server and watch it converge — the way recipe-maintainer tests on cctest. This gives you live logs + container state to actually SEE what the upgrade does (a failed migration, a crash-loop, a config gap), instead of only a pass/fail CI card. It is ADDITIONAL to — not a replacement for — the !testme verification + its 3-attempt budget (steps 4–5); it just front-loads the diagnosis so you arrive at CI already knowing the upgrade converges, and waste fewer of those 3 runs on basics.

Deploy the recipe head you just edited with --chaos (deploys the local WIP without a clean published commit), under a dedicated dev-<recipe> domain on the local swarm (--server default). Apply the usual gotchas (pseudo-TTY wrap every abra call; creds baked into the mirror origin; stash the untracked overlay). abra app new generates the .env + secrets:

ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; set -a; . /srv/cc-ci/.testenv; set +a; \
  R=<recipe>; D=dev-$R.ci.commoninternet.net; \
  script -qec "abra app new $R --server default --domain $D -n" /dev/null 2>&1 | tail -3; \
  script -qec "abra app deploy $D --chaos -n" /dev/null 2>&1 | tail -20'

• Inspect logs directly — prefer docker service logs over ssh (more reliable than abra app logs non-interactively, per recipe-maintainer references/recipe-maintainer/learnings.md): ssh cc-ci 'docker stack services ${D//./_}; docker service logs --tail 200 ${D//./_}_app' (swap _app for the failing service; pseudo-TTY wrap any abra app logs/abra app cmd).
• Iterate: edit the recipe → abra app deploy $D --chaos --force to cycle → re-read logs, until it converges, serves, and any migration/config behaves. This is where you debug the real upgrade with full visibility — fold what you learn back into the recipe edit (step 2).
• Tear down — MANDATORY, on every exit path (success, RED, or abort — shared swarm): the dev deploy is yours and you MUST remove it before you finish this recipe / move to the next one. Treat it as a finally:

  ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; D=dev-<recipe>.ci.commoninternet.net; \
    script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
    script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
  

  Then verify nothing leaked: ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe with . as _>"' should print 0 (and no dev-<recipe>_* volumes remain). If the recipe failed, tear the dev deploy down anyway, THEN report the failure — never leave it running. Backstop (defence-in-depth, NOT a substitute for the explicit teardown above): /upgrade-all runs the dev-* reaper (reap-dev-deploys.sh) + the orphan-sweep at the START and END of every run, so a crashed/abandoned loop's dev- deploy is cleared by the next run — but you must still clean up yourself.
• Caveats: shared swarm — keep to ONE dev-<recipe> instance at a time and tear it down before the next recipe; the dev-<recipe> domain is distinct from the harness's per-run domains and from the warm-* canonicals, so the sweep removes a leaked one without touching live services.
• 🚫 This loop deploys + inspects ONLY — it must NEVER modify the cc-ci tests or the harness. You have host access here, but the cc-ci tests/** and runner/** are the independent gate, not yours to touch: do not edit a test/overlay to make a dev- deploy or a !testme go green. Editing cc-ci tests is gated exclusively by --with-tests (step 5b); in DEFAULT mode a stale test gets a PR comment, never an edit. Fix the recipe, never the gate. (Interim rule while the upgrader still runs on the cc-ci host — see cc-ci-plan/IDEAS.md, "Move the upgrader's direct deploy onto separate infra".)

3. Open the recipe PR (never merge)

set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh

This (a) re-syncs mirror main to true upstream main, (b) closes any open PR already merged upstream (merging it would be a no-op), then handles the upgrade PR: if an open upgrade PR already exists for this recipe (branch upgrade-*), the new work is pushed onto THAT PR's branch and the PR is updated + re-tested — one evolving upgrade PR per recipe, not a second parallel one. Only if none exists is a fresh upgrade-<version> PR opened. Unrelated open PRs (e.g. a backup fix) are NOT closed or touched — the operator decides on those. The PR's diff is exactly this upgrade against the real current upstream main. Capture the PR_URL. Optionally export RECIPE_PR_BODY first (image-tag table + the planned operator-action notes). Re-running with the same target version updates the existing same-branch PR rather than duplicating it.

4. VERIFY by running !testme ON THE PR (results visible in the PR; iterate ≤3×)

Trigger the real CI on the recipe PR by posting !testme — the bridge runs the harness on cc-ci and posts the run + its results back to the PR, so the operator sees them in the PR. Use the PR index from step 3's PR_URL (.../pulls/<N>):

/srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>     # POST=1: posts !testme + polls
# if it prints VERDICT=PENDING, poll again WITHOUT re-triggering:
POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>

• GREEN → the recipe PR is verified on cc-ci, results in the PR, ready for operator merge. Done.
• RED → go to step 5. You get up to 3 !testme runs total on this PR (initial + 2 retries): use them to fix and re-test a real upgrade regression (step 5a). Each retry is a fresh !testme after you push a fix, so every attempt's result is recorded in the PR.

5. If !testme is RED → diagnose from the run (recipe regression vs stale cc-ci TEST)

Read the failing run (linked from the PR / the harness log on cc-ci) and classify:

5a. The UPGRADE itself is broken (real regression: bad tag, missing migration, config gap) — fix it on the recipe branch, push, and re-run !testme (within the 3-run budget). The recipe PR is only "working" once !testme is GREEN. If still red after 3 runs, leave the PR open (its red !testme results stand as the evidence) and report FAILED — upgrade not green after 3 !testme runs.

5b. The upgrade is correct but a cc-ci TEST is genuinely stale/wrong for the new version (asserts old behavior, an overlay needs updating, a readiness gate changed). !testme can't go green without a test change, and a test change is gated by --with-tests:

• DEFAULT (no --with-tests) — comment, don't touch tests. Post a comment on the recipe PR that: states the upgrade itself looks correct; identifies the specific test that failed and why it appears stale (what the new version changed vs what the test asserts); and tells the operator they can re-run /recipe-upgrade <recipe> --with-tests to also open + verify a cc-ci test-update PR. Do NOT modify any test. Report SUCCESS-PENDING-TESTS (recipe PR open; !testme red on a stale test; operator to decide).
• --with-tests — open + verify a cc-ci test PR. Make it the ci-test-review way:
  1. Branch recipe-maintainers/cc-ci in a separate clone (single-writer: never push main, never touch the build loops' /cc-ci /cc-ci-adv clones); update the test/overlay.
  2. Verify the recipe upgrade WITH the updated test applied. !testme on the recipe PR uses the deployed/main cc-ci tests, so it can't see the unmerged test change — verify this combo by running the harness with the cc-ci branch checked out on cc-ci: RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh (after checking out the cc-ci test branch on cc-ci + rebuilding if needed), plus a small regression sample. Green ⇔ the upgrade passes under the corrected test.
  3. Open the cc-ci test PR via the Gitea API. Note in BOTH PRs that they are a dependent pair (the recipe PR's !testme goes green only once the test PR is merged). Report SUCCESS+TESTPR.

FLAKY (passes on a re-!testme) → note it; don't author a fix for a flake.

Never weaken a test to turn a red upgrade green.

6. Report — single parseable RESULT line + summary

Write the report to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md and print, as the last line, one of these exact prefixes (so /upgrade-all can collect it):

• RESULT: SUCCESS — <recipe> <old> → <new>, !testme GREEN, recipe PR: <url>
• RESULT: SUCCESS-PENDING-TESTS — <recipe> <old> → <new>, recipe PR: <url>; !testme RED on a stale test (commented; re-run --with-tests to update tests)
• RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>; recipe PR: <url>; cc-ci test PR: <url> (verified with the test change; pair)
• RESULT: FAILED — <recipe> at <step>: <one-line reason> (e.g. upgrade not green after 3 !testme runs)
• RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>

Always state explicitly that nothing was merged — the PR(s) await operator review.

Guardrails

• cc-ci is the gate, via !testme on the PR — the harness decides pass/fail and the results live in the PR (operator-visible); AI plans, implements, diagnoses. Up to 3 !testme runs per PR.
• Create + verify, NEVER merge. Recipe PR (and any cc-ci test PR) are operator-merged after review.
• Tests are opt-in. Default = recipe PR only, existing tests; a stale-test failure gets an explanatory PR comment, never a test edit. Only --with-tests opens a cc-ci test PR. The weekly /upgrade-all cron always runs the default — it never auto-updates tests.
• 🚫 NEVER modify the cc-ci tests or harness (tests/**, runner/**) unless explicitly invoked with --with-tests. This is absolute and mode-gated — it holds in every step, including the step-2b direct-deploy loop where you have cc-ci host access. The tests are the independent gate; never edit them to turn a dev- deploy or !testme green. In DEFAULT mode, fix the recipe or leave a PR comment — never touch the gate. (Interim safeguard while the upgrader runs on the cc-ci host; see IDEAS.md.)
• Mirror reflects reality. Each run force-syncs the recipe-maintainers/<recipe> mirror main to true upstream main and closes open PRs already merged upstream. It does NOT close superseded/unrelated open PRs. An existing open upgrade PR is extended (new work pushed onto its branch + re-tested) rather than spawning a second upgrade PR — one evolving upgrade PR per recipe.
• Prefer a recipe-only PR. Only open a cc-ci test PR (under --with-tests) when the upgrade is correct but a test is genuinely stale — never to paper over a real upgrade regression.
• Never weaken a test; bounded changes (the upgrade + minimal test update, not rewrites).
• Real abra path throughout (abra recipe upgrade / abra app deploy via the harness).
• Single-writer / coordination: dedicated branches + separate clones; never push main or disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). See ci-test-review for the shared rules.