cc-ci-orchestrator/.claude/skills/recipe-upgrade/SKILL.md

---
name: recipe-upgrade
description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades — fetching each image's release notes via a persisted per-recipe registry (cc-ci-plan/upstream/<recipe>.md) + the recipe README — plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), opens a recipe PR, and verifies it by posting `!testme` on the PR (real CI; results visible in the PR; iterates up to 3×). DEFAULT: recipe PR only, using existing tests — if a test fails because it is genuinely stale, it leaves an explanatory COMMENT on the PR for the operator (does NOT touch tests). With `--with-tests`: also opens + verifies a PR to update the stale cc-ci test. NEVER merges. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade <recipe> [--with-tests].
---

# recipe-upgrade

Autonomous, end-to-end upgrade of **one** recipe, **gated by the cc-ci CI server**. This is the
cc-ci analogue of recipe-maintainer's `/recipe-upgrade-full`, but the test gate is **cc-ci** (the real
recipe CI), and it inherits `ci-test-review`'s discipline: classify recipe-vs-CI, create + **verify**
fix PRs, **never merge**, never weaken a test.

It drives cc-ci over `ssh cc-ci` (cc-ci has abra, the recipe checkouts at `~/.abra/recipes/`, the
harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's
`/srv/cc-ci/.testenv` creds. Plans/reports go to `/srv/cc-ci/.cc-ci-logs/upgrades/`.

**Boundaries (same as ci-test-review):** the test execution + PR verification are deterministic (the
cc-ci harness decides pass/fail). **Create + verify, NEVER merge** (operator merges). **Never weaken a
test** to make a PR green.

## Arguments
- `<recipe>` — the recipe to upgrade (required).
- `--with-tests` (optional) — **opt in to updating cc-ci tests.** Without it (the DEFAULT, and what the
  weekly `/upgrade-all` cron uses), this skill **only ever opens a recipe PR using the existing tests**;
  if a test fails and is diagnosed as genuinely stale, it **leaves a comment** on the recipe PR
  explaining why and stops — it does **not** modify any test. With `--with-tests`, a stale-test failure
  additionally gets a **verified cc-ci test-update PR** (step 5b). A real upgrade regression is fixed +
  re-tested in **both** modes.

## Procedure

> ### ⚠️ Run abra over a pseudo-TTY (or it FATAs `inappropriate ioctl for device`)
> `abra` needs a TTY. Over plain `ssh cc-ci 'abra …'` there is none, so many abra commands die with
> **`FATA inappropriate ioctl for device`** / `the input device is not a TTY`. This is how the working
> harness avoids it (`runner/harness/abra.py`): wrap every abra call in a pseudo-TTY via util-linux
> `script`, and pass `-n` (`--no-input`) so it never prompts. **Always invoke abra like this:**
> ```
> ssh cc-ci 'script -qec "abra <args> -n" /dev/null'
> ```
> (`git`/other commands do NOT need the wrapper — only `abra`.) Do this for every abra command below.

### 1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)

> ### ⚠️ Two gotchas that silently skip recipes — handle BOTH before the version check
> 1. **go-git auth to the private mirror.** Recipes hosted on `git.autonomic.zone` (custom-html(-tiny),
>    discourse, ghost, immich, lasuite-docs, plausible, uptime-kuma) fail with
>    **`FATA unable to fetch tags … authentication required: Unauthorized`** because abra's embedded
>    go-git reads `remote.origin.url` from `.git/config` **literally** and **ignores `git url.insteadOf`
>    / credential helpers**. Fix: bake the bot creds into the origin URL first (idempotent):
>    `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"` (only when origin is on git.autonomic.zone).
> 2. **Dirty worktree** is usually just the untracked cc-ci overlay (`compose.ccci.yml`) — don't skip on
>    that. `git stash -u` before the checks and `stash pop` after.

**Sync the mirror from TRUE upstream FIRST, then check for upgrades against that** — so the check sees
the real current recipe, not a stale mirror.

- **(a) Reconcile the mirror from coopcloud (always — do this BEFORE the check).** Run:
  ```
  set -a; . /srv/cc-ci/.testenv; set +a
  ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
     < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
  ```
  This fetches **true upstream from coopcloud** (`git.coopcloud.tech/coop-cloud/<recipe>`, default branch
  main OR master — every recipe has a coopcloud correspondent and none are forked) and force-syncs the
  `recipe-maintainers/<recipe>` mirror's `main` + tags to be **identical to it**. (The checkout's
  `origin` is the mirror, so the reconcile fetches coopcloud *explicitly* via an `upstream` remote — it
  does NOT sync the mirror to itself.) It also **closes any open mirror PR already in upstream main**.

- **(b) Refresh the local checkout + check for upgrades** (creds + stash per the box; each abra call
  pty-wrapped) — the checkout pulls from the now-coopcloud-synced mirror:
  ```
  ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
    url=$(git -C ~/.abra/recipes/$R config --get remote.origin.url); \
    case "$url" in https://git.autonomic.zone/*) \
      git -C ~/.abra/recipes/$R remote set-url origin "https://'"$GITEA_USERNAME"':'"$GITEA_PASSWORD"'@${url#https://}";; esac; \
    git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
    script -qec "abra recipe fetch $R --force -n" /dev/null; \
    git -C ~/.abra/recipes/$R fetch origin main; \
    script -qec "abra recipe versions $R -m -n" /dev/null; \
    script -qec "abra recipe upgrade $R -m -n" /dev/null; \
    git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
  ```
  Skip a recipe as `SKIPPED — dirty-worktree` **only if it is still dirty with TRACKED changes after the
  stash** (real local edits), not for the untracked overlay.
- **No upgrades available → stop** (status `SKIPPED — up-to-date`) — after the reconcile above.
- Check `git log HEAD..origin/main` and upstream PRs (`git.coopcloud.tech/coop-cloud/<recipe>/pulls`)
  — if someone already started the bump, **re-plan from the tip of `origin/main`**, not from scratch.
- **Release-notes registry (discover once, reuse — parity with recipe-maintainer's `upstream.md`).**
  Read `cc-ci-plan/upstream/<recipe>.md` — a per-image map of source repo + releases/changelog URL. If
  it's missing, or an image in `compose.yml` isn't covered (new/renamed service), discover the upstream
  project + releases page for each uncovered image (WebSearch the image → its GitHub/source repo →
  releases/changelog), **write/update** `cc-ci-plan/upstream/<recipe>.md` and commit it
  (`git -C /srv/cc-ci-orch add cc-ci-plan/upstream/<recipe>.md && git -C /srv/cc-ci-orch commit -m "upstream(<recipe>): release-notes sources" && git -C /srv/cc-ci-orch push`).
  Format + how it's used: `cc-ci-plan/upstream/README.md`.
- **For each service with an upgrade, fetch the release notes from the registry URLs** (WebFetch)
  between current → target versions, and call out **breaking changes / required migrations / new-or-
  renamed config / dependency version requirements** in an "Operator Action Required" section.
- **Read the recipe's README** (`~/.abra/recipes/<recipe>/README.md`, if present) for upgrade-specific
  instructions, migration steps, or breaking-change notes shipped with the recipe itself.
- Write the plan to `/srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md`: goal, image
  tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/
  config changes, risks. (No human review gate — proceed straight to implement.)

### 2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)
On cc-ci's `~/.abra/recipes/<recipe>` (wrap every abra call per the pseudo-TTY box above —
`ssh cc-ci 'script -qec "abra … -n" /dev/null'`):
- `script -qec "abra recipe upgrade <recipe> -n" /dev/null` for the tags it can bump; hand-edit
  `compose.yml` for any tags it won't, plus the config/env/volume/label changes the plan calls for.
- **tag+digest pins (`image:tag@sha256:…`) — abra can't bump these (it FATAs `tag and digest … not
  supported`), so do it by hand, and KEEP the digest pin:** decide the new tag from the direct
  upstream-tag check (see `/upgrade-all` box item 4 — list the registry tags, pick the newest one the
  app version supports; do not blindly take the max). Then **re-resolve the digest for the new tag** and
  re-append it so the pin style is preserved (a digest is a reproducibility/security pin — never silently
  drop it): `docker buildx imagetools inspect <image>:<newtag> --format '{{.Manifest.Digest}}'` (or
  `docker manifest inspect`), then write `image: <image>:<newtag>@<newdigest>`. If the app and its
  digest-pinned sidecars must move together (e.g. immich-server ↔ its `postgres:…-vectorchord…` DB),
  bump them as a matched, compatibility-checked set per the plan.
- Bump the `coop-cloud.${STACK_NAME}.version` label to the new version string from the plan.
- `script -qec "abra recipe lint <recipe> -C -n" /dev/null` — fix obvious lint errors; if unfixable,
  record it and continue.
- Commit on a branch: `git commit -m "chore: upgrade to <new-version>"` (the commit message drives the
  PR branch name `upgrade-<new-version>`). Do **not** push to upstream; do **not** tag-push.

### 2b. Direct deploy + inspect on cc-ci — live feedback BEFORE CI (recipe-maintainer style)
Before opening the PR / running `!testme`, deploy the WIP recipe **directly** on the cc-ci server and
watch it converge — the way recipe-maintainer tests on `cctest`. This gives you **live logs +
container state** to actually SEE what the upgrade does (a failed migration, a crash-loop, a config
gap), instead of only a pass/fail CI card. It is **ADDITIONAL to — not a replacement for** — the
`!testme` verification + its 3-attempt budget (steps 4–5); it just front-loads the diagnosis so you
arrive at CI already knowing the upgrade converges, and waste fewer of those 3 runs on basics.

Deploy the recipe head you just edited with **`--chaos`** (deploys the local WIP without a clean
published commit), under a dedicated `dev-<recipe>` domain on the local swarm (`--server default`).
Apply the usual gotchas (pseudo-TTY wrap every abra call; creds baked into the mirror origin; stash
the untracked overlay). `abra app new` generates the `.env` + secrets:
```
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; set -a; . /srv/cc-ci/.testenv; set +a; \
  R=<recipe>; D=dev-$R.ci.commoninternet.net; \
  script -qec "abra app new $R --server default --domain $D -n" /dev/null 2>&1 | tail -3; \
  script -qec "abra app deploy $D --chaos -n" /dev/null 2>&1 | tail -20'
```
- **Inspect logs directly** — prefer `docker service logs` over ssh (more reliable than `abra app
  logs` non-interactively, per recipe-maintainer `references/recipe-maintainer/learnings.md`):
  `ssh cc-ci 'docker stack services ${D//./_}; docker service logs --tail 200 ${D//./_}_app'`
  (swap `_app` for the failing service; pseudo-TTY wrap any `abra app logs`/`abra app cmd`).
- **Iterate:** edit the recipe → `abra app deploy $D --chaos --force` to cycle → re-read logs, until it
  converges, serves, and any migration/config behaves. This is where you debug the real upgrade with
  full visibility — fold what you learn back into the recipe edit (step 2).
- **Tear down — MANDATORY, on every exit path** (success, RED, or abort — shared swarm): the dev
  deploy is yours and you MUST remove it before you finish this recipe / move to the next one. Treat
  it as a `finally`:
  ```
  ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; D=dev-<recipe>.ci.commoninternet.net; \
    script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
    script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
  ```
  Then **verify nothing leaked**: `ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe with . as _>"'`
  should print `0` (and no `dev-<recipe>_*` volumes remain). If the recipe failed, tear the dev deploy
  down anyway, THEN report the failure — never leave it running.
  Backstop (defence-in-depth, NOT a substitute for the explicit teardown above): `/upgrade-all` runs
  the `dev-*` reaper (`reap-dev-deploys.sh`) + the orphan-sweep at the START and END of every run, so a
  crashed/abandoned loop's `dev-` deploy is cleared by the next run — but you must still clean up yourself.
- Caveats: shared swarm — keep to **ONE** `dev-<recipe>` instance at a time and tear it down before the
  next recipe; the `dev-<recipe>` domain is distinct from the harness's per-run domains and from the
  `warm-*` canonicals, so the sweep removes a leaked one without touching live services.
- 🚫 **This loop deploys + inspects ONLY — it must NEVER modify the cc-ci tests or the harness.** You
  have host access here, but the cc-ci `tests/**` and `runner/**` are the independent gate, not yours to
  touch: do not edit a test/overlay to make a `dev-` deploy or a `!testme` go green. Editing cc-ci tests
  is gated **exclusively** by `--with-tests` (step 5b); in DEFAULT mode a stale test gets a PR **comment**,
  never an edit. Fix the *recipe*, never the gate. (Interim rule while the upgrader still runs on the
  cc-ci host — see `cc-ci-plan/IDEAS.md`, "Move the upgrader's direct deploy onto separate infra".)

### 3. Open the recipe PR (never merge)
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
```
This (a) re-syncs mirror `main` to true upstream main, (b) **closes any open PR already merged
upstream** (merging it would be a no-op), then handles the upgrade PR: **if an open upgrade PR already
exists for this recipe (branch `upgrade-*`), the new work is pushed onto THAT PR's branch and the PR is
updated + re-tested** — one evolving upgrade PR per recipe, not a second parallel one. Only if none
exists is a fresh `upgrade-<version>` PR opened. Unrelated open PRs (e.g. a backup fix) are **NOT**
closed or touched — the operator decides on those. The PR's diff is exactly this upgrade against the
real current upstream main. Capture the `PR_URL`. Optionally export `RECIPE_PR_BODY` first (image-tag
table + the planned operator-action notes). Re-running with the same target version updates the
existing same-branch PR rather than duplicating it.

### 4. VERIFY by running `!testme` ON THE PR (results visible in the PR; iterate ≤3×)
Trigger the **real** CI on the recipe PR by posting `!testme` — the bridge runs the harness on cc-ci
and posts the run + its results back to the PR, so the operator sees them in the PR. Use the PR index
from step 3's `PR_URL` (`.../pulls/<N>`):
```
/srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>     # POST=1: posts !testme + polls
# if it prints VERDICT=PENDING, poll again WITHOUT re-triggering:
POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>
```
- **GREEN** → the recipe PR is verified on cc-ci, results in the PR, ready for operator merge. Done.
- **RED** → go to step 5. You get **up to 3 `!testme` runs total** on this PR (initial + 2 retries):
  use them to fix and re-test a real upgrade regression (step 5a). Each retry is a fresh `!testme`
  after you push a fix, so every attempt's result is recorded in the PR.

### 5. If `!testme` is RED → diagnose from the run (recipe regression vs stale cc-ci TEST)
Read the failing run (linked from the PR / the harness log on cc-ci) and classify:

**5a. The UPGRADE itself is broken** (real regression: bad tag, missing migration, config gap) — fix
it on the recipe branch, push, and re-run `!testme` (within the 3-run budget). The recipe PR is only
"working" once `!testme` is GREEN. If still red after 3 runs, leave the PR open (its red `!testme`
results stand as the evidence) and report `FAILED — upgrade not green after 3 !testme runs`.

**5b. The upgrade is correct but a cc-ci TEST is genuinely stale/wrong** for the new version (asserts
old behavior, an overlay needs updating, a readiness gate changed). `!testme` can't go green without a
test change, and a test change is **gated by `--with-tests`**:

- **DEFAULT (no `--with-tests`) — comment, don't touch tests.** Post a **comment on the recipe PR**
  that: states the upgrade itself looks correct; identifies the specific test that failed and **why it
  appears stale** (what the new version changed vs what the test asserts); and tells the operator they
  can **re-run `/recipe-upgrade <recipe> --with-tests`** to also open + verify a cc-ci test-update PR.
  Do **NOT** modify any test. Report `SUCCESS-PENDING-TESTS` (recipe PR open; `!testme` red on a
  stale test; operator to decide).
- **`--with-tests` — open + verify a cc-ci test PR.** Make it the `ci-test-review` way:
  1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
     never touch the build loops' `/cc-ci` `/cc-ci-adv` clones); update the test/overlay.
  2. **Verify the recipe upgrade WITH the updated test applied.** `!testme` on the recipe PR uses the
     *deployed/main* cc-ci tests, so it can't see the unmerged test change — verify this combo by
     running the harness with the cc-ci branch checked out on cc-ci:
     `RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh`
     (after checking out the cc-ci test branch on cc-ci + rebuilding if needed), plus a small
     regression sample. Green ⇔ the upgrade passes under the corrected test.
  3. Open the **cc-ci test PR** via the Gitea API. Note in BOTH PRs that they are a dependent pair
     (the recipe PR's `!testme` goes green only once the test PR is merged). Report `SUCCESS+TESTPR`.

**FLAKY** (passes on a re-`!testme`) → note it; don't author a fix for a flake.

Never weaken a test to turn a red upgrade green.

### 6. Report — single parseable RESULT line + summary
Write the report to `/srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md` and print, as
the **last line**, one of these exact prefixes (so `/upgrade-all` can collect it):

- `RESULT: SUCCESS — <recipe> <old> → <new>, !testme GREEN, recipe PR: <url>`
- `RESULT: SUCCESS-PENDING-TESTS — <recipe> <old> → <new>, recipe PR: <url>; !testme RED on a stale test (commented; re-run --with-tests to update tests)`
- `RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>; recipe PR: <url>; cc-ci test PR: <url> (verified with the test change; pair)`
- `RESULT: FAILED — <recipe> at <step>: <one-line reason>` (e.g. upgrade not green after 3 !testme runs)
- `RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>`

Always state explicitly that **nothing was merged** — the PR(s) await operator review.

## Guardrails
- **cc-ci is the gate, via `!testme` on the PR** — the harness decides pass/fail and the results live
  in the PR (operator-visible); AI plans, implements, diagnoses. Up to 3 `!testme` runs per PR.
- **Create + verify, NEVER merge.** Recipe PR (and any cc-ci test PR) are operator-merged after review.
- **Tests are opt-in.** Default = recipe PR only, existing tests; a stale-test failure gets an
  explanatory PR **comment**, never a test edit. Only `--with-tests` opens a cc-ci test PR. The weekly
  `/upgrade-all` cron always runs the **default** — it never auto-updates tests.
- 🚫 **NEVER modify the cc-ci tests or harness (`tests/**`, `runner/**`) unless explicitly invoked with
  `--with-tests`.** This is absolute and mode-gated — it holds in every step, including the step-2b
  direct-deploy loop where you have cc-ci host access. The tests are the independent gate; never edit
  them to turn a `dev-` deploy or `!testme` green. In DEFAULT mode, fix the recipe or leave a PR comment
  — never touch the gate. (Interim safeguard while the upgrader runs on the cc-ci host; see `IDEAS.md`.)
- **Mirror reflects reality.** Each run force-syncs the `recipe-maintainers/<recipe>` mirror `main` to
  true upstream main and closes open PRs already merged upstream. It does NOT close superseded/unrelated
  open PRs. **An existing open upgrade PR is extended (new work pushed onto its branch + re-tested)
  rather than spawning a second upgrade PR** — one evolving upgrade PR per recipe.
- **Prefer a recipe-only PR.** Only open a cc-ci test PR (under `--with-tests`) when the upgrade is
  correct but a test is genuinely stale — never to paper over a real upgrade regression.
- **Never weaken a test**; **bounded** changes (the upgrade + minimal test update, not rewrites).
- **Real abra path** throughout (`abra recipe upgrade` / `abra app deploy` via the harness).
- **Single-writer / coordination:** dedicated branches + separate clones; never push `main` or disturb
  the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have
  effectively-exclusive use of the host (or serialize). See `ci-test-review` for the shared rules.