recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add /recipe-upgrade + /upgrade-all skills (cc-ci-gated upgrades, never merge)

Browse Source 
Per-recipe and fleet-wide upgrade skills modelled on recipe-maintainer's
recipe-upgrade-full / recipe-upgrade-cron-all, but gated by the cc-ci CI server
and inheriting ci-test-review's create+verify+never-merge discipline.

- recipe-upgrade/: plan (release notes, breaking changes) -> implement (abra
  recipe upgrade + version bump + config, lint) -> open the recipe PR -> VERIFY
  green on cc-ci (full suite cold against the PR head via verify-pr.sh). If the
  upgrade is correct but a cc-ci TEST went stale, also update the test, verify
  it, and open a second PR to recipe-maintainers/cc-ci. Never merges; never
  weakens a test; prefers a recipe-only PR. Emits a parseable RESULT line.
  + open-recipe-pr.sh: adapted recipe-create-pr; runs on cc-ci (has the recipe
    checkout + bot token), creds passed from the orchestrator .testenv;
    force-syncs the mirror main so the PR diff is exactly the upgrade.
- upgrade-all/: weekly fan-out — enumerate enrolled recipes, survey upgrades,
  run /recipe-upgrade per upgradeable recipe via subagent (sequential default,
  --parallel / --dry-run), collect into one PR-list summary. Coordination +
  single-writer + shared-Swarm-teardown guardrails; built for a weekly cron.
- ci-test-review/verify-pr.sh: pass SRC (recipe-maintainers/<recipe>) alongside
  REF so the harness clones the mirror PR head correctly (its real contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
notplants
2026-05-29 17:19:20 +01:00
co-authored by Claude Opus 4.8
parent 27480b3513
commit db31c08d6a
4 changed files with 303 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/recipe-upgrade/SKILL.md
+112
View File
@@ -0,0 +1,112 @@
---
name: recipe-upgrade
description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), then VERIFIES the change green on cc-ci (full suite, cold, against the PR head) and opens a recipe PR. If the upgrade is correct but a cc-ci TEST is now stale, it also updates the test, verifies that, and opens a second PR to the cc-ci repo. NEVER merges — leaves verified, ready-to-merge PRs. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade <recipe>.
---
# recipe-upgrade
Autonomous, end-to-end upgrade of **one** recipe, **gated by the cc-ci CI server**. This is the
cc-ci analogue of recipe-maintainer's `/recipe-upgrade-full`, but the test gate is **cc-ci** (the real
recipe CI), and it inherits `ci-test-review`'s discipline: classify recipe-vs-CI, create + **verify**
fix PRs, **never merge**, never weaken a test.
It drives cc-ci over `ssh cc-ci` (cc-ci has abra, the recipe checkouts at `~/.abra/recipes/`, the
harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's
`/srv/cc-ci/.testenv` creds. Plans/reports go to `/srv/cc-ci/.cc-ci-logs/upgrades/`.
**Boundaries (same as ci-test-review):** the test execution + PR verification are deterministic (the
cc-ci harness decides pass/fail). **Create + verify, NEVER merge** (operator merges). **Never weaken a
test** to make a PR green.
## Procedure
### 1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)
On cc-ci, refresh the recipe and see what's available:
```
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \
git -C ~/.abra/recipes/<recipe> status --short; \
abra recipe fetch <recipe> --force; \
git -C ~/.abra/recipes/<recipe> fetch origin main; \
abra recipe versions <recipe> -m; \
abra recipe upgrade <recipe> -m -n'
```
- **Dirty worktree on cc-ci → abort this recipe** (status `SKIPPED — dirty-worktree`); don't prompt.
- **No upgrades available → stop** (status `SKIPPED — up-to-date`).
- Check `git log HEAD..origin/main` and upstream PRs (`git.coopcloud.tech/coop-cloud/<recipe>/pulls`)
— if someone already started the bump, **re-plan from the tip of `origin/main`**, not from scratch.
- For each service with an upgrade, fetch upstream **release notes** (WebFetch) between current and
target versions and call out **breaking changes / required migrations / new-or-renamed config /
dependency bumps** in an "Operator Action Required" section.
- Write the plan to `/srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md`: goal, image
tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/
config changes, risks. (No human review gate — proceed straight to implement.)
### 2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)
On cc-ci's `~/.abra/recipes/<recipe>`:
- `abra recipe upgrade <recipe> -n` for the tags it can bump; hand-edit `compose.yml` for any tags it
won't, plus the config/env/volume/label changes the plan calls for.
- Bump the `coop-cloud.${STACK_NAME}.version` label to the new version string from the plan.
- `abra recipe lint <recipe> -C` — fix obvious lint errors; if unfixable, record it and continue.
- Commit on a branch: `git commit -m "chore: upgrade to <new-version>"` (the commit message drives the
PR branch name `upgrade-<new-version>`). Do **not** push to upstream; do **not** tag-push.
### 3. Open the recipe PR (never merge)
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
```
This pushes the upgrade branch to `recipe-maintainers/<recipe>` (force-syncing mirror `main` from
upstream so the diff is exactly the upgrade) and opens the PR. Capture the `PR_URL`. Optionally export
`RECIPE_PR_BODY` first (image-tag table + the planned operator-action notes).
### 4. VERIFY the upgrade on the CI server (the gate)
Run the cc-ci full suite **cold** against the PR head — the dogfood gate from `ci-test-review`:
```
RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh
```
(`SRC` defaults to `recipe-maintainers/<recipe>`.) **GREEN** ⇔ harness exits 0 → the recipe PR is
verified, ready for operator merge. **General bar = one cold green**; use `REPEAT=3` only for a recipe
already known to be FLAKY (e.g. lasuite-drive). Record the run summary in the report.
### 5. If verification is RED → diagnose (recipe vs cc-ci TEST), per ci-test-review
Read the harness log and classify the failure:
- **The UPGRADE itself is broken** (real regression: bad tag, missing migration, config gap) → fix it
on the recipe branch (bounded — ≤3 iterations), re-push, re-verify. The recipe PR is only "working"
once cc-ci is green. If still red after the budget, leave the PR open and report
`FAILED — upgrade not green`.
- **The upgrade is correct but a cc-ci TEST is now stale/wrong** for the new version (asserts old
behavior, an overlay needs updating, a readiness gate changed) → this is a **CI-server fix**. Make
it the `ci-test-review` way:
1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`,
never touch the build loops' `/cc-ci` `/cc-ci-adv` clones), update the test/overlay.
2. **Verify** the recipe upgrade **with the updated test applied**: check the cc-ci branch out on
cc-ci, rebuild if needed, re-run `verify-pr.sh` for the recipe + a small regression sample.
3. Open the **cc-ci test PR** via the Gitea API (`/srv/cc-ci/.testenv` creds). Capture its URL.
4. The recipe PR + the cc-ci test PR are a **pair** — note in each that it depends on the other.
- **FLAKY** (passes on a re-run) → note it; don't author a fix for a flake.
Never weaken a test to turn a red upgrade green.
### 6. Report — single parseable RESULT line + summary
Write the report to `/srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md` and print, as
the **last line**, one of these exact prefixes (so `/upgrade-all` can collect it):
- `RESULT: SUCCESS — <recipe> <old> → <new>, cc-ci GREEN, recipe PR: <url>`
- `RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>, cc-ci GREEN; recipe PR: <url>; cc-ci test PR: <url>`
- `RESULT: FAILED — <recipe> at <step>: <one-line reason>` (e.g. upgrade not green after 3 tries)
- `RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>`
Always state explicitly that **nothing was merged** — the PR(s) await operator review.
## Guardrails
- **cc-ci is the gate** — deterministic harness decides pass/fail; AI plans, implements, diagnoses.
- **Create + verify, NEVER merge.** Recipe PR (and any cc-ci test PR) are operator-merged after review.
- **Prefer a recipe-only PR.** Only open a cc-ci test PR when the upgrade is correct but a test is
genuinely stale — not to paper over a real upgrade regression.
- **Never weaken a test**; **bounded** changes (the upgrade + minimal test update, not rewrites).
- **Real abra path** throughout (`abra recipe upgrade` / `abra app deploy` via the harness).
- **Single-writer / coordination:** dedicated branches + separate clones; never push `main` or disturb
the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have
effectively-exclusive use of the host (or serialize). See `ci-test-review` for the shared rules.
.claude/skills/recipe-upgrade/open-recipe-pr.sh
Executable
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
# recipe-upgrade :: open a recipe PR on git.autonomic.zone (adapted from
# recipe-maintainer's recipe-create-pr for the cc-ci context).
# -------------------------------------------------------------------------
# RUNS ON cc-ci (it has the recipe checkout at ~/.abra/recipes/<recipe>).
# Invoke from the orchestrator, feeding creds + the script over ssh:
#
# set -a; . /srv/cc-ci/.testenv; set +a
# ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' \
# GITEA_URL='$GITEA_URL' bash -s <recipe>" \
# < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
#
# Preconditions on cc-ci: ~/.abra/recipes/<recipe> is a git checkout with the
# upgrade commit(s) on HEAD beyond origin/main (the apply step made them).
# Force-syncs the gitea mirror's main from upstream origin/main so the PR diff
# is exactly the upgrade. Prints the PR URL. Never merges.
set -o errexit -o nounset -o pipefail
RECIPE="${1:?usage: open-recipe-pr.sh <recipe>}"
: "${GITEA_USERNAME:?missing GITEA_USERNAME (pass via ssh env)}"
: "${GITEA_PASSWORD:?missing GITEA_PASSWORD}"
: "${GITEA_URL:?missing GITEA_URL}"
NAMESPACE="${GITEA_NAMESPACE:-recipe-maintainers}"
RECIPE_DIR="${HOME}/.abra/recipes/${RECIPE}"
export PATH="/run/current-system/sw/bin:${PATH}"
PASS_ENC=$(python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1],safe=''))" "${GITEA_PASSWORD}")
API="https://${GITEA_URL}/api/v1"
AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
[ -d "${RECIPE_DIR}/.git" ] || { echo "ERROR: ${RECIPE_DIR} is not a git repo (run 'abra recipe fetch ${RECIPE}')"; exit 1; }
cd "${RECIPE_DIR}"
echo "→ Fetching upstream main..."
git fetch origin main
DIVERGED=$(git log --oneline origin/main..HEAD 2>/dev/null || true)
[ -n "${DIVERGED}" ] || { echo "ERROR: HEAD has no commits beyond origin/main. Nothing to PR."; exit 1; }
echo "→ Local commits to PR:"; echo "${DIVERGED}" | sed 's/^/ /'
LATEST_MSG=$(git log -1 --pretty=%s HEAD)
if echo "${LATEST_MSG}" | grep -qiE "upgrade to [0-9]"; then
VERSION=$(echo "${LATEST_MSG}" | grep -oiE "upgrade to [0-9][^[:space:]]+" | awk '{print $NF}')
BRANCH="upgrade-${VERSION}"
else
BRANCH="upgrade-$(git rev-parse --short HEAD)"
fi
echo "→ PR branch: ${BRANCH}"
# Check / create the mirror repo.
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" "${API}/repos/${NAMESPACE}/${RECIPE}")
if [ "${STATUS}" = "404" ]; then
echo "→ Mirror ${NAMESPACE}/${RECIPE} missing; creating..."
BODY=$(python3 -c "import json;print(json.dumps({'name':'${RECIPE}','private':True,'default_branch':'main','auto_init':False}))")
CS=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" -X POST "${API}/orgs/${NAMESPACE}/repos" -d "${BODY}")
if [ "${CS}" != "201" ]; then
CS=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" -X POST "${API}/user/repos" -d "${BODY}")
[ "${CS}" = "201" ] || { echo "ERROR: repo create failed (HTTP ${CS})"; exit 1; }
NAMESPACE="${GITEA_USERNAME}"
fi
echo " ✓ created ${NAMESPACE}/${RECIPE}"
elif [ "${STATUS}" != "200" ]; then
echo "ERROR: unexpected HTTP ${STATUS} checking mirror repo"; exit 1
fi
REMOTE_URL="https://${GITEA_USERNAME}:${PASS_ENC}@${GITEA_URL}/${NAMESPACE}/${RECIPE}.git"
git remote | grep -qx gitea && git remote set-url gitea "${REMOTE_URL}" || git remote add gitea "${REMOTE_URL}"
echo "→ Force-syncing gitea/main from origin/main (clean diff)..."
git push --force gitea "refs/remotes/origin/main:refs/heads/main"
echo "→ Pushing branch '${BRANCH}'..."
git push --force gitea "HEAD:refs/heads/${BRANCH}"
PR_BODY="${RECIPE_PR_BODY:-$(printf 'Recipe upgrade.\n\nCommits on top of upstream main:\n\n%s\n' "$(git log origin/main..HEAD --pretty='- %h %s')")}"
PR_BODY="${PR_BODY}
Tested green on the cc-ci recipe CI server (full suite, cold, against this PR head). NOT merged — for operator review.
cc @trav @notplants"
PAYLOAD=$(python3 -c "
import json,sys
print(json.dumps({'title':sys.argv[1],'body':sys.argv[2],'head':sys.argv[3],'base':'main','reviewers':['trav','notplants']}))" \
"${LATEST_MSG}" "${PR_BODY}" "${BRANCH}")
RESP=$(mktemp)
PS=$(curl -s -o "${RESP}" -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" -X POST "${API}/repos/${NAMESPACE}/${RECIPE}/pulls" -d "${PAYLOAD}")
if [ "${PS}" = "201" ]; then
echo "PR_URL=$(python3 -c "import json;print(json.load(open('${RESP}'))['html_url'])")"
elif [ "${PS}" = "409" ] || grep -q "pull request already exists" "${RESP}" 2>/dev/null; then
echo "PR_URL=https://${GITEA_URL}/${NAMESPACE}/${RECIPE}/pulls (branch ${BRANCH} already has a PR)"
else
echo "ERROR: PR creation failed (HTTP ${PS}):"; cat "${RESP}"; rm -f "${RESP}"; exit 1
fi
rm -f "${RESP}"