diff --git a/.claude/skills/ci-test-review/verify-pr.sh b/.claude/skills/ci-test-review/verify-pr.sh index ac9937d..d223347 100755 --- a/.claude/skills/ci-test-review/verify-pr.sh +++ b/.claude/skills/ci-test-review/verify-pr.sh @@ -21,19 +21,23 @@ SSH="${SSH:-cc-ci}" REPEAT="${REPEAT:-1}" : "${RECIPE:?set RECIPE (e.g. ghost)}" : "${REF:?set REF (the PR head branch or sha)}" +# SRC = the mirror repo under git.autonomic.zone holding the PR branch. The harness +# clones SRC at REF (then pulls upstream tags so the upgrade tier can deploy a prior +# published version). Defaults to the recipe's recipe-maintainers mirror. +SRC="${SRC:-recipe-maintainers/${RECIPE}}" RUNID="$(date -u +%Y%m%dT%H%M%SZ)" REMOTE_LOG="/root/cc-ci-review-logs/verify-${RECIPE}-${RUNID}" ssh "$SSH" "mkdir -p /root/cc-ci-review-logs" -echo "verify-pr: RECIPE=$RECIPE REF=$REF cold full-suite x${REPEAT} on ${SSH}" >&2 +echo "verify-pr: RECIPE=$RECIPE SRC=$SRC REF=$REF cold full-suite x${REPEAT} on ${SSH}" >&2 green=1 for i in $(seq 1 "$REPEAT"); do log="${REMOTE_LOG}.${i}.log" rc=0 - # Real harness, cold (no --quick), against the PR head — same path as !testme. - ssh "$SSH" "cd /root/cc-ci && RECIPE='${RECIPE}' REF='${REF}' cc-ci-run runner/run_recipe_ci.py >'${log}' 2>&1" || rc=$? + # Real harness, cold (no --quick), against the mirror PR head — same path as !testme. + ssh "$SSH" "cd /root/cc-ci && RECIPE='${RECIPE}' SRC='${SRC}' REF='${REF}' cc-ci-run runner/run_recipe_ci.py >'${log}' 2>&1" || rc=$? echo "--- pass ${i}/${REPEAT}: exit ${rc} (log ${SSH}:${log}) ---" >&2 ssh "$SSH" "awk '/===== RUN SUMMARY =====/{f=1} f{print}' '${log}'" >&2 || true [ "$rc" = "0" ] || green=0 diff --git a/.claude/skills/recipe-upgrade/SKILL.md b/.claude/skills/recipe-upgrade/SKILL.md new file mode 100644 index 0000000..3c0ce96 --- /dev/null +++ b/.claude/skills/recipe-upgrade/SKILL.md @@ -0,0 +1,112 @@ +--- +name: recipe-upgrade +description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades, plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), then VERIFIES the change green on cc-ci (full suite, cold, against the PR head) and opens a recipe PR. If the upgrade is correct but a cc-ci TEST is now stale, it also updates the test, verifies that, and opens a second PR to the cc-ci repo. NEVER merges — leaves verified, ready-to-merge PRs. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade . +--- + +# recipe-upgrade + +Autonomous, end-to-end upgrade of **one** recipe, **gated by the cc-ci CI server**. This is the +cc-ci analogue of recipe-maintainer's `/recipe-upgrade-full`, but the test gate is **cc-ci** (the real +recipe CI), and it inherits `ci-test-review`'s discipline: classify recipe-vs-CI, create + **verify** +fix PRs, **never merge**, never weaken a test. + +It drives cc-ci over `ssh cc-ci` (cc-ci has abra, the recipe checkouts at `~/.abra/recipes/`, the +harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's +`/srv/cc-ci/.testenv` creds. Plans/reports go to `/srv/cc-ci/.cc-ci-logs/upgrades/`. + +**Boundaries (same as ci-test-review):** the test execution + PR verification are deterministic (the +cc-ci harness decides pass/fail). **Create + verify, NEVER merge** (operator merges). **Never weaken a +test** to make a PR green. + +## Procedure + +### 1. Plan the upgrade (research — follow recipe-upgrade-plan methodology) +On cc-ci, refresh the recipe and see what's available: +``` +ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \ + git -C ~/.abra/recipes/ status --short; \ + abra recipe fetch --force; \ + git -C ~/.abra/recipes/ fetch origin main; \ + abra recipe versions -m; \ + abra recipe upgrade -m -n' +``` +- **Dirty worktree on cc-ci → abort this recipe** (status `SKIPPED — dirty-worktree`); don't prompt. +- **No upgrades available → stop** (status `SKIPPED — up-to-date`). +- Check `git log HEAD..origin/main` and upstream PRs (`git.coopcloud.tech/coop-cloud//pulls`) + — if someone already started the bump, **re-plan from the tip of `origin/main`**, not from scratch. +- For each service with an upgrade, fetch upstream **release notes** (WebFetch) between current and + target versions and call out **breaking changes / required migrations / new-or-renamed config / + dependency bumps** in an "Operator Action Required" section. +- Write the plan to `/srv/cc-ci/.cc-ci-logs/upgrades/-upgrade-.md`: goal, image + tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ + config changes, risks. (No human review gate — proceed straight to implement.) + +### 2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci) +On cc-ci's `~/.abra/recipes/`: +- `abra recipe upgrade -n` for the tags it can bump; hand-edit `compose.yml` for any tags it + won't, plus the config/env/volume/label changes the plan calls for. +- Bump the `coop-cloud.${STACK_NAME}.version` label to the new version string from the plan. +- `abra recipe lint -C` — fix obvious lint errors; if unfixable, record it and continue. +- Commit on a branch: `git commit -m "chore: upgrade to "` (the commit message drives the + PR branch name `upgrade-`). Do **not** push to upstream; do **not** tag-push. + +### 3. Open the recipe PR (never merge) +``` +set -a; . /srv/cc-ci/.testenv; set +a +ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s " \ + < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh +``` +This pushes the upgrade branch to `recipe-maintainers/` (force-syncing mirror `main` from +upstream so the diff is exactly the upgrade) and opens the PR. Capture the `PR_URL`. Optionally export +`RECIPE_PR_BODY` first (image-tag table + the planned operator-action notes). + +### 4. VERIFY the upgrade on the CI server (the gate) +Run the cc-ci full suite **cold** against the PR head — the dogfood gate from `ci-test-review`: +``` +RECIPE= REF=upgrade- /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh +``` +(`SRC` defaults to `recipe-maintainers/`.) **GREEN** ⇔ harness exits 0 → the recipe PR is +verified, ready for operator merge. **General bar = one cold green**; use `REPEAT=3` only for a recipe +already known to be FLAKY (e.g. lasuite-drive). Record the run summary in the report. + +### 5. If verification is RED → diagnose (recipe vs cc-ci TEST), per ci-test-review +Read the harness log and classify the failure: + +- **The UPGRADE itself is broken** (real regression: bad tag, missing migration, config gap) → fix it + on the recipe branch (bounded — ≤3 iterations), re-push, re-verify. The recipe PR is only "working" + once cc-ci is green. If still red after the budget, leave the PR open and report + `FAILED — upgrade not green`. +- **The upgrade is correct but a cc-ci TEST is now stale/wrong** for the new version (asserts old + behavior, an overlay needs updating, a readiness gate changed) → this is a **CI-server fix**. Make + it the `ci-test-review` way: + 1. Branch `recipe-maintainers/cc-ci` in a **separate clone** (single-writer: never push `main`, + never touch the build loops' `/cc-ci` `/cc-ci-adv` clones), update the test/overlay. + 2. **Verify** the recipe upgrade **with the updated test applied**: check the cc-ci branch out on + cc-ci, rebuild if needed, re-run `verify-pr.sh` for the recipe + a small regression sample. + 3. Open the **cc-ci test PR** via the Gitea API (`/srv/cc-ci/.testenv` creds). Capture its URL. + 4. The recipe PR + the cc-ci test PR are a **pair** — note in each that it depends on the other. +- **FLAKY** (passes on a re-run) → note it; don't author a fix for a flake. + +Never weaken a test to turn a red upgrade green. + +### 6. Report — single parseable RESULT line + summary +Write the report to `/srv/cc-ci/.cc-ci-logs/upgrades/-upgrade-.md` and print, as +the **last line**, one of these exact prefixes (so `/upgrade-all` can collect it): + +- `RESULT: SUCCESS — → , cc-ci GREEN, recipe PR: ` +- `RESULT: SUCCESS+TESTPR — → , cc-ci GREEN; recipe PR: ; cc-ci test PR: ` +- `RESULT: FAILED — at : ` (e.g. upgrade not green after 3 tries) +- `RESULT: SKIPPED — : ` + +Always state explicitly that **nothing was merged** — the PR(s) await operator review. + +## Guardrails +- **cc-ci is the gate** — deterministic harness decides pass/fail; AI plans, implements, diagnoses. +- **Create + verify, NEVER merge.** Recipe PR (and any cc-ci test PR) are operator-merged after review. +- **Prefer a recipe-only PR.** Only open a cc-ci test PR when the upgrade is correct but a test is + genuinely stale — not to paper over a real upgrade regression. +- **Never weaken a test**; **bounded** changes (the upgrade + minimal test update, not rewrites). +- **Real abra path** throughout (`abra recipe upgrade` / `abra app deploy` via the harness). +- **Single-writer / coordination:** dedicated branches + separate clones; never push `main` or disturb + the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have + effectively-exclusive use of the host (or serialize). See `ci-test-review` for the shared rules. diff --git a/.claude/skills/recipe-upgrade/open-recipe-pr.sh b/.claude/skills/recipe-upgrade/open-recipe-pr.sh new file mode 100755 index 0000000..f0dfded --- /dev/null +++ b/.claude/skills/recipe-upgrade/open-recipe-pr.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# recipe-upgrade :: open a recipe PR on git.autonomic.zone (adapted from +# recipe-maintainer's recipe-create-pr for the cc-ci context). +# ------------------------------------------------------------------------- +# RUNS ON cc-ci (it has the recipe checkout at ~/.abra/recipes/). +# Invoke from the orchestrator, feeding creds + the script over ssh: +# +# set -a; . /srv/cc-ci/.testenv; set +a +# ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' \ +# GITEA_URL='$GITEA_URL' bash -s " \ +# < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh +# +# Preconditions on cc-ci: ~/.abra/recipes/ is a git checkout with the +# upgrade commit(s) on HEAD beyond origin/main (the apply step made them). +# Force-syncs the gitea mirror's main from upstream origin/main so the PR diff +# is exactly the upgrade. Prints the PR URL. Never merges. +set -o errexit -o nounset -o pipefail + +RECIPE="${1:?usage: open-recipe-pr.sh }" +: "${GITEA_USERNAME:?missing GITEA_USERNAME (pass via ssh env)}" +: "${GITEA_PASSWORD:?missing GITEA_PASSWORD}" +: "${GITEA_URL:?missing GITEA_URL}" +NAMESPACE="${GITEA_NAMESPACE:-recipe-maintainers}" +RECIPE_DIR="${HOME}/.abra/recipes/${RECIPE}" + +export PATH="/run/current-system/sw/bin:${PATH}" +PASS_ENC=$(python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1],safe=''))" "${GITEA_PASSWORD}") +API="https://${GITEA_URL}/api/v1" +AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}") + +[ -d "${RECIPE_DIR}/.git" ] || { echo "ERROR: ${RECIPE_DIR} is not a git repo (run 'abra recipe fetch ${RECIPE}')"; exit 1; } +cd "${RECIPE_DIR}" + +echo "→ Fetching upstream main..." +git fetch origin main + +DIVERGED=$(git log --oneline origin/main..HEAD 2>/dev/null || true) +[ -n "${DIVERGED}" ] || { echo "ERROR: HEAD has no commits beyond origin/main. Nothing to PR."; exit 1; } +echo "→ Local commits to PR:"; echo "${DIVERGED}" | sed 's/^/ /' + +LATEST_MSG=$(git log -1 --pretty=%s HEAD) +if echo "${LATEST_MSG}" | grep -qiE "upgrade to [0-9]"; then + VERSION=$(echo "${LATEST_MSG}" | grep -oiE "upgrade to [0-9][^[:space:]]+" | awk '{print $NF}') + BRANCH="upgrade-${VERSION}" +else + BRANCH="upgrade-$(git rev-parse --short HEAD)" +fi +echo "→ PR branch: ${BRANCH}" + +# Check / create the mirror repo. +STATUS=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" "${API}/repos/${NAMESPACE}/${RECIPE}") +if [ "${STATUS}" = "404" ]; then + echo "→ Mirror ${NAMESPACE}/${RECIPE} missing; creating..." + BODY=$(python3 -c "import json;print(json.dumps({'name':'${RECIPE}','private':True,'default_branch':'main','auto_init':False}))") + CS=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" -X POST "${API}/orgs/${NAMESPACE}/repos" -d "${BODY}") + if [ "${CS}" != "201" ]; then + CS=$(curl -s -o /dev/null -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" -X POST "${API}/user/repos" -d "${BODY}") + [ "${CS}" = "201" ] || { echo "ERROR: repo create failed (HTTP ${CS})"; exit 1; } + NAMESPACE="${GITEA_USERNAME}" + fi + echo " ✓ created ${NAMESPACE}/${RECIPE}" +elif [ "${STATUS}" != "200" ]; then + echo "ERROR: unexpected HTTP ${STATUS} checking mirror repo"; exit 1 +fi + +REMOTE_URL="https://${GITEA_USERNAME}:${PASS_ENC}@${GITEA_URL}/${NAMESPACE}/${RECIPE}.git" +git remote | grep -qx gitea && git remote set-url gitea "${REMOTE_URL}" || git remote add gitea "${REMOTE_URL}" + +echo "→ Force-syncing gitea/main from origin/main (clean diff)..." +git push --force gitea "refs/remotes/origin/main:refs/heads/main" +echo "→ Pushing branch '${BRANCH}'..." +git push --force gitea "HEAD:refs/heads/${BRANCH}" + +PR_BODY="${RECIPE_PR_BODY:-$(printf 'Recipe upgrade.\n\nCommits on top of upstream main:\n\n%s\n' "$(git log origin/main..HEAD --pretty='- %h %s')")}" +PR_BODY="${PR_BODY} + +Tested green on the cc-ci recipe CI server (full suite, cold, against this PR head). NOT merged — for operator review. + +cc @trav @notplants" +PAYLOAD=$(python3 -c " +import json,sys +print(json.dumps({'title':sys.argv[1],'body':sys.argv[2],'head':sys.argv[3],'base':'main','reviewers':['trav','notplants']}))" \ + "${LATEST_MSG}" "${PR_BODY}" "${BRANCH}") + +RESP=$(mktemp) +PS=$(curl -s -o "${RESP}" -w "%{http_code}" "${AUTH[@]}" -H "Content-Type: application/json" -X POST "${API}/repos/${NAMESPACE}/${RECIPE}/pulls" -d "${PAYLOAD}") +if [ "${PS}" = "201" ]; then + echo "PR_URL=$(python3 -c "import json;print(json.load(open('${RESP}'))['html_url'])")" +elif [ "${PS}" = "409" ] || grep -q "pull request already exists" "${RESP}" 2>/dev/null; then + echo "PR_URL=https://${GITEA_URL}/${NAMESPACE}/${RECIPE}/pulls (branch ${BRANCH} already has a PR)" +else + echo "ERROR: PR creation failed (HTTP ${PS}):"; cat "${RESP}"; rm -f "${RESP}"; exit 1 +fi +rm -f "${RESP}" diff --git a/.claude/skills/upgrade-all/SKILL.md b/.claude/skills/upgrade-all/SKILL.md new file mode 100644 index 0000000..d06cee1 --- /dev/null +++ b/.claude/skills/upgrade-all/SKILL.md @@ -0,0 +1,90 @@ +--- +name: upgrade-all +description: Weekly autonomous upgrade run for the cc-ci CI server. Surveys every enrolled recipe for available upstream upgrades, then runs /recipe-upgrade on each upgradeable one via a subagent — plan, implement, verify green on cc-ci, open a recipe PR (and, only if a cc-ci test went stale, a verified cc-ci test PR). Collects results into one summary listing every PR to review. Sequential by default (shared Swarm); --parallel to fan out; --dry-run to preview. NEVER merges. Built to run once weekly on a cron. Invoke as /upgrade-all. +--- + +# upgrade-all + +The cc-ci analogue of recipe-maintainer's `/recipe-upgrade-cron-all`: an unattended **weekly** pass +that keeps every enrolled recipe current, with **cc-ci as the test gate** and a human in the loop only +where it matters — **PR review**. It surveys upgrades, runs `/recipe-upgrade ` per upgradeable +recipe, and writes one summary of every PR to review. It **never pushes upstream and never merges**. + +Drives cc-ci over `ssh cc-ci`. Logs/summary go to `/srv/cc-ci/.cc-ci-logs/upgrades/`. + +## Arguments (optional `$ARGUMENTS`) +- A space-separated list of recipe names → only those (else all enrolled recipes). +- `--dry-run` → survey + print what WOULD upgrade; spawn nothing. +- `--parallel` → fan out all per-recipe subagents at once (faster, more host load — see safety below). + +## 1. Build the candidate list +Enrolled recipes = the cc-ci `tests//` dirs (same set `ci-test-review` sweeps): +``` +ssh cc-ci 'cd /root/cc-ci/tests && ls -d */' | sed 's#/##' | grep -vE '^(_generic|unit|__pycache__)$' +``` +(or the names passed in `$ARGUMENTS`). For each, on cc-ci, check availability — skip dirty/up-to-date: +``` +ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \ + git -C ~/.abra/recipes/ status --short; \ + abra recipe fetch --force; \ + abra recipe upgrade -m -n' +``` +Build `RECIPES_TO_UPGRADE` = recipes with a **clean worktree** AND **≥1 available upgrade**. Others go +to `SKIPPED_UPFRONT` with a reason (`dirty-worktree`, `up-to-date`, `not-fetchable`). + +## 2. Print the plan +A table — Recipe | Status (will upgrade / skipped:reason) | Available upgrade(s) — plus the mode +(`Sequential` default / `Parallel`). If `--dry-run`, **stop here**. + +## 3. Upgrade each recipe via a subagent +For each recipe in `RECIPES_TO_UPGRADE`, spawn an Agent (`subagent_type: "general-purpose"`, +description `"Upgrade on cc-ci"`) with a prompt like: + +> Run the `/recipe-upgrade ` skill end-to-end +> (`/srv/cc-ci/.claude/skills/recipe-upgrade/SKILL.md`): plan, implement, **verify green on cc-ci**, +> open a recipe PR, and — only if the upgrade is correct but a cc-ci test went stale — open a verified +> cc-ci test PR too. Drive cc-ci over `ssh cc-ci`. Do NOT prompt. Do NOT push upstream. Do NOT merge. +> Print exactly one `RESULT:` line as your final line (the SUCCESS / SUCCESS+TESTPR / FAILED / SKIPPED +> forms from the recipe-upgrade skill). + +- **Sequential (default):** one Agent at a time; wait, collect its `RESULT:`, then the next. If the + Agent tool call itself errors, record `FAILED — agent tool error` and continue. One failure never + aborts the run. +- **Parallel (`--parallel`):** emit all Agent calls in one message (no `run_in_background` — you need + the results). Failures are isolated per recipe. + +## 4. Collect results +Parse each final `RESULT:` line into SUCCESS / SUCCESS+TESTPR / FAILED / SKIPPED. A subagent that +emitted no `RESULT:` line → `FAILED — no result emitted`. + +## 5. Write + print the summary +Write `/srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-.md` and print it, **leading with the +PR list** (the actionable output): +```markdown +# cc-ci Weekly Upgrade Run — +## Summary +- Considered: N · Upgraded (PR opened): N · With cc-ci test PR: N · Failed: N · Skipped: N +## PRs to review (NOT merged) +- → — recipe PR: [+ cc-ci test PR: ] +## Failed (investigate) +- — at : (log: .cc-ci-logs/upgrades/-upgrade-.md) +## Skipped +| recipe | reason | +``` +End with the report path and a reminder that **nothing was merged**. + +## Safety / coordination (this matters — shared host with the build loops) +- **Sequential is the default for a reason.** Recipe deploys are **stateful on the shared Swarm** and + parallel deploys can OOM/collide. Between sequential recipes, the per-recipe `recipe-upgrade` tears + down what it deployed; verify a recipe is undeployed before the next starts (`abra app ls` on cc-ci). +- **Single-writer:** every PR (recipe or cc-ci test) is on a dedicated branch; **never push `main`**, + never touch the build loops' `/cc-ci` `/cc-ci-adv` working clones or their in-flight state. +- **Contention with active loop development:** while the loops are still building cc-ci, this run + competes for the host. Prefer a quiescent window; if a recipe fails due to contention it's simply + retried next week. (Once cc-ci is built and the loops are idle, this is the steady-state weekly job.) +- **Never merges**; failures/ skips are surfaced and retried next week — safe to re-run anytime. + +## Cron +Designed for a weekly Claude Code scheduled task (configured separately) that invokes `/upgrade-all` +in `/srv/cc-ci`. Re-running is idempotent: already-current recipes report `SKIPPED — up-to-date`; +recipes with an open PR for the same branch report the existing PR rather than duplicating it.