cc-ci-orchestrator/.claude/skills/recipe-upgrade/SKILL.md

---

name: recipe-upgrade description: Upgrade ONE Co-op Cloud recipe end-to-end and verify it on the cc-ci CI server. Researches available upstream upgrades — fetching each image's release notes via a persisted per-recipe registry (cc-ci-plan/upstream/.md) + the recipe README — plans them (breaking changes, migrations, config), implements the bump (image tags + recipe version label + config), opens a recipe PR, and verifies it by posting !testme on the PR (real CI; results visible in the PR; iterates up to 3×). DEFAULT: recipe PR only, using existing tests — if a test fails because it is genuinely stale, it leaves an explanatory COMMENT on the PR for the operator (does NOT touch tests). With --with-tests: also opens + verifies a PR to update the stale cc-ci test. NEVER merges. The per-recipe worker behind /upgrade-all. Invoke as /recipe-upgrade [--with-tests].

recipe-upgrade

Autonomous, end-to-end upgrade of one recipe, gated by the cc-ci CI server. This is the cc-ci analogue of recipe-maintainer's /recipe-upgrade-full, but the test gate is cc-ci (the real recipe CI), and it inherits ci-test-review's discipline: classify recipe-vs-CI, create + verify fix PRs, never merge, never weaken a test.

It drives cc-ci over ssh cc-ci (cc-ci has abra, the recipe checkouts at ~/.abra/recipes/, the harness, and the bot token to push mirror branches). The Gitea PR API call uses the orchestrator's /srv/cc-ci/.testenv creds. Plans/reports go to /srv/cc-ci/.cc-ci-logs/upgrades/.

Boundaries (same as ci-test-review): the test execution + PR verification are deterministic (the cc-ci harness decides pass/fail). Create + verify, NEVER merge (operator merges). Never weaken a test to make a PR green.

Arguments

• <recipe> — the recipe to upgrade (required).
• --with-tests (optional) — opt in to updating cc-ci tests. Without it (the DEFAULT, and what the weekly /upgrade-all cron uses), this skill only ever opens a recipe PR using the existing tests; if a test fails and is diagnosed as genuinely stale, it leaves a comment on the recipe PR explaining why and stops — it does not modify any test. With --with-tests, a stale-test failure additionally gets a verified cc-ci test-update PR (step 5b). A real upgrade regression is fixed + re-tested in both modes.

Procedure

⚠️ Run abra over a pseudo-TTY (or it FATAs inappropriate ioctl for device)

abra needs a TTY. Over plain ssh cc-ci 'abra …' there is none, so many abra commands die with FATA inappropriate ioctl for device / the input device is not a TTY. This is how the working harness avoids it (runner/harness/abra.py): wrap every abra call in a pseudo-TTY via util-linux script, and pass -n (--no-input) so it never prompts. Always invoke abra like this:

ssh cc-ci 'script -qec "abra <args> -n" /dev/null'

(git/other commands do NOT need the wrapper — only abra.) Do this for every abra command below.

1. Plan the upgrade (research — follow recipe-upgrade-plan methodology)

⚠️ Two gotchas that silently skip recipes — handle BOTH before the version check

1. go-git auth to the private mirror. Recipes hosted on git.autonomic.zone (custom-html(-tiny), discourse, ghost, immich, lasuite-docs, plausible, uptime-kuma) fail with FATA unable to fetch tags … authentication required: Unauthorized because abra's embedded go-git reads remote.origin.url from .git/config literally and ignores git url.insteadOf / credential helpers. Fix: bake the bot creds into the origin URL first (idempotent): git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git" (only when origin is on git.autonomic.zone).
2. Dirty worktree is usually just the untracked cc-ci overlay (compose.ccci.yml) — don't skip on that. git stash -u before the checks and stash pop after.

Sync the mirror from TRUE upstream FIRST, then check for upgrades against that — so the check sees the real current recipe, not a stale mirror.

• (a) Reconcile the mirror from coopcloud (always — do this BEFORE the check). Run:

  set -a; . /srv/cc-ci/.testenv; set +a
  ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
     < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
  

  This fetches true upstream from coopcloud (git.coopcloud.tech/coop-cloud/<recipe>, default branch main OR master — every recipe has a coopcloud correspondent and none are forked) and force-syncs the recipe-maintainers/<recipe> mirror's main + tags to be identical to it. (The checkout's origin is the mirror, so the reconcile fetches coopcloud explicitly via an upstream remote — it does NOT sync the mirror to itself.) It also closes any open mirror PR already in upstream main.

• (b) Refresh the local checkout + check for upgrades (creds + stash per the box; each abra call pty-wrapped) — the checkout pulls from the now-coopcloud-synced mirror:

  ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<recipe>; \
    url=$(git -C ~/.abra/recipes/$R config --get remote.origin.url); \
    case "$url" in https://git.autonomic.zone/*) \
      git -C ~/.abra/recipes/$R remote set-url origin "https://'"$GITEA_USERNAME"':'"$GITEA_PASSWORD"'@${url#https://}";; esac; \
    git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
    script -qec "abra recipe fetch $R --force -n" /dev/null; \
    git -C ~/.abra/recipes/$R fetch origin main; \
    script -qec "abra recipe versions $R -m -n" /dev/null; \
    script -qec "abra recipe upgrade $R -m -n" /dev/null; \
    git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
  

  Skip a recipe as SKIPPED — dirty-worktree only if it is still dirty with TRACKED changes after the stash (real local edits), not for the untracked overlay.

• abra says no upgrades → trust abra for every image whose tag it CAN read; only cross-check upstream for images whose tag format abra CAN'T parse. abra recipe upgrade is the first and trusted approach: for a normally-tagged image (a plain semver/calver tag abra can evaluate) where abra finds no newer version, that service genuinely is up-to-date — do not second-guess it with an upstream check. The blind spot is only images with a tag format abra cannot handle, where it silently contributes no candidate:

  • tag+digest pins (image:tag@sha256:…) — abra FATAs Docker references with both a tag and digest are currently not supported (this is exactly how immich kept getting skipped);
  • a digest-only pin (image@sha256:…, no tag) — nothing for abra to compare. So before writing SKIPPED — up-to-date, do a direct upstream check ONLY for those abra-unparseable images (not for images abra evaluated fine):
  1. Read cc-ci-plan/upstream/<recipe>.md (per-image releases/changelog URLs; discover + write it per the registry bullet below if missing).
  2. For each such image, list the upstream registry tags and pick the newest tag the app version supports (the §2 "list the registry tags, pick the newest one the app supports — don't blindly take the max" method), and compare it to the currently pinned tag.
  3. If any has a newer supported upstream tag than what's pinned → there is an upgrade: proceed to plan/implement it (per §2, preserving any digest pin).
  • Conclude SKIPPED — up-to-date only when abra found no bump for the images it could read and the direct check finds nothing newer for the abra-unparseable images. If every image had a normal tag abra could read, abra's verdict stands alone — no upstream check needed. A tag and digest … not supported FATA is a trigger for the upstream check on that image, never a skip on its own.

• Check git log HEAD..origin/main and upstream PRs (git.coopcloud.tech/coop-cloud/<recipe>/pulls) — if someone already started the bump, re-plan from the tip of origin/main, not from scratch.

• Release-notes registry (discover once, reuse — parity with recipe-maintainer's upstream.md). Read cc-ci-plan/upstream/<recipe>.md — a per-image map of source repo + releases/changelog URL. If it's missing, or an image in compose.yml isn't covered (new/renamed service), discover the upstream project + releases page for each uncovered image (WebSearch the image → its GitHub/source repo → releases/changelog), write/update cc-ci-plan/upstream/<recipe>.md and commit it (git -C /srv/cc-ci-orch add cc-ci-plan/upstream/<recipe>.md && git -C /srv/cc-ci-orch commit -m "upstream(<recipe>): release-notes sources" && git -C /srv/cc-ci-orch push). Format + how it's used: cc-ci-plan/upstream/README.md.

• For each service with an upgrade, fetch the release notes from the registry URLs (WebFetch) between current → target versions, and call out breaking changes / required migrations / new-or- renamed config / dependency version requirements in an "Operator Action Required" section.

• Read the recipe's README (~/.abra/recipes/<recipe>/README.md, if present) for upgrade-specific instructions, migration steps, or breaking-change notes shipped with the recipe itself.

• Write the plan to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md: goal, image tag table (service / current → new), recipe version bump (+ semver reasoning), required compose/ config changes, risks. (No human review gate — proceed straight to implement.)

2. Implement the upgrade (follow recipe-upgrade-apply methodology, on cc-ci)

On cc-ci's ~/.abra/recipes/<recipe> (wrap every abra call per the pseudo-TTY box above — ssh cc-ci 'script -qec "abra … -n" /dev/null'):

• script -qec "abra recipe upgrade <recipe> -n" /dev/null for the tags it can bump; hand-edit compose.yml for any tags it won't, plus the config/env/volume/label changes the plan calls for.
• tag+digest pins (image:tag@sha256:…) — abra can't bump these (it FATAs tag and digest … not supported), so do it by hand, and KEEP the digest pin: decide the new tag from the direct upstream-tag check (see /upgrade-all box item 4 — list the registry tags, pick the newest one the app version supports; do not blindly take the max). Then re-resolve the digest for the new tag and re-append it so the pin style is preserved (a digest is a reproducibility/security pin — never silently drop it): docker buildx imagetools inspect <image>:<newtag> --format '{{.Manifest.Digest}}' (or docker manifest inspect), then write image: <image>:<newtag>@<newdigest>. If the app and its digest-pinned sidecars must move together (e.g. immich-server ↔ its postgres:…-vectorchord… DB), bump them as a matched, compatibility-checked set per the plan.
• Do NOT bump the coop-cloud.${STACK_NAME}.version label in the PR — record the recommended release bump instead. The version bump + tag + publish are all done at the very end by a single real abra recipe release (run by the operator after the upstream PR merges — see recipe-maintainer's /recipe-upstream); the upgrade PR only carries the image-tag + config changes. So leave the version label untouched and just decide the semver bump from the upgrade's nature per the plan (breaking→-x major, new feature→-y minor, patch/security→-z patch). Put the recommended release command — abra recipe release <recipe> -<x|y|z> (NO --dry-run) — in the PR body (§3) so the operator can run it verbatim to publish. At release time abra recipe release computes the final a.b.c+x.y.z from the current label + the flag + the app image tag and syncs the label then. ⚠️ Never run abra recipe release in this skill (with or without --dry-run) — a real release PUBLISHES (pushes the new version to the recipe's git origin) and we NEVER push the recipe's upstream/main; we open a PR from a branch on the mirror. Publishing is the operator's final step. (abra recipe upgrade <recipe> -n for the image-tag bumps above is unchanged.)
• script -qec "abra recipe lint <recipe> -C -n" /dev/null — fix obvious lint errors; if unfixable, record it and continue.
• Commit on a branch describing the image-tag/config changes, e.g. git commit -m "chore: upgrade <service> to <new-tag>". Do not mention a recipe version (none is bumped here) and do not create/push a tag — the branch name falls back to upgrade-<short-sha> (see open-recipe-pr.sh). Do not push to upstream; the version bump + tag + publish are the operator's final abra recipe release step.

2a. Advisory scan (deterministic; ADDITIVE — run it, never skip it)

Run the deterministic scanner for the exact upgrade window and paste its markdown block verbatim into the per-recipe log:

python3 /srv/cc-ci/cc-ci-plan/advisory-scan.py <recipe> --from <old-app-version> --to <new-app-version> \
        [--image <name>=<old>:<new>]...

Pass an --image for EVERY sidecar you upgraded (redis, postgres, nginx …), not just the app — each image is judged by its own versions, and an image you don't name is not counted at all. Repeat the flag for each one and pass them all in a single call (the count is a union across images). e.g. discourse moving app 3.5.3→2026.7.1 and redis 7.4→8.10:

... --from 3.5.3 --to 2026.7.1 --image redis=7.4:8.10

→ 140 CVEs (123 app + 17 redis), where the redis seventeen include a critical (CVE-2025-49844) that is invisible if the sidecar is left out. <name> is substring-matched against source repo names, so make it specific enough to hit exactly one.

If the scan reports advisories it could NOT judge, re-run it with --adjudicate. That is a second pass: it collects each open case's full evidence (advisory prose, references, affected ranges, every release naming the CVE) and asks YOU to decide FIXED / NOT-FIXED / STILL-UNKNOWN with a reason citing that evidence. The deterministic number is a floor — add every FIXED to the count. Say STILL-UNKNOWN rather than inferring from memory, and never record an undecided CVE as unaffected. It also shows what pass 1 already decided; if a verdict looks wrong given its evidence, say so.

It queries, per recipe: the GitHub Security Advisories API for every source repo in cc-ci-plan/upstream/<recipe>.md (CVE + GHSA + severity + vulnerable/patched ranges, so "fixed by THIS upgrade" is computed, not guessed), every vendor release/security URL in that registry (fetched + regex-scanned for CVE ids), and OSV where a package mapping exists.

This does NOT replace your own release-note reading — it is an ADDITIONAL evidence source. Do exactly what you did before, then union the two: the CVE count you report is the union of the CVEs you found in the notes and the CVEs the scan found. Never let the scan lower a count you established by reading.

Why it exists: gitea 1.27.1 fixed CVE-2026-60004 and CVE-2026-59774 (both CVSS 9.8). Both are named only in the vendor's blog security section — the GitHub release notes mention neither — so the release-note read found one unrelated minor item and the weekly report printed a CVE count of "1", then "none". Advisory databases lagged too (OSV 404'd on both; NVD's API had neither by CPE, id, or keyword), which is why the GitHub advisory API and the vendor pages lead.

If the scanner reports failed sources, say so in the log — an incomplete scan must not read as a clean one. If a vendor publishes security notes at a URL the registry lacks (gitea's blog.gitea.com), add it to cc-ci-plan/upstream/<recipe>.md so the next scan sees it.

2b. Direct deploy + inspect on cc-ci — live feedback BEFORE CI (recipe-maintainer style)

Before opening the PR / running !testme, deploy the WIP recipe directly on the cc-ci server and watch it converge — the way recipe-maintainer tests on cctest. This gives you live logs + container state to actually SEE what the upgrade does (a failed migration, a crash-loop, a config gap), instead of only a pass/fail CI card. It is ADDITIONAL to — not a replacement for — the !testme verification + its 3-attempt budget (steps 4–5); it just front-loads the diagnosis so you arrive at CI already knowing the upgrade converges, and waste fewer of those 3 runs on basics.

Deploy the recipe head you just edited with --chaos (deploys the local WIP without a clean published commit), under a dedicated dev-<recipe> domain on the local swarm (--server default). Apply the usual gotchas (pseudo-TTY wrap every abra call; creds baked into the mirror origin; stash the untracked overlay). abra app new generates the .env + secrets:

ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; set -a; . /srv/cc-ci/.testenv; set +a; \
  R=<recipe>; D=dev-$R.ci.commoninternet.net; \
  script -qec "abra app new $R --server default --domain $D -n" /dev/null 2>&1 | tail -3; \
  script -qec "abra app deploy $D --chaos -n" /dev/null 2>&1 | tail -20'

• Inspect logs directly — prefer docker service logs over ssh (more reliable than abra app logs non-interactively, per recipe-maintainer references/recipe-maintainer/learnings.md): ssh cc-ci 'docker stack services ${D//./_}; docker service logs --tail 200 ${D//./_}_app' (swap _app for the failing service; pseudo-TTY wrap any abra app logs/abra app cmd).
• Iterate: edit the recipe → abra app deploy $D --chaos --force to cycle → re-read logs, until it converges, serves, and any migration/config behaves. This is where you debug the real upgrade with full visibility — fold what you learn back into the recipe edit (step 2).
• Tear down — MANDATORY, on every exit path (success, RED, or abort — shared swarm): the dev deploy is yours and you MUST remove it before you finish this recipe / move to the next one. Treat it as a finally:

  ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; D=dev-<recipe>.ci.commoninternet.net; \
    script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
    script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
  

  Then verify nothing leaked: ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe with . as _>"' should print 0 (and no dev-<recipe>_* volumes remain). If the recipe failed, tear the dev deploy down anyway, THEN report the failure — never leave it running. Backstop (defence-in-depth, NOT a substitute for the explicit teardown above): /upgrade-all runs the dev-* reaper (reap-dev-deploys.sh) + the orphan-sweep at the START and END of every run, so a crashed/abandoned loop's dev- deploy is cleared by the next run — but you must still clean up yourself.
• Caveats: shared swarm — keep to ONE dev-<recipe> instance at a time and tear it down before the next recipe; the dev-<recipe> domain is distinct from the harness's per-run domains and from the warm-* canonicals, so the sweep removes a leaked one without touching live services.
• 🚫 This loop deploys + inspects ONLY — it must NEVER modify the cc-ci tests or the harness. You have host access here, but the cc-ci tests/** and runner/** are the independent gate, not yours to touch: do not edit a test/overlay to make a dev- deploy or a !testme go green. Editing cc-ci tests is gated exclusively by --with-tests (step 5b); in DEFAULT mode a stale test gets a PR comment, never an edit. Fix the recipe, never the gate. (Interim rule while the upgrader still runs on the cc-ci host — see cc-ci-plan/IDEAS.md, "Move the upgrader's direct deploy onto separate infra".)

3. Open the recipe PR (never merge)

set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe>" \
   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh

This (a) re-syncs mirror main to true upstream main, (b) closes any open PR already merged upstream (merging it would be a no-op), then handles the upgrade PR: if an open upgrade PR already exists for this recipe (branch upgrade-*), the new work is pushed onto THAT PR's branch and the PR is updated + re-tested — one evolving upgrade PR per recipe, not a second parallel one. Only if none exists is a fresh upgrade-<version> PR opened. Unrelated open PRs (e.g. a backup fix) are NOT closed or touched — the operator decides on those. The PR's diff is exactly this upgrade against the real current upstream main. Capture the PR_URL. Export RECIPE_PR_BODY first (image-tag table + the planned operator-action notes) — and the PR body MUST link the upstream release notes: for each upgraded image/service, pull its release-notes URL from the registry cc-ci-plan/upstream/<recipe>.md (between the current → new version) and include one explicit line per service in the body, e.g. **Upstream release notes:** <service> <old>→<new>: <url>. These links go in the PR comment/body itself (so a reviewer sees exactly what changed upstream) — do NOT relegate them to the plan/report only. The body must also carry the recommended release command — abra recipe release <recipe> -<x|y|z> (NO --dry-run) — on its own line, since the PR does not bump the version label: this is the operator's final publish step (bumps the label + tags + pushes) after the upstream PR merges. Re-running with the same target version updates the existing same-branch PR rather than duplicating it.

4. VERIFY by running !testme ON THE PR (results visible in the PR; iterate ≤3×)

Trigger the real CI on the recipe PR by posting !testme — the bridge runs the harness on cc-ci and posts the run + its results back to the PR, so the operator sees them in the PR. Use the PR index from step 3's PR_URL (.../pulls/<N>):

/srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>     # POST=1: posts !testme + polls
# if it prints VERDICT=PENDING, poll again WITHOUT re-triggering:
POST=0 /srv/cc-ci/.claude/skills/recipe-upgrade/testme-on-pr.sh <recipe> <pr-index>

• GREEN → the recipe PR is verified on cc-ci, results in the PR, ready for operator merge. Done.
• RED → go to step 5. You get up to 3 !testme runs total on this PR (initial + 2 retries): use them to fix and re-test a real upgrade regression (step 5a). Each retry is a fresh !testme after you push a fix, so every attempt's result is recorded in the PR.

5. If !testme is RED → diagnose from the run (recipe regression vs stale cc-ci TEST)

Read the failing run (linked from the PR / the harness log on cc-ci) and classify:

5a. The UPGRADE itself is broken (real regression: bad tag, missing migration, config gap) — fix it on the recipe branch, push, and re-run !testme (within the 3-run budget). The recipe PR is only "working" once !testme is GREEN. If still red after 3 runs, leave the PR open (its red !testme results stand as the evidence) and report FAILED — upgrade not green after 3 !testme runs.

5b. The upgrade is correct but a cc-ci TEST is genuinely stale/wrong for the new version (asserts old behavior, an overlay needs updating, a readiness gate changed). !testme can't go green without a test change, and a test change is gated by --with-tests:

• DEFAULT (no --with-tests) — comment, don't touch tests. Post a comment on the recipe PR that: states the upgrade itself looks correct; identifies the specific test that failed and why it appears stale (what the new version changed vs what the test asserts); and tells the operator they can re-run /recipe-upgrade <recipe> --with-tests to also open + verify a cc-ci test-update PR. Do NOT modify any test. Report SUCCESS-PENDING-TESTS (recipe PR open; !testme red on a stale test; operator to decide).
• --with-tests — open + verify a cc-ci test PR. Make it the ci-test-review way: 0. READ tests/STYLE.md in the cc-ci repo FIRST. It is the rulebook for changing a test, and it is written against the failures this pipeline has actually produced. The two that matter most here: set state up through the app's own interface, never its database (a plausible fixture that INSERTed rows passed on v2 and silently broke on v3, holding the recipe RED for six weeks), and gate on version rather than writing a fixture that supports both — old-version tests can simply be deleted, since the older version is only exercised through the upgrade tier.
  1. Branch recipe-maintainers/cc-ci in a separate clone (single-writer: never push main, never touch the build loops' /cc-ci /cc-ci-adv clones); update the test/overlay.
  2. Verify the recipe upgrade WITH the updated test applied. !testme on the recipe PR uses the deployed/main cc-ci tests, so it can't see the unmerged test change — verify this combo by running the harness with the cc-ci branch checked out on cc-ci: RECIPE=<recipe> REF=upgrade-<new-version> /srv/cc-ci/.claude/skills/ci-test-review/verify-pr.sh (after checking out the cc-ci test branch on cc-ci + rebuilding if needed), plus a small regression sample. Green ⇔ the upgrade passes under the corrected test.
  3. Open the cc-ci test PR via the Gitea API. Note in BOTH PRs that they are a dependent pair (the recipe PR's !testme goes green only once the test PR is merged). Report SUCCESS+TESTPR.

FLAKY (passes on a re-!testme) → note it; don't author a fix for a flake.

Never weaken a test to turn a red upgrade green.

6. Report — single parseable RESULT line + summary

Write the report to /srv/cc-ci/.cc-ci-logs/upgrades/<recipe>-upgrade-<YYYY-MM-DD>.md and print, as the last line, one of these exact prefixes (so /upgrade-all can collect it):

• RESULT: SUCCESS — <recipe> <old> → <new>, !testme GREEN, recipe PR: <url>
• RESULT: SUCCESS-PENDING-TESTS — <recipe> <old> → <new>, recipe PR: <url>; !testme RED on a stale test (commented; re-run --with-tests to update tests)
• RESULT: SUCCESS+TESTPR — <recipe> <old> → <new>; recipe PR: <url>; cc-ci test PR: <url> (verified with the test change; pair)
• RESULT: FAILED — <recipe> at <step>: <one-line reason> (e.g. upgrade not green after 3 !testme runs)
• RESULT: SKIPPED — <recipe>: <up-to-date | dirty-worktree | reason>

Always state explicitly that nothing was merged — the PR(s) await operator review.

Guardrails

• cc-ci is the gate, via !testme on the PR — the harness decides pass/fail and the results live in the PR (operator-visible); AI plans, implements, diagnoses. Up to 3 !testme runs per PR.
• Create + verify, NEVER merge. Recipe PR (and any cc-ci test PR) are operator-merged after review.
• Tests are opt-in. Default = recipe PR only, existing tests; a stale-test failure gets an explanatory PR comment, never a test edit. Only --with-tests opens a cc-ci test PR. The weekly /upgrade-all cron always runs the default — it never auto-updates tests.
• 🚫 NEVER modify the cc-ci tests or harness (tests/**, runner/**) unless explicitly invoked with --with-tests. This is absolute and mode-gated — it holds in every step, including the step-2b direct-deploy loop where you have cc-ci host access. The tests are the independent gate; never edit them to turn a dev- deploy or !testme green. In DEFAULT mode, fix the recipe or leave a PR comment — never touch the gate. (Interim safeguard while the upgrader runs on the cc-ci host; see IDEAS.md.)
• Mirror reflects reality. Each run force-syncs the recipe-maintainers/<recipe> mirror main to true upstream main and closes open PRs already merged upstream. It does NOT close superseded/unrelated open PRs. An existing open upgrade PR is extended (new work pushed onto its branch + re-tested) rather than spawning a second upgrade PR — one evolving upgrade PR per recipe.
• Prefer a recipe-only PR. Only open a cc-ci test PR (under --with-tests) when the upgrade is correct but a test is genuinely stale — never to paper over a real upgrade regression.
• Never weaken a test; bounded changes (the upgrade + minimal test update, not rewrites).
• Real abra path throughout (abra recipe upgrade / abra app deploy via the harness).
• Single-writer / coordination: dedicated branches + separate clones; never push main or disturb the build loops' clones; the shared Swarm is stateful — tear down what you deploy; run when you have effectively-exclusive use of the host (or serialize). See ci-test-review for the shared rules.