cc-ci-orchestrator/.claude/skills/upgrade-all/SKILL.md

---
name: upgrade-all
description: Weekly autonomous upgrade run for the cc-ci CI server. Surveys every enrolled recipe for available upstream upgrades, then runs /recipe-upgrade on each upgradeable one via a subagent — plan, implement, verify green on cc-ci, open a recipe PR (and, only if a cc-ci test went stale, a verified cc-ci test PR). Collects results into one summary listing every PR to review. Sequential by default (shared Swarm); --parallel to fan out; --dry-run to preview. NEVER merges. Built to run once weekly on a cron. Invoke as /upgrade-all.
---

# upgrade-all

The cc-ci analogue of recipe-maintainer's `/recipe-upgrade-cron-all`: an unattended **weekly** pass
that keeps every enrolled recipe current, with **cc-ci as the test gate** and a human in the loop only
where it matters — **PR review**. It surveys upgrades, runs `/recipe-upgrade <recipe>` per upgradeable
recipe, and writes one summary of every PR to review. It **never pushes upstream and never merges**.

Drives cc-ci over `ssh cc-ci`. Logs/summary go to `/srv/cc-ci/.cc-ci-logs/upgrades/`.

**Runs as the `cc-ci-upgrader` agent.** This skill is normally executed by a dedicated, observable
**one-shot job agent** — `cc-ci-upgrader` — spun up under remote-control (viewable/steerable at
claude.ai/code, like the Builder) by `cc-ci-plan/launch-upgrader.sh`. That agent runs this skill to
completion, then **stops and stays idle** so the run + summary remain reviewable in the web UI (it does
NOT self-terminate). The weekly cron just invokes `launch-upgrader.sh start`; the next week's run
clears the idle session and starts fresh. You can also run `/upgrade-all` inline in any `/srv/cc-ci`
session, but the agent is the intended path so the weekly run isn't buried in headless output.

## Arguments (optional `$ARGUMENTS`)
- A space-separated list of recipe names → only those (else all enrolled recipes).
- `--dry-run` → survey + print what WOULD upgrade; spawn nothing.
- `--parallel` → fan out all per-recipe subagents at once (faster, more host load — see safety below).

## 0. Sweep orphans from previous runs (FIRST, before anything else)
A prior run's teardown can crash, an agent can be killed mid-deploy, or a manual debug probe can be
left running — leaving an orphan test stack, a standalone debug container, leaked volumes, or a stuck
`docker run` wrapper that contends for the shared Swarm and skews the survey. Clear them before
surveying. The sweep is **safe by allowlist** — it removes only what is NOT infra (traefik/drone/
backups/the `ccci-*` control plane) and NOT a `warm-*` canonical (whose retained volumes are spared),
so it can never take down a live service:
```
ssh cc-ci 'bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/sweep-orphans.sh
```
It is idempotent (a no-op when the host is already clean) and prints what it removed plus the
surviving Swarm services (which should be infra + `warm-*` only — eyeball that before continuing). If
anything legitimate looks at risk, stop and investigate rather than proceeding.

Then **reap any leftover step-2b dev deploys** from a prior run (the `dev-*` stacks `/recipe-upgrade`
step 2b creates to debug an upgrade with live logs). The full sweep above already removes them, but run
the dedicated reaper too so the start/end cleanup is explicit and symmetric (`THRESHOLD=0` = clear ALL
`dev-*`, since the run is quiescent now):
```
ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh
```

## 1. Build the candidate list
Enrolled recipes = the cc-ci `tests/<recipe>/` dirs (same set `ci-test-review` sweeps):
```
ssh cc-ci 'cd /root/cc-ci/tests && ls -d */' | sed 's#/##' | grep -vE '^(_generic|unit|__pycache__)$'
```
(or the names passed in `$ARGUMENTS`). For each, on cc-ci, check availability — skip dirty/up-to-date.

> ⚠️ **Four things that silently skip recipes — handle ALL FOUR per recipe before the version check:**
> 1. **pseudo-TTY:** abra FATAs `inappropriate ioctl for device` under plain ssh — wrap every abra call
>    in `script`: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'` (git/other commands need none).
> 2. **go-git auth to git.autonomic.zone:** recipes on the private mirror FATA `unable to fetch tags …
>    authentication required: Unauthorized` — abra/go-git reads `remote.origin.url` literally and ignores
>    `git url.insteadOf`. Bake creds into origin first (idempotent, only if origin is git.autonomic.zone):
>    `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`.
> 3. **dirty worktree** is usually just the untracked cc-ci overlay (`compose.ccci.yml`) — `git stash -u`
>    before / `stash pop` after; only skip `dirty-worktree` if TRACKED changes remain.
> 4. **tag+digest image pins abra can't parse — DO NOT SKIP, check upstream directly.** abra FATAs
>    `failed to parse image <ref>, saw: Docker references with both a tag and digest are currently not
>    supported` and **aborts the WHOLE recipe** — even images it already parsed (e.g. immich:
>    `immich-server` parses fine, then it dies on `postgres:14-vectorchord…@sha256:…`). This recipe is
>    **NOT** `not-fetchable`. **Catch the failure and check the upstream registry yourself:**
>    - Use abra's partial output for the images it DID parse.
>    - Enumerate every `image:` ref in the compose; for each image abra could not parse (the FATA names
>      it; plus any it never reached), strip the `@sha256:…` mentally and look up the **live upstream
>      tags** directly — Docker Hub `https://hub.docker.com/v2/repositories/<repo>/tags?page_size=100`,
>      ghcr `https://ghcr.io/v2/<repo>/tags/list` (anon bearer from
>      `https://ghcr.io/token?scope=repository:<repo>:pull&service=ghcr.io`), or
>      `script -qec "abra ... " /dev/null`-free `docker buildx imagetools inspect <image>:<tag>`.
>    - **Use judgment** for non-semver tags: match the variant the app version supports, do NOT blindly
>      take the highest (immich pins `<pgmajor>-vectorchord<x>-pgvectors<y>` to a combo immich-server
>      supports — pick the newest tag on the SAME scheme/compatibility, not pg-18 just because it exists).
>    - A recipe is **upgradeable** if abra OR this direct check finds a newer tag. The actual bump +
>      digest re-pin happens in `/recipe-upgrade` (see its Implement step).
```
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<r>; \
  url=$(git -C ~/.abra/recipes/$R config --get remote.origin.url); \
  case "$url" in https://git.autonomic.zone/*) \
    git -C ~/.abra/recipes/$R remote set-url origin "https://'"$GITEA_USERNAME"':'"$GITEA_PASSWORD"'@${url#https://}";; esac; \
  git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
  script -qec "abra recipe fetch $R --force -n" /dev/null; \
  script -qec "abra recipe upgrade $R -m -n" /dev/null; \
  git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
```
Build `RECIPES_TO_UPGRADE` = recipes with **≥1 available upgrade** (after the stash, so the untracked
overlay doesn't count as dirty). Others go to `SKIPPED_UPFRONT` with a reason (`dirty-worktree` only for
real TRACKED edits, `up-to-date`, `not-fetchable`). **A `tag and digest … not supported` parse FATA is
NOT `not-fetchable`** — per box item 4, catch it and resolve availability via the direct upstream-tag
check; only `not-fetchable` when the mirror genuinely can't be fetched (network/auth), never for a
digest-pinned image.

**Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new
upgrade:
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
```
(The per-recipe `/recipe-upgrade` also reconciles, so this is belt-and-suspenders for skipped recipes —
count closed-merged PRs in the summary.)

## 2. Print the plan
A table — Recipe | Status (will upgrade / skipped:reason) | Available upgrade(s) — plus the mode
(`Sequential` default / `Parallel`). If `--dry-run`, **stop here**.

## 3. Upgrade each recipe via a subagent
For each recipe in `RECIPES_TO_UPGRADE`, spawn an Agent (`subagent_type: "general-purpose"`,
description `"Upgrade <recipe> on cc-ci"`) with a prompt like:

> Run the `/recipe-upgrade <recipe>` skill end-to-end — **DEFAULT mode, do NOT pass `--with-tests`**
> (`/srv/cc-ci/.claude/skills/recipe-upgrade/SKILL.md`): plan, implement, **directly deploy + inspect
> the WIP on cc-ci for live feedback (step 2b, `--chaos`, tear it down after)**, open a recipe PR, and
> verify it by posting `!testme` on the PR (results visible in the PR; iterate ≤3×). Use the **existing
> tests** — if a test fails because it is genuinely stale, **leave an explanatory comment on the PR**
> for the operator and do NOT modify any test. Drive cc-ci over `ssh cc-ci`. Do NOT prompt. Do NOT push
> upstream. Do NOT merge. Print exactly one `RESULT:` line as your final line (SUCCESS /
> SUCCESS-PENDING-TESTS / FAILED / SKIPPED — `SUCCESS+TESTPR` will not occur in default mode).

**Why default (no `--with-tests`):** the weekly cron must never auto-edit cc-ci tests unattended — a
test change deserves a human decision. So the cron opens recipe PRs only; where a recipe's existing
test looks stale, the operator sees the explanation in the PR comment and can re-run that one recipe
with `/recipe-upgrade <recipe> --with-tests` to also get a verified test-update PR.

- **Sequential (default):** one Agent at a time; wait, collect its `RESULT:`, then the next. If the
  Agent tool call itself errors, record `FAILED — agent tool error` and continue. One failure never
  aborts the run.
- **Parallel (`--parallel`):** emit all Agent calls in one message (no `run_in_background` — you need
  the results). Failures are isolated per recipe.

## 4. Collect results
Parse each final `RESULT:` line into SUCCESS / SUCCESS-PENDING-TESTS / FAILED / SKIPPED (default mode
won't emit `SUCCESS+TESTPR`). A subagent that emitted no `RESULT:` line → `FAILED — no result emitted`.

## 4b. Reap dev deploys (END of run)
Every `/recipe-upgrade` subagent is required to tear down its own step-2b `dev-<recipe>` deploy, but
once ALL recipes are done, reap any that leaked (a crashed/killed subagent) — the symmetric end-of-run
cleanup to Step 0. The run is quiescent here, so clear ALL `dev-*` unconditionally (`THRESHOLD=0`):
```
ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh
```
(Scoped to `dev-*` only — never touches the recipe PRs, CI per-run stacks, `warm-*`, or infra.)

## 5. Write + print the summary
Write `/srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-<YYYY-MM-DD>.md` and print it, **leading with the
PR list** (the actionable output):
```markdown
# cc-ci Weekly Upgrade Run — <YYYY-MM-DD>
## Summary
- Considered: N · PR green (!testme): N · PR open but tests look stale (commented): N · Failed: N · Skipped: N
## PRs to review (NOT merged)
- <recipe> <old> → <new> — recipe PR: <url> — !testme GREEN
## PRs where a test looks stale (operator: re-run `--with-tests` to update tests)
- <recipe> <old> → <new> — recipe PR: <url> — !testme RED on <test>; see PR comment
## Failed (investigate)
- <recipe> — at <step>: <reason>   (log: .cc-ci-logs/upgrades/<recipe>-upgrade-<date>.md)
## Skipped
| recipe | reason |
```
End with the report path and a reminder that **nothing was merged**.

## 6. Launch the public Recipe Report
Once the summary is written, kick off the weekly public report (its own agent, separate model — default
opus): **`python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`** (use `fresh`, not `start` — the report
is a one-shot and must always run a NEW session for THIS week, even if a previous report session is
still around). It runs `/recipe-report`, reviews this run + the live recipe/PR state, and publishes to
https://report.ci.commoninternet.net. Fire-and-forget — it runs independently; you can then go idle.

## Safety / coordination (this matters — shared host with the build loops)
- **Sequential is the default for a reason.** Recipe deploys are **stateful on the shared Swarm** and
  parallel deploys can OOM/collide. Between sequential recipes, the per-recipe `recipe-upgrade` tears
  down what it deployed; verify a recipe is undeployed before the next starts
  (`ssh cc-ci 'script -qec "abra app ls -n" /dev/null'` — pseudo-TTY wrapped, per the box above).
- **Single-writer:** every PR (recipe or cc-ci test) is on a dedicated branch; **never push `main`**,
  never touch the build loops' `/cc-ci` `/cc-ci-adv` working clones or their in-flight state.
- **Contention with active loop development:** while the loops are still building cc-ci, this run
  competes for the host. Prefer a quiescent window; if a recipe fails due to contention it's simply
  retried next week. (Once cc-ci is built and the loops are idle, this is the steady-state weekly job.)
- **Never merges**; failures/ skips are surfaced and retried next week — safe to re-run anytime.

## Cron
Designed for a weekly Claude Code scheduled task that runs **`cc-ci-plan/launch-upgrader.sh start`** —
which spins up the `cc-ci-upgrader` remote-control agent to run this skill to completion (the agent
then stays idle/viewable). The cron does NOT invoke `/upgrade-all` inline.

**Schedule (operator, 2026-05-29):** the weekly cron is installed as the **closing action of the final
build phase (Phase 5)**, and its **first run is ~1 hour after the build completes, weekly from then on**
— i.e. anchored to actual completion, not a fixed clock slot. Compute `T0 = completion + 1h` and
install a weekly job at T0's day-of-week + `HH:MM` (`MM HH * * DOW`) running `launch-upgrader.sh
start`. Then **verify the first kickoff** actually launches the `cc-ci-upgrader` agent. Do NOT install
it earlier — while the loops are still building cc-ci it would contend for the shared host; until then
run it manually via `launch-upgrader.sh start`/`fresh`. Full procedure: Phase 5 plan §4
(`cc-ci-plan/plan-phase5-verify-upgrade-flow.md`).

Re-running is idempotent: already-current recipes report `SKIPPED — up-to-date`; recipes with an open
PR for the same branch report the existing PR rather than duplicating it.