cc-ci-orchestrator/.claude/skills/upgrade-all/SKILL.md

name, description

name

description

upgrade-all

Weekly autonomous upgrade run for the cc-ci CI server. Surveys every enrolled recipe (except those tagged `external` in cc-ci-plan/used-recipes.md — used/tested but maintained elsewhere, e.g. uptime-kuma) for available upstream upgrades, then runs /recipe-upgrade on each upgradeable one via a subagent — plan, implement, verify green on cc-ci, open a recipe PR (and, only if a cc-ci test went stale, a verified cc-ci test PR). Collects results into one summary listing every PR to review. Rolling pool by default — works through recipes ALPHABETICALLY keeping DRONE_RUNNER_CAPACITY (the drone runner's slots, currently 2) subagents running at once, starting the next as each finishes; --sequential for one-at-a-time, --capacity N to override the pool size, --parallel to start all at once, --dry-run to preview. NEVER merges. Built to run once weekly on a cron. Invoke as /upgrade-all.

upgrade-all

The cc-ci analogue of recipe-maintainer's /recipe-upgrade-cron-all: an unattended weekly pass that keeps every enrolled recipe current, with cc-ci as the test gate and a human in the loop only where it matters — PR review. It surveys upgrades, runs /recipe-upgrade <recipe> per upgradeable recipe, and writes one summary of every PR to review. It never pushes upstream and never merges.

Drives cc-ci over ssh cc-ci. Logs/summary go to /srv/cc-ci/.cc-ci-logs/upgrades/.

Runs as the cc-ci-upgrader agent. This skill is normally executed by a dedicated, observable one-shot job agent — cc-ci-upgrader — spun up under remote-control (viewable/steerable at claude.ai/code, like the Builder) by cc-ci-plan/launch-upgrader.sh. That agent runs this skill to completion, then stops and stays idle so the run + summary remain reviewable in the web UI (it does NOT self-terminate). The weekly cron just invokes launch-upgrader.sh start; the next week's run clears the idle session and starts fresh. You can also run /upgrade-all inline in any /srv/cc-ci session, but the agent is the intended path so the weekly run isn't buried in headless output.

Arguments (optional $ARGUMENTS)

• A space-separated list of recipe names → only those (else all enrolled recipes).
• --dry-run → survey + print what WOULD upgrade; spawn nothing.
• Default = rolling pool, alphabetical: work through the recipes in alphabetical order keeping DRONE_RUNNER_CAPACITY subagents running at once (the drone runner's slots — currently 2), starting the next recipe the moment one finishes. The 2026-06-10 concurrency restructure (docs/concurrency.md) makes concurrent recipe runs SAFE (per-run recipe trees + app-domain locks + isolation), and the capacity knob is the operator's resource-tuned ceiling — so matching the subagent pool to it uses all available concurrency without oversubscribing.
• --capacity N → override the pool size (else the live DRONE_RUNNER_CAPACITY).
• --sequential → force one subagent at a time (the old cautious default; use if the host is also busy with the build loops).
• --parallel → unbounded: fan out ALL subagents at once, ignoring capacity (heaviest host load; drone still queues !testme beyond capacity, but every subagent's step-2b chaos deploy runs concurrently — only when the box is otherwise idle).

0. Sweep orphans from previous runs (FIRST, before anything else)

A prior run's teardown can crash, an agent can be killed mid-deploy, or a manual debug probe can be left running — leaving an orphan test stack, a standalone debug container, leaked volumes, or a stuck docker run wrapper that contends for the shared Swarm and skews the survey. Clear them before surveying. The sweep is safe by allowlist — it removes only what is NOT infra (traefik/drone/ backups/the ccci-* control plane) and NOT a warm-* canonical (whose retained volumes are spared), so it can never take down a live service:

ssh cc-ci 'bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/sweep-orphans.sh

It is idempotent (a no-op when the host is already clean) and prints what it removed plus the surviving Swarm services (which should be infra + warm-* only — eyeball that before continuing). If anything legitimate looks at risk, stop and investigate rather than proceeding.

Then reap any leftover step-2b dev deploys from a prior run (the dev-* stacks /recipe-upgrade step 2b creates to debug an upgrade with live logs). The full sweep above already removes them, but run the dedicated reaper too so the start/end cleanup is explicit and symmetric (THRESHOLD=0 = clear ALL dev-*, since the run is quiescent now):

ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh

Then reclaim leaked overlay IPs and guard against proxy VIP exhaustion. The shared proxy overlay (a /24 = 254 VIPs that EVERY recipe deploy joins) leaks endpoints under concurrent stack rm (a Swarm endpoint-GC race); over many days of churn the pool exhausts and new test deploys hang in Swarm New state with could not find an available IP while allocating VIP — which looks exactly like a recipe failure but is infra (root-caused 2026-06-12; see cc-ci-plan plan-proxy-vip-exhaustion-fix.md). Reclaim before the run:

# 1. reclaim leaked per-stack overlay networks (cheap, always safe)
ssh cc-ci 'docker network prune -f'
# 2. proxy VIP-exhaustion guard: if the allocator recently failed to assign a VIP, the leak has hit
#    the ceiling — rebuild the allocator with a docker restart (the box is QUIESCENT at run start, so
#    this is a ~30s infra blip that auto-recovers). Only fires when actually needed.
VIPFAIL=$(ssh cc-ci 'journalctl -u docker --since "26 hours ago" --no-pager 2>/dev/null | grep -c "available IP while allocating VIP"')
if [ "${VIPFAIL:-0}" -gt 0 ]; then
  echo "!! proxy VIP exhaustion detected ($VIPFAIL recent failures) — restarting docker to reclaim leaked endpoints"
  ssh cc-ci 'sudo systemctl restart docker'; sleep 25
  ssh cc-ci 'docker node ls && docker service ls --format "{{.Replicas}}" | grep -c "/"'   # sanity: node Ready, infra back
fi

(The durable fix — enlarging the proxy subnet to a /16 — landed 2026-06-13 (phase pvfix). This guard remains as belt-and-suspenders even after the /16 fix: it fires on the exact error signature and restarts docker to reclaim leaked endpoints if VIP exhaustion ever recurs despite the larger subnet.)

Then reclaim STALE docker images so the run can't fill the disk mid-flight. A full run deploys ~16 recipes; their images accumulate week over week and can run the cc-ci root FS to 100% (ENOSPC), which killed the 2026-07-03 run mid-way (lasuite-drive, Wave 6). Clear only stale images — unused by any container AND older than a week — so this week's likely-reused images are kept:

ssh cc-ci 'docker image prune -af --filter until=168h 2>&1 | tail -1; df -h / | tail -1'

(When the run is launched via launch-upgrader.py this is done automatically as step 0 — the prereclaim_cc_ci() pre-step — so you only run it by hand for a manual /upgrade-all.)

1. Build the candidate list

Enrolled recipes = the cc-ci tests/<recipe>/ dirs (same set ci-test-review sweeps), MINUS any recipe tagged external in cc-ci-plan/used-recipes.md — recipes cc-ci uses/tests but does NOT maintain (someone else upgrades them, e.g. uptime-kuma). used-recipes.md is the canonical inventory of every recipe we use; only its weekly rows get an upgrade survey + PR here.

EXTERNAL=$(awk '!/^[[:space:]]*#/ && $2=="external"{print $1}' /srv/cc-ci/cc-ci-plan/used-recipes.md)
ssh cc-ci 'cd /root/cc-ci/tests && ls -d */' | sed 's#/##' \
  | grep -vE '^(_generic|unit|__pycache__)$' \
  | grep -vxF -f <(printf '%s\n' "$EXTERNAL")    # drop externally-maintained recipes

(or the names passed in $ARGUMENTS — an explicit recipe name overrides the external skip, so you can still upgrade one on request.) For each candidate, on cc-ci, check availability — skip dirty/up-to-date. (If /root/cc-ci isn't present on the host, stage it first — see cc-ci-plan/plan-proxy-vip-exhaustion-fix.md / the host-rebuild memory for the staging step.)

⚠️ Four things that silently skip recipes — handle ALL FOUR per recipe before the version check:

1. pseudo-TTY: abra FATAs inappropriate ioctl for device under plain ssh — wrap every abra call in script: ssh cc-ci 'script -qec "abra <args> -n" /dev/null' (git/other commands need none).
2. go-git auth to git.autonomic.zone: recipes on the private mirror FATA unable to fetch tags … authentication required: Unauthorized — abra/go-git reads remote.origin.url literally and ignores git url.insteadOf. Bake creds into origin first (idempotent, only if origin is git.autonomic.zone): git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git".
3. dirty worktree is usually just the untracked cc-ci overlay (compose.ccci.yml) — git stash -u before / stash pop after; only skip dirty-worktree if TRACKED changes remain.
4. tag+digest image pins abra can't parse — DO NOT SKIP, check upstream directly. abra FATAs failed to parse image <ref>, saw: Docker references with both a tag and digest are currently not supported and aborts the WHOLE recipe — even images it already parsed (e.g. immich: immich-server parses fine, then it dies on postgres:14-vectorchord…@sha256:…). This recipe is NOT not-fetchable. Catch the failure and check the upstream registry yourself:
   • Use abra's partial output for the images it DID parse.
   • Enumerate every image: ref in the compose; for each image abra could not parse (the FATA names it; plus any it never reached), strip the @sha256:… mentally and look up the live upstream tags directly — Docker Hub https://hub.docker.com/v2/repositories/<repo>/tags?page_size=100, ghcr https://ghcr.io/v2/<repo>/tags/list (anon bearer from https://ghcr.io/token?scope=repository:<repo>:pull&service=ghcr.io), or script -qec "abra ... " /dev/null-free docker buildx imagetools inspect <image>:<tag>.
   • Use judgment for non-semver tags: match the variant the app version supports, do NOT blindly take the highest (immich pins <pgmajor>-vectorchord<x>-pgvectors<y> to a combo immich-server supports — pick the newest tag on the SAME scheme/compatibility, not pg-18 just because it exists).
   • A recipe is upgradeable if abra OR this direct check finds a newer tag. The actual bump + digest re-pin happens in /recipe-upgrade (see its Implement step).

ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<r>; \
  url=$(git -C ~/.abra/recipes/$R config --get remote.origin.url); \
  case "$url" in https://git.autonomic.zone/*) \
    git -C ~/.abra/recipes/$R remote set-url origin "https://'"$GITEA_USERNAME"':'"$GITEA_PASSWORD"'@${url#https://}";; esac; \
  git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
  script -qec "abra recipe fetch $R --force -n" /dev/null; \
  script -qec "abra recipe upgrade $R -m -n" /dev/null; \
  git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'

Build RECIPES_TO_UPGRADE = recipes with ≥1 available upgrade (after the stash, so the untracked overlay doesn't count as dirty). Others go to SKIPPED_UPFRONT with a reason (dirty-worktree only for real TRACKED edits, up-to-date, not-fetchable). A tag and digest … not supported parse FATA is NOT not-fetchable — per box item 4, catch it and resolve availability via the direct upstream-tag check; only not-fetchable when the mirror genuinely can't be fetched (network/auth), never for a digest-pinned image.

Reconcile every candidate's mirror during the survey (even the up-to-date ones), so merged-upstream PRs are closed and every mirror main tracks true upstream — fleet-wide, not just where there's a new upgrade:

set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
   < /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh

(The per-recipe /recipe-upgrade also reconciles, so this is belt-and-suspenders for skipped recipes — count closed-merged PRs in the summary.)

2. Determine concurrency, then print the plan

Read the live capacity (unless --capacity N / --sequential / --parallel was passed):

CAP=$(ssh cc-ci 'systemctl show drone-runner-exec -p Environment 2>/dev/null' | grep -oE 'DRONE_RUNNER_CAPACITY=[0-9]+' | cut -d= -f2)
CAP=${CAP:-2}   # fallback to the documented default if the query fails

--sequential → CAP=1; --capacity N → CAP=N; --parallel → CAP=∞ (all at once). Then print a table — Recipe | Status (will upgrade / skipped:reason) | Available upgrade(s) — plus the mode (Rolling pool (N=<CAP>, alphabetical) default / Sequential / Parallel). If --dry-run, stop here.

3. Upgrade each recipe via a subagent

For each recipe in RECIPES_TO_UPGRADE, spawn an Agent (subagent_type: "general-purpose", description "Upgrade <recipe> on cc-ci") with a prompt like:

Run the /recipe-upgrade <recipe> skill end-to-end — DEFAULT mode, do NOT pass --with-tests (/srv/cc-ci/.claude/skills/recipe-upgrade/SKILL.md): plan, implement, directly deploy + inspect the WIP on cc-ci for live feedback (step 2b, --chaos, tear it down after), open a recipe PR, and verify it by posting !testme on the PR (results visible in the PR; iterate ≤3×). Use the existing tests — if a test fails because it is genuinely stale, leave an explanatory comment on the PR for the operator and do NOT modify any test. Drive cc-ci over ssh cc-ci. Do NOT prompt. Do NOT push upstream. Do NOT merge. Print exactly one RESULT: line as your final line (SUCCESS / SUCCESS-PENDING-TESTS / FAILED / SKIPPED — SUCCESS+TESTPR will not occur in default mode).

Why default (no --with-tests): the weekly cron must never auto-edit cc-ci tests unattended — a test change deserves a human decision. So the cron opens recipe PRs only; where a recipe's existing test looks stale, the operator sees the explanation in the PR comment and can re-run that one recipe with /recipe-upgrade <recipe> --with-tests to also get a verified test-update PR.

Rolling pool (default), CAP running at once, recipes in ALPHABETICAL order. Keep exactly CAP recipe subagents in flight at all times — as soon as one finishes, immediately start the next alphabetical recipe. No waves, no heavy/light classification — just two (=CAP) always running. How:

1. Sort RECIPES_TO_UPGRADE alphabetically.
2. Start the first CAP as background Agents (run_in_background: true) — you are notified when each completes and can read its final output for the RESULT: line.
3. On each completion: read that agent's RESULT: and record it; if recipes remain, immediately start the next alphabetical one as a background Agent (keeping CAP in flight).
4. Repeat until every recipe has been started AND every subagent has completed, then go to §4. CAP=1 → strictly one-at-a-time (sequential). An Agent that dies/errors → record FAILED — agent tool error and still start its replacement; one failure never aborts the run.

• Parallel (--parallel): no pool cap — start ALL recipes as background Agents at once. Heaviest load (every step-2b chaos deploy concurrent); only when the box is otherwise idle.

4. Collect results

Parse each final RESULT: line into SUCCESS / SUCCESS-PENDING-TESTS / FAILED / SKIPPED (default mode won't emit SUCCESS+TESTPR). A subagent that emitted no RESULT: line → FAILED — no result emitted.

4b. Reap dev deploys (END of run)

Every /recipe-upgrade subagent is required to tear down its own step-2b dev-<recipe> deploy, but once ALL recipes are done, reap any that leaked (a crashed/killed subagent) — the symmetric end-of-run cleanup to Step 0. The run is quiescent here, so clear ALL dev-* unconditionally (THRESHOLD=0):

ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh

(Scoped to dev-* only — never touches the recipe PRs, CI per-run stacks, warm-*, or infra.)

5. Write + print the summary

Write /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-<YYYY-MM-DD>.md and print it, leading with the PR list (the actionable output):

# cc-ci Weekly Upgrade Run — <YYYY-MM-DD>
## Summary
- Considered: N · PR green (!testme): N · PR open but tests look stale (commented): N · Failed: N · Skipped: N
## PRs to review (NOT merged)
- <recipe> <old> → <new> — recipe PR: <url> — !testme GREEN
## PRs where a test looks stale (operator: re-run `--with-tests` to update tests)
- <recipe> <old> → <new> — recipe PR: <url> — !testme RED on <test>; see PR comment
## Failed (investigate)
- <recipe> — at <step>: <reason>   (log: .cc-ci-logs/upgrades/<recipe>-upgrade-<date>.md)
## Skipped
| recipe | reason |

End with the report path and a reminder that nothing was merged.

6. Launch the public Recipe Report

Once the summary is written, kick off the weekly public report (its own agent; model overridable via REPORT_BACKEND/REPORT_MODEL, defaults to opencode-go/glm-5.2): python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh (use fresh, not start — the report is a one-shot and must always run a NEW session for THIS week, even if a previous report session is still around). It runs /recipe-report, reviews this run + the live recipe/PR state, and publishes to https://report.ci.commoninternet.net. Fire-and-forget — it runs independently; you can then go idle.

Safety / coordination (this matters — shared host with the build loops)

• Concurrency is bounded to the drone capacity, and that's deliberate. The 2026-06-10 concurrency restructure (docs/concurrency.md) made concurrent recipe runs correct — per-run recipe trees, app-domain locks, isolation — so two recipes no longer collide/corrupt. The remaining limit is host RESOURCES (memory), which is exactly what DRONE_RUNNER_CAPACITY (=2) is tuned to. Running CAP subagents matches the runner's slots; do NOT exceed it on the shared box (that's what --parallel is for, and only when the box is idle). Each recipe-upgrade still tears down its own step-2b deploy; the end-of-run reap (§4b) clears any that leaked.
• If the host is ALSO running the build loops, prefer --sequential (CAP=1) — the loops and a capacity-2 upgrade together can oversubscribe the 7.6 GB box (a Swarm task can wedge in New state under load). The orchestrator should serialize: don't run a capacity>1 upgrade concurrently with active phase loops.
• Single-writer: every PR (recipe or cc-ci test) is on a dedicated branch; never push main, never touch the build loops' /cc-ci /cc-ci-adv working clones or their in-flight state.
• Contention with active loop development: while the loops are still building cc-ci, this run competes for the host. Prefer a quiescent window; if a recipe fails due to contention it's simply retried next week. (Once cc-ci is built and the loops are idle, this is the steady-state weekly job.)
• Never merges; failures/ skips are surfaced and retried next week — safe to re-run anytime.

Cron

Designed for a weekly Claude Code scheduled task that runs cc-ci-plan/launch-upgrader.sh start — which spins up the cc-ci-upgrader remote-control agent to run this skill to completion (the agent then stays idle/viewable). The cron does NOT invoke /upgrade-all inline.

Schedule (operator, 2026-05-29): the weekly cron is installed as the closing action of the final build phase (Phase 5), and its first run is ~1 hour after the build completes, weekly from then on — i.e. anchored to actual completion, not a fixed clock slot. Compute T0 = completion + 1h and install a weekly job at T0's day-of-week + HH:MM (MM HH * * DOW) running launch-upgrader.sh start. Then verify the first kickoff actually launches the cc-ci-upgrader agent. Do NOT install it earlier — while the loops are still building cc-ci it would contend for the shared host; until then run it manually via launch-upgrader.sh start/fresh. Full procedure: Phase 5 plan §4 (cc-ci-plan/plan-phase5-verify-upgrade-flow.md).

Re-running is idempotent: already-current recipes report SKIPPED — up-to-date; recipes with an open PR for the same branch report the existing PR rather than duplicating it.