Files
cc-ci-orchestrator/.claude/skills/upgrade-all/SKILL.md
T
autonomic-bot f744c79e2d upgrade-all: alternate heavy/light per wave (not heaviest-first)
Host memory is the binding limit, so never schedule two HEAVY recipes in the
same capacity wave — pair each heavy (discourse/immich/matrix-synapse/
lasuite-drive/mattermost-lts/ghost) with a light one to bound peak memory while
keeping both slots busy. Heaviest-first could co-schedule two heavies and OOM/
wedge the box (the disc-50cc8a 'New'-state wedge). For CAP>2 cap heavies at
~CAP/2; if only heavies remain, run one-per-wave.
2026-06-12 01:47:22 +00:00

247 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
name: upgrade-all
description: Weekly autonomous upgrade run for the cc-ci CI server. Surveys every enrolled recipe for available upstream upgrades, then runs /recipe-upgrade on each upgradeable one via a subagent — plan, implement, verify green on cc-ci, open a recipe PR (and, only if a cc-ci test went stale, a verified cc-ci test PR). Collects results into one summary listing every PR to review. Concurrency-bounded by default — runs up to DRONE_RUNNER_CAPACITY (the drone runner's slots, currently 2) recipe subagents at a time; --sequential for one-at-a-time, --capacity N to override, --parallel to fan out all, --dry-run to preview. NEVER merges. Built to run once weekly on a cron. Invoke as /upgrade-all.
---
# upgrade-all
The cc-ci analogue of recipe-maintainer's `/recipe-upgrade-cron-all`: an unattended **weekly** pass
that keeps every enrolled recipe current, with **cc-ci as the test gate** and a human in the loop only
where it matters — **PR review**. It surveys upgrades, runs `/recipe-upgrade <recipe>` per upgradeable
recipe, and writes one summary of every PR to review. It **never pushes upstream and never merges**.
Drives cc-ci over `ssh cc-ci`. Logs/summary go to `/srv/cc-ci/.cc-ci-logs/upgrades/`.
**Runs as the `cc-ci-upgrader` agent.** This skill is normally executed by a dedicated, observable
**one-shot job agent** — `cc-ci-upgrader` — spun up under remote-control (viewable/steerable at
claude.ai/code, like the Builder) by `cc-ci-plan/launch-upgrader.sh`. That agent runs this skill to
completion, then **stops and stays idle** so the run + summary remain reviewable in the web UI (it does
NOT self-terminate). The weekly cron just invokes `launch-upgrader.sh start`; the next week's run
clears the idle session and starts fresh. You can also run `/upgrade-all` inline in any `/srv/cc-ci`
session, but the agent is the intended path so the weekly run isn't buried in headless output.
## Arguments (optional `$ARGUMENTS`)
- A space-separated list of recipe names → only those (else all enrolled recipes).
- `--dry-run` → survey + print what WOULD upgrade; spawn nothing.
- **Default = concurrency-bounded:** run up to **`DRONE_RUNNER_CAPACITY`** recipe subagents at a
time (the drone runner's slots — currently `2`). The 2026-06-10 concurrency restructure
(`docs/concurrency.md`) makes concurrent recipe runs SAFE (per-run recipe trees + app-domain
locks + isolation), and the capacity knob is the operator's resource-tuned ceiling — so matching
the subagent pool to it uses all available concurrency without oversubscribing.
- `--capacity N` → override the pool size (else the live `DRONE_RUNNER_CAPACITY`).
- `--sequential` → force one subagent at a time (the old cautious default; use if the host is also
busy with the build loops).
- `--parallel` → unbounded: fan out ALL subagents at once, ignoring capacity (heaviest host load;
drone still queues `!testme` beyond capacity, but every subagent's step-2b chaos deploy runs
concurrently — only when the box is otherwise idle).
## 0. Sweep orphans from previous runs (FIRST, before anything else)
A prior run's teardown can crash, an agent can be killed mid-deploy, or a manual debug probe can be
left running — leaving an orphan test stack, a standalone debug container, leaked volumes, or a stuck
`docker run` wrapper that contends for the shared Swarm and skews the survey. Clear them before
surveying. The sweep is **safe by allowlist** — it removes only what is NOT infra (traefik/drone/
backups/the `ccci-*` control plane) and NOT a `warm-*` canonical (whose retained volumes are spared),
so it can never take down a live service:
```
ssh cc-ci 'bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/sweep-orphans.sh
```
It is idempotent (a no-op when the host is already clean) and prints what it removed plus the
surviving Swarm services (which should be infra + `warm-*` only — eyeball that before continuing). If
anything legitimate looks at risk, stop and investigate rather than proceeding.
Then **reap any leftover step-2b dev deploys** from a prior run (the `dev-*` stacks `/recipe-upgrade`
step 2b creates to debug an upgrade with live logs). The full sweep above already removes them, but run
the dedicated reaper too so the start/end cleanup is explicit and symmetric (`THRESHOLD=0` = clear ALL
`dev-*`, since the run is quiescent now):
```
ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh
```
## 1. Build the candidate list
Enrolled recipes = the cc-ci `tests/<recipe>/` dirs (same set `ci-test-review` sweeps):
```
ssh cc-ci 'cd /root/cc-ci/tests && ls -d */' | sed 's#/##' | grep -vE '^(_generic|unit|__pycache__)$'
```
(or the names passed in `$ARGUMENTS`). For each, on cc-ci, check availability — skip dirty/up-to-date.
> ⚠️ **Four things that silently skip recipes — handle ALL FOUR per recipe before the version check:**
> 1. **pseudo-TTY:** abra FATAs `inappropriate ioctl for device` under plain ssh — wrap every abra call
> in `script`: `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'` (git/other commands need none).
> 2. **go-git auth to git.autonomic.zone:** recipes on the private mirror FATA `unable to fetch tags …
> authentication required: Unauthorized` — abra/go-git reads `remote.origin.url` literally and ignores
> `git url.insteadOf`. Bake creds into origin first (idempotent, only if origin is git.autonomic.zone):
> `git -C ~/.abra/recipes/<r> remote set-url origin "https://$GITEA_USERNAME:$GITEA_PASSWORD@git.autonomic.zone/recipe-maintainers/<r>.git"`.
> 3. **dirty worktree** is usually just the untracked cc-ci overlay (`compose.ccci.yml`) — `git stash -u`
> before / `stash pop` after; only skip `dirty-worktree` if TRACKED changes remain.
> 4. **tag+digest image pins abra can't parse — DO NOT SKIP, check upstream directly.** abra FATAs
> `failed to parse image <ref>, saw: Docker references with both a tag and digest are currently not
> supported` and **aborts the WHOLE recipe** — even images it already parsed (e.g. immich:
> `immich-server` parses fine, then it dies on `postgres:14-vectorchord…@sha256:…`). This recipe is
> **NOT** `not-fetchable`. **Catch the failure and check the upstream registry yourself:**
> - Use abra's partial output for the images it DID parse.
> - Enumerate every `image:` ref in the compose; for each image abra could not parse (the FATA names
> it; plus any it never reached), strip the `@sha256:…` mentally and look up the **live upstream
> tags** directly — Docker Hub `https://hub.docker.com/v2/repositories/<repo>/tags?page_size=100`,
> ghcr `https://ghcr.io/v2/<repo>/tags/list` (anon bearer from
> `https://ghcr.io/token?scope=repository:<repo>:pull&service=ghcr.io`), or
> `script -qec "abra ... " /dev/null`-free `docker buildx imagetools inspect <image>:<tag>`.
> - **Use judgment** for non-semver tags: match the variant the app version supports, do NOT blindly
> take the highest (immich pins `<pgmajor>-vectorchord<x>-pgvectors<y>` to a combo immich-server
> supports — pick the newest tag on the SAME scheme/compatibility, not pg-18 just because it exists).
> - A recipe is **upgradeable** if abra OR this direct check finds a newer tag. The actual bump +
> digest re-pin happens in `/recipe-upgrade` (see its Implement step).
```
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; R=<r>; \
url=$(git -C ~/.abra/recipes/$R config --get remote.origin.url); \
case "$url" in https://git.autonomic.zone/*) \
git -C ~/.abra/recipes/$R remote set-url origin "https://'"$GITEA_USERNAME"':'"$GITEA_PASSWORD"'@${url#https://}";; esac; \
git -C ~/.abra/recipes/$R stash -u >/dev/null 2>&1 || true; \
script -qec "abra recipe fetch $R --force -n" /dev/null; \
script -qec "abra recipe upgrade $R -m -n" /dev/null; \
git -C ~/.abra/recipes/$R stash pop >/dev/null 2>&1 || true'
```
Build `RECIPES_TO_UPGRADE` = recipes with **≥1 available upgrade** (after the stash, so the untracked
overlay doesn't count as dirty). Others go to `SKIPPED_UPFRONT` with a reason (`dirty-worktree` only for
real TRACKED edits, `up-to-date`, `not-fetchable`). **A `tag and digest … not supported` parse FATA is
NOT `not-fetchable`** — per box item 4, catch it and resolve availability via the direct upstream-tag
check; only `not-fetchable` when the mirror genuinely can't be fetched (network/auth), never for a
digest-pinned image.
**Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new
upgrade:
```
set -a; . /srv/cc-ci/.testenv; set +a
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
```
(The per-recipe `/recipe-upgrade` also reconciles, so this is belt-and-suspenders for skipped recipes —
count closed-merged PRs in the summary.)
## 2. Determine concurrency, then print the plan
Read the live capacity (unless `--capacity N` / `--sequential` / `--parallel` was passed):
```
CAP=$(ssh cc-ci 'systemctl show drone-runner-exec -p Environment 2>/dev/null' | grep -oE 'DRONE_RUNNER_CAPACITY=[0-9]+' | cut -d= -f2)
CAP=${CAP:-2} # fallback to the documented default if the query fails
```
`--sequential` → `CAP=1`; `--capacity N` → `CAP=N`; `--parallel` → `CAP=∞` (all at once).
Then print a table — Recipe | Status (will upgrade / skipped:reason) | Available upgrade(s) — plus
the mode (`Concurrency-bounded (N=<CAP>)` default / `Sequential` / `Parallel`). If `--dry-run`,
**stop here**.
## 3. Upgrade each recipe via a subagent
For each recipe in `RECIPES_TO_UPGRADE`, spawn an Agent (`subagent_type: "general-purpose"`,
description `"Upgrade <recipe> on cc-ci"`) with a prompt like:
> Run the `/recipe-upgrade <recipe>` skill end-to-end — **DEFAULT mode, do NOT pass `--with-tests`**
> (`/srv/cc-ci/.claude/skills/recipe-upgrade/SKILL.md`): plan, implement, **directly deploy + inspect
> the WIP on cc-ci for live feedback (step 2b, `--chaos`, tear it down after)**, open a recipe PR, and
> verify it by posting `!testme` on the PR (results visible in the PR; iterate ≤3×). Use the **existing
> tests** — if a test fails because it is genuinely stale, **leave an explanatory comment on the PR**
> for the operator and do NOT modify any test. Drive cc-ci over `ssh cc-ci`. Do NOT prompt. Do NOT push
> upstream. Do NOT merge. Print exactly one `RESULT:` line as your final line (SUCCESS /
> SUCCESS-PENDING-TESTS / FAILED / SKIPPED — `SUCCESS+TESTPR` will not occur in default mode).
**Why default (no `--with-tests`):** the weekly cron must never auto-edit cc-ci tests unattended — a
test change deserves a human decision. So the cron opens recipe PRs only; where a recipe's existing
test looks stale, the operator sees the explanation in the PR comment and can re-run that one recipe
with `/recipe-upgrade <recipe> --with-tests` to also get a verified test-update PR.
- **Concurrency-bounded (default), `CAP` at a time:** process `RECIPES_TO_UPGRADE` in waves of
`CAP`. For each wave, emit `CAP` Agent calls **in one message** (they run concurrently — no
`run_in_background`, you need their `RESULT:` lines); wait for all `CAP` to return; collect their
results; then start the next wave. This keeps the drone runner's `CAP` slots busy without
oversubscribing. `CAP=1` degenerates to one-at-a-time (sequential). An Agent tool-call error →
record `FAILED — agent tool error` and continue; one failure never aborts the run or its wave.
- **Parallel (`--parallel`):** `CAP=∞` — emit ALL Agent calls in one message. Heaviest load;
failures still isolated per recipe.
**Wave composition — alternate heavy and light (do NOT just go heaviest-first).** The binding limit
is host memory, so never put two HEAVY recipes in the same wave: pair each heavy with a light one so
peak memory stays bounded while both slots stay busy. Classify from the survey's image set:
- **HEAVY** (multi-GB images / many services / slow deploy): discourse, immich, matrix-synapse,
lasuite-drive, mattermost-lts, ghost.
- **LIGHT / moderate** (one small image or a few light services): custom-html*, cryptpad, n8n,
uptime-kuma, lasuite-docs, lasuite-meet, keycloak, mailu, plausible.
Build the wave order by drawing one HEAVY + one LIGHT per wave (capacity 2) until a bucket empties,
then pair the remainder. For `CAP>2`, cap heavies per wave at ~`CAP/2`. If only heavies remain, run
them one-per-wave (effectively sequential) rather than risk an OOM.
Note (wave barrier): a wave waits for its slowest recipe before the next starts, so the light slot
idles while its heavy wave-mate finishes — accepted for the weekly run. Do NOT maintain a rolling
pool with `run_in_background` (you'd lose the `RESULT:` lines).
## 4. Collect results
Parse each final `RESULT:` line into SUCCESS / SUCCESS-PENDING-TESTS / FAILED / SKIPPED (default mode
won't emit `SUCCESS+TESTPR`). A subagent that emitted no `RESULT:` line → `FAILED — no result emitted`.
## 4b. Reap dev deploys (END of run)
Every `/recipe-upgrade` subagent is required to tear down its own step-2b `dev-<recipe>` deploy, but
once ALL recipes are done, reap any that leaked (a crashed/killed subagent) — the symmetric end-of-run
cleanup to Step 0. The run is quiescent here, so clear ALL `dev-*` unconditionally (`THRESHOLD=0`):
```
ssh cc-ci 'THRESHOLD=0 bash -s' < /srv/cc-ci/.claude/skills/upgrade-all/reap-dev-deploys.sh
```
(Scoped to `dev-*` only — never touches the recipe PRs, CI per-run stacks, `warm-*`, or infra.)
## 5. Write + print the summary
Write `/srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-<YYYY-MM-DD>.md` and print it, **leading with the
PR list** (the actionable output):
```markdown
# cc-ci Weekly Upgrade Run — <YYYY-MM-DD>
## Summary
- Considered: N · PR green (!testme): N · PR open but tests look stale (commented): N · Failed: N · Skipped: N
## PRs to review (NOT merged)
- <recipe> <old> → <new> — recipe PR: <url> — !testme GREEN
## PRs where a test looks stale (operator: re-run `--with-tests` to update tests)
- <recipe> <old> → <new> — recipe PR: <url> — !testme RED on <test>; see PR comment
## Failed (investigate)
- <recipe> — at <step>: <reason> (log: .cc-ci-logs/upgrades/<recipe>-upgrade-<date>.md)
## Skipped
| recipe | reason |
```
End with the report path and a reminder that **nothing was merged**.
## 6. Launch the public Recipe Report
Once the summary is written, kick off the weekly public report (its own agent, separate model — default
opus): **`python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`** (use `fresh`, not `start` — the report
is a one-shot and must always run a NEW session for THIS week, even if a previous report session is
still around). It runs `/recipe-report`, reviews this run + the live recipe/PR state, and publishes to
https://report.ci.commoninternet.net. Fire-and-forget — it runs independently; you can then go idle.
## Safety / coordination (this matters — shared host with the build loops)
- **Concurrency is bounded to the drone capacity, and that's deliberate.** The 2026-06-10
concurrency restructure (`docs/concurrency.md`) made concurrent recipe runs *correct* — per-run
recipe trees, app-domain locks, isolation — so two recipes no longer collide/corrupt. The
remaining limit is host RESOURCES (memory), which is exactly what `DRONE_RUNNER_CAPACITY` (=2) is
tuned to. Running `CAP` subagents matches the runner's slots; do NOT exceed it on the shared box
(that's what `--parallel` is for, and only when the box is idle). Each `recipe-upgrade` still
tears down its own step-2b deploy; the end-of-run reap (§4b) clears any that leaked.
- **If the host is ALSO running the build loops, prefer `--sequential`** (CAP=1) — the loops and a
capacity-2 upgrade together can oversubscribe the 7.6 GB box (a Swarm task can wedge in `New`
state under load). The orchestrator should serialize: don't run a capacity>1 upgrade concurrently
with active phase loops.
- **Single-writer:** every PR (recipe or cc-ci test) is on a dedicated branch; **never push `main`**,
never touch the build loops' `/cc-ci` `/cc-ci-adv` working clones or their in-flight state.
- **Contention with active loop development:** while the loops are still building cc-ci, this run
competes for the host. Prefer a quiescent window; if a recipe fails due to contention it's simply
retried next week. (Once cc-ci is built and the loops are idle, this is the steady-state weekly job.)
- **Never merges**; failures/ skips are surfaced and retried next week — safe to re-run anytime.
## Cron
Designed for a weekly Claude Code scheduled task that runs **`cc-ci-plan/launch-upgrader.sh start`** —
which spins up the `cc-ci-upgrader` remote-control agent to run this skill to completion (the agent
then stays idle/viewable). The cron does NOT invoke `/upgrade-all` inline.
**Schedule (operator, 2026-05-29):** the weekly cron is installed as the **closing action of the final
build phase (Phase 5)**, and its **first run is ~1 hour after the build completes, weekly from then on**
— i.e. anchored to actual completion, not a fixed clock slot. Compute `T0 = completion + 1h` and
install a weekly job at T0's day-of-week + `HH:MM` (`MM HH * * DOW`) running `launch-upgrader.sh
start`. Then **verify the first kickoff** actually launches the `cc-ci-upgrader` agent. Do NOT install
it earlier — while the loops are still building cc-ci it would contend for the shared host; until then
run it manually via `launch-upgrader.sh start`/`fresh`. Full procedure: Phase 5 plan §4
(`cc-ci-plan/plan-phase5-verify-upgrade-flow.md`).
Re-running is idempotent: already-current recipes report `SKIPPED — up-to-date`; recipes with an open
PR for the same branch report the existing PR rather than duplicating it.