Per operator: - Verify via `!testme` posted ON the recipe PR (the real CI path) so results are viewable in the PR; iterate up to 3 !testme runs (fix a real regression + re-test). New helper testme-on-pr.sh posts !testme and polls the PR head commit status for the verdict (POST=0 to keep polling without re-triggering). - Test updates are now OPT-IN via `--with-tests`. DEFAULT: recipe PR only using existing tests; if a test fails and is genuinely stale, leave an explanatory COMMENT on the PR (upgrade looks correct; re-run --with-tests to update tests) and do NOT touch any test. --with-tests keeps the verified cc-ci test-update PR path (verified via the branch-checkout harness run, since !testme uses prod tests). - upgrade-all (weekly cron) calls the DEFAULT — never auto-edits tests unattended; surfaces "tests look stale" PRs in the summary for the operator to opt in per-recipe. - New RESULT: SUCCESS-PENDING-TESTS for the recipe-green-but-test-stale default case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
120 lines
7.2 KiB
Markdown
120 lines
7.2 KiB
Markdown
---
|
||
name: upgrade-all
|
||
description: Weekly autonomous upgrade run for the cc-ci CI server. Surveys every enrolled recipe for available upstream upgrades, then runs /recipe-upgrade on each upgradeable one via a subagent — plan, implement, verify green on cc-ci, open a recipe PR (and, only if a cc-ci test went stale, a verified cc-ci test PR). Collects results into one summary listing every PR to review. Sequential by default (shared Swarm); --parallel to fan out; --dry-run to preview. NEVER merges. Built to run once weekly on a cron. Invoke as /upgrade-all.
|
||
---
|
||
|
||
# upgrade-all
|
||
|
||
The cc-ci analogue of recipe-maintainer's `/recipe-upgrade-cron-all`: an unattended **weekly** pass
|
||
that keeps every enrolled recipe current, with **cc-ci as the test gate** and a human in the loop only
|
||
where it matters — **PR review**. It surveys upgrades, runs `/recipe-upgrade <recipe>` per upgradeable
|
||
recipe, and writes one summary of every PR to review. It **never pushes upstream and never merges**.
|
||
|
||
Drives cc-ci over `ssh cc-ci`. Logs/summary go to `/srv/cc-ci/.cc-ci-logs/upgrades/`.
|
||
|
||
## Arguments (optional `$ARGUMENTS`)
|
||
- A space-separated list of recipe names → only those (else all enrolled recipes).
|
||
- `--dry-run` → survey + print what WOULD upgrade; spawn nothing.
|
||
- `--parallel` → fan out all per-recipe subagents at once (faster, more host load — see safety below).
|
||
|
||
## 1. Build the candidate list
|
||
Enrolled recipes = the cc-ci `tests/<recipe>/` dirs (same set `ci-test-review` sweeps):
|
||
```
|
||
ssh cc-ci 'cd /root/cc-ci/tests && ls -d */' | sed 's#/##' | grep -vE '^(_generic|unit|__pycache__)$'
|
||
```
|
||
(or the names passed in `$ARGUMENTS`). For each, on cc-ci, check availability — skip dirty/up-to-date:
|
||
```
|
||
ssh cc-ci 'export PATH=/run/current-system/sw/bin:$PATH; \
|
||
git -C ~/.abra/recipes/<r> status --short; \
|
||
abra recipe fetch <r> --force; \
|
||
abra recipe upgrade <r> -m -n'
|
||
```
|
||
Build `RECIPES_TO_UPGRADE` = recipes with a **clean worktree** AND **≥1 available upgrade**. Others go
|
||
to `SKIPPED_UPFRONT` with a reason (`dirty-worktree`, `up-to-date`, `not-fetchable`).
|
||
|
||
**Reconcile every candidate's mirror during the survey** (even the up-to-date ones), so merged-upstream
|
||
PRs are closed and every mirror `main` tracks true upstream — fleet-wide, not just where there's a new
|
||
upgrade:
|
||
```
|
||
set -a; . /srv/cc-ci/.testenv; set +a
|
||
ssh cc-ci "GITEA_USERNAME='$GITEA_USERNAME' GITEA_PASSWORD='$GITEA_PASSWORD' GITEA_URL='$GITEA_URL' bash -s <recipe> --reconcile-only" \
|
||
< /srv/cc-ci/.claude/skills/recipe-upgrade/open-recipe-pr.sh
|
||
```
|
||
(The per-recipe `/recipe-upgrade` also reconciles, so this is belt-and-suspenders for skipped recipes —
|
||
count closed-merged PRs in the summary.)
|
||
|
||
## 2. Print the plan
|
||
A table — Recipe | Status (will upgrade / skipped:reason) | Available upgrade(s) — plus the mode
|
||
(`Sequential` default / `Parallel`). If `--dry-run`, **stop here**.
|
||
|
||
## 3. Upgrade each recipe via a subagent
|
||
For each recipe in `RECIPES_TO_UPGRADE`, spawn an Agent (`subagent_type: "general-purpose"`,
|
||
description `"Upgrade <recipe> on cc-ci"`) with a prompt like:
|
||
|
||
> Run the `/recipe-upgrade <recipe>` skill end-to-end — **DEFAULT mode, do NOT pass `--with-tests`**
|
||
> (`/srv/cc-ci/.claude/skills/recipe-upgrade/SKILL.md`): plan, implement, open a recipe PR, and verify
|
||
> it by posting `!testme` on the PR (results visible in the PR; iterate ≤3×). Use the **existing
|
||
> tests** — if a test fails because it is genuinely stale, **leave an explanatory comment on the PR**
|
||
> for the operator and do NOT modify any test. Drive cc-ci over `ssh cc-ci`. Do NOT prompt. Do NOT push
|
||
> upstream. Do NOT merge. Print exactly one `RESULT:` line as your final line (SUCCESS /
|
||
> SUCCESS-PENDING-TESTS / FAILED / SKIPPED — `SUCCESS+TESTPR` will not occur in default mode).
|
||
|
||
**Why default (no `--with-tests`):** the weekly cron must never auto-edit cc-ci tests unattended — a
|
||
test change deserves a human decision. So the cron opens recipe PRs only; where a recipe's existing
|
||
test looks stale, the operator sees the explanation in the PR comment and can re-run that one recipe
|
||
with `/recipe-upgrade <recipe> --with-tests` to also get a verified test-update PR.
|
||
|
||
- **Sequential (default):** one Agent at a time; wait, collect its `RESULT:`, then the next. If the
|
||
Agent tool call itself errors, record `FAILED — agent tool error` and continue. One failure never
|
||
aborts the run.
|
||
- **Parallel (`--parallel`):** emit all Agent calls in one message (no `run_in_background` — you need
|
||
the results). Failures are isolated per recipe.
|
||
|
||
## 4. Collect results
|
||
Parse each final `RESULT:` line into SUCCESS / SUCCESS-PENDING-TESTS / FAILED / SKIPPED (default mode
|
||
won't emit `SUCCESS+TESTPR`). A subagent that emitted no `RESULT:` line → `FAILED — no result emitted`.
|
||
|
||
## 5. Write + print the summary
|
||
Write `/srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-<YYYY-MM-DD>.md` and print it, **leading with the
|
||
PR list** (the actionable output):
|
||
```markdown
|
||
# cc-ci Weekly Upgrade Run — <YYYY-MM-DD>
|
||
## Summary
|
||
- Considered: N · PR green (!testme): N · PR open but tests look stale (commented): N · Failed: N · Skipped: N
|
||
## PRs to review (NOT merged)
|
||
- <recipe> <old> → <new> — recipe PR: <url> — !testme GREEN
|
||
## PRs where a test looks stale (operator: re-run `--with-tests` to update tests)
|
||
- <recipe> <old> → <new> — recipe PR: <url> — !testme RED on <test>; see PR comment
|
||
## Failed (investigate)
|
||
- <recipe> — at <step>: <reason> (log: .cc-ci-logs/upgrades/<recipe>-upgrade-<date>.md)
|
||
## Skipped
|
||
| recipe | reason |
|
||
```
|
||
End with the report path and a reminder that **nothing was merged**.
|
||
|
||
## Safety / coordination (this matters — shared host with the build loops)
|
||
- **Sequential is the default for a reason.** Recipe deploys are **stateful on the shared Swarm** and
|
||
parallel deploys can OOM/collide. Between sequential recipes, the per-recipe `recipe-upgrade` tears
|
||
down what it deployed; verify a recipe is undeployed before the next starts (`abra app ls` on cc-ci).
|
||
- **Single-writer:** every PR (recipe or cc-ci test) is on a dedicated branch; **never push `main`**,
|
||
never touch the build loops' `/cc-ci` `/cc-ci-adv` working clones or their in-flight state.
|
||
- **Contention with active loop development:** while the loops are still building cc-ci, this run
|
||
competes for the host. Prefer a quiescent window; if a recipe fails due to contention it's simply
|
||
retried next week. (Once cc-ci is built and the loops are idle, this is the steady-state weekly job.)
|
||
- **Never merges**; failures/ skips are surfaced and retried next week — safe to re-run anytime.
|
||
|
||
## Cron
|
||
Designed for a weekly Claude Code scheduled task that invokes `/upgrade-all` in `/srv/cc-ci`.
|
||
|
||
**Agreed schedule:** **Saturday 03:00 UTC** (`0 3 * * 6`) — low-traffic weekend window, PRs waiting by
|
||
Monday.
|
||
|
||
**Activation trigger (operator, 2026-05-29):** do NOT activate while the build loops are still
|
||
constructing cc-ci — it would contend with them for the shared host. **Activate this weekly cron only
|
||
once the cc-ci build is complete (loops finished / cc-ci stable.)** Until then it's run manually /
|
||
on-demand. When activating, create a scheduled task that runs `/upgrade-all` in `/srv/cc-ci` at
|
||
`0 3 * * 6` UTC.
|
||
|
||
Re-running is idempotent: already-current recipes report `SKIPPED — up-to-date`; recipes with an open
|
||
PR for the same branch report the existing PR rather than duplicating it.
|