autonomic-bot
f283a371bb
Sanitized single-commit public mirror of recipe-maintainer. - Removed test-ssh/.testenv (live creds); added test-ssh/.testenv.example placeholders. - Removed plans/ and planned-updates/ (deployment-planning docs) so no client/ deployment domains appear in the public repo. - All other secret stores were already gitignored. - docs.coopcloud.tech retained as a submodule (public upstream).
199 lines
12 KiB
Markdown
199 lines
12 KiB
Markdown
---
|
||
description: Autonomous weekly upgrade run — overview all recipes, upgrade each end-to-end (sequentially by default, parallel with --parallel), open PRs
|
||
allowed-tools: [Bash, Read, Write, Glob, Grep, Agent]
|
||
---
|
||
|
||
# Recipe Upgrade Cron All
|
||
|
||
Designed to run unattended (e.g. on a weekly cron). Surveys every maintained recipe for available upgrades, then runs `/recipe-upgrade-full` for each upgradeable recipe via a subagent. Each subagent plans, applies, deploys, tests, and opens a review PR on `git.autonomic.zone`. The cron driver collects results and writes one summary report listing every PR URL.
|
||
|
||
By default recipes are processed **sequentially** — one fully completes before the next begins. Pass `--parallel` to fan out all subagents at once.
|
||
|
||
PRs are reviewed and merged manually by a human afterwards — this skill never pushes to upstream or merges anything.
|
||
|
||
Read and follow the instructions in `.claude/commands/includes/logging.md`.
|
||
Read and follow the instructions in `.claude/commands/includes/guidelines.md`.
|
||
|
||
## Arguments
|
||
|
||
This skill takes no required arguments. Optional `$ARGUMENTS`:
|
||
|
||
- A space-separated list of recipe names → only consider those. Otherwise use the full `maintained-recipes.md` list.
|
||
- The literal token `--dry-run` → do the overview phase and print which recipes WOULD be upgraded, but do not spawn any subagents.
|
||
- The literal token `--parallel` → fan out all subagent upgrades concurrently instead of one-at-a-time.
|
||
|
||
## Why this skill exists
|
||
|
||
Manually walking each recipe through `/recipe-upgrade-plan` then `/recipe-upgrade-apply` doesn't scale to ~10 recipes per week. This skill keeps the human in the loop where it matters (PR review) and out of the loop where it doesn't (mechanical bumps + test runs). Failures are skipped and surfaced in the summary so a human can investigate the next morning.
|
||
|
||
## Parallel safety (relevant when --parallel is used)
|
||
|
||
The apply phase deploys to the shared test instance. Two concerns:
|
||
1. **`context_reset.py` clobbering peers.** Each subagent holds `/workspace/in-use/<recipe>.lock` for its whole run. `context_reset.py` reads that directory and protects every locked recipe (see `scripts/context_reset.py`). So any `context_reset --recipe X` — whether the sequential driver's per-recipe reset (step 3) or a stray/manual invocation — leaves every *other* locked recipe deployed. The lock is what makes the reset safe.
|
||
2. **Server memory pressure.** Parallel deploys can OOM the test server. In parallel mode there is no safe point to undeploy between concurrent subagents, so apps from all of them coexist. Failures are caught per-recipe (step 4 below), reported in the summary, and retried automatically on the next cron run. Acceptable for unattended use.
|
||
|
||
When running sequentially (the default), neither concern bites: only one recipe is worked on at a time, and the driver runs `context_reset --recipe <recipe>` before each subagent (step 3) so the previous recipe is torn down before the next deploys.
|
||
|
||
## Steps
|
||
|
||
### 1. Build the candidate list
|
||
|
||
Determine which recipes to consider:
|
||
|
||
- If `$ARGUMENTS` contains recipe names (anything other than `--dry-run` / `--parallel` alone), use those.
|
||
- Otherwise read `/workspace/maintained-recipes.md` and parse the recipe names (lines starting with `- `).
|
||
|
||
For each candidate recipe, check what upgrades are available — this is the same logic as `/recipe-overview` step 3 (run in a single batch, not as a subagent):
|
||
|
||
```bash
|
||
git -C ~/.abra/recipes/<recipe> status --short # skip if dirty
|
||
abra recipe fetch <recipe> --force # only if clean
|
||
abra recipe upgrade <recipe> -m -n # what's available?
|
||
```
|
||
|
||
Build a filtered list `RECIPES_TO_UPGRADE` containing every recipe that:
|
||
- Is initialized (`recipe-info/<recipe>/recipe.toml` exists), AND
|
||
- Has a clean worktree at `~/.abra/recipes/<recipe>`, AND
|
||
- Has at least one available upgrade.
|
||
|
||
Recipes that fail any of these checks go into a `SKIPPED_UPFRONT` list with a reason (`not-initialized`, `dirty-worktree`, `up-to-date`).
|
||
|
||
### 2. Print the plan
|
||
|
||
Print a table summarising the candidate set:
|
||
|
||
| Recipe | Status | Available upgrades |
|
||
|--------|--------|--------------------|
|
||
| keycloak | will upgrade | 26.0.5 → 26.1.0 |
|
||
| immich | will upgrade | 1.119.0 → 1.122.3 |
|
||
| matrix-synapse | skipped (dirty-worktree) | — |
|
||
| ... | ... | ... |
|
||
|
||
Also print the execution mode: `Sequential` (default) or `Parallel (--parallel)`.
|
||
|
||
If `$ARGUMENTS` contains `--dry-run`, stop here.
|
||
|
||
### 3. Upgrade each recipe via subagent
|
||
|
||
**First, clear orphaned locks from a previous run.** At the start of a run no subagent has been spawned yet, so any `in-use/*.lock` present is an orphan left by a subagent that died without cleaning up. A surviving orphan would protect its recipe from every `context_reset` for the whole run (including the per-recipe resets and the final teardown), re-introducing the accumulation problem. Remove them all before spawning anything:
|
||
|
||
```bash
|
||
ls /workspace/in-use/*.lock 2>/dev/null && rm -f /workspace/in-use/*.lock
|
||
```
|
||
|
||
(In parallel mode this is still correct: subagents create their own locks *after* this point.)
|
||
|
||
The subagent prompt is the same regardless of execution mode. For each recipe in `RECIPES_TO_UPGRADE`, use a prompt like:
|
||
|
||
> You are upgrading the `<recipe>` Co-op Cloud recipe end-to-end as part of an unattended weekly upgrade run. The workspace is `/workspace`. The active test instance is whatever `python3 scripts/get_test_instance.py` reports.
|
||
>
|
||
> Run the full `/recipe-upgrade-full <recipe>` skill (file: `.claude/commands/recipe-upgrade-full.md`) — plan, apply, deploy, test, commit, tag, and open a PR on `git.autonomic.zone`. Do NOT prompt for confirmation at any point. Do NOT push to upstream. Do NOT merge anything.
|
||
>
|
||
> Hold `/workspace/in-use/<recipe>.lock` for the duration so sibling subagents working on other recipes don't context_reset you out.
|
||
>
|
||
> On exit, print one of these single-line statuses as your last line:
|
||
> - `RESULT: SUCCESS — <recipe> upgraded <old> → <new>, PR: <url>`
|
||
> - `RESULT: SUCCESS-NO-PR — <recipe> upgraded but PR creation failed: <reason>`
|
||
> - `RESULT: FAILED — <recipe> at step <step>: <reason>`
|
||
> - `RESULT: SKIPPED — <recipe>: <reason>`
|
||
|
||
Use `subagent_type: "general-purpose"` and pass a short `description` like `"Upgrade <recipe> end-to-end"`.
|
||
|
||
**Sequential mode (default):** Spawn one `Agent` at a time. Wait for it to return, collect its result, then spawn the next. Do not move on until the current recipe finishes. **If the `Agent` tool call itself errors or throws (not just returns `RESULT: FAILED`), catch the error, record the recipe as `FAILED — agent tool error: <error message>`, and continue to the next recipe.** Never let a single failure abort the whole run.
|
||
|
||
**Free the server before each sequential subagent.** Immediately before spawning the subagent for recipe `R`, undeploy the previously-upgraded recipe so apps do not accumulate across the run (this is what caused earlier runs to OOM the test server once many recipes had been processed):
|
||
|
||
```bash
|
||
python3 scripts/context_reset.py --recipe R
|
||
```
|
||
|
||
`--recipe R` protects `R` and its dependencies (plus any `in-use/*.lock` recipes) and undeploys everything else, so the prior recipe is torn down while the one about to be worked on is preserved. Run this for **every** recipe including the first (it also clears any leftover apps from a previous run). If this reset fails, log it and continue to spawn the subagent anyway — the subagent's own deploy may still succeed, and a failure here must not abort the run.
|
||
|
||
**Parallel mode does not get per-recipe resets** — all subagents deploy concurrently, so there is no safe point to undeploy between them. Parallel runs rely solely on the per-recipe `in-use` locks and accept the memory-pressure tradeoff documented in the "Parallel safety" section above.
|
||
|
||
**Parallel mode (`--parallel`):** Emit all `Agent` tool calls in a single assistant message so they run concurrently. Do NOT use `run_in_background` — you need the results to build the summary. Individual subagent failures do not affect siblings.
|
||
|
||
### 4. Collect results
|
||
|
||
When all subagents return, parse the final `RESULT:` line from each to classify into:
|
||
|
||
- `SUCCESS` — upgrade PR opened
|
||
- `SUCCESS-NO-PR` — upgrade committed locally, PR creation failed (human can retry with `/recipe-create-pr <recipe>`)
|
||
- `FAILED` — upgrade did not complete (deploy or test failure, etc.)
|
||
- `SKIPPED` — already up to date, dirty worktree, or other pre-flight skip
|
||
|
||
If a subagent crashed without emitting a `RESULT:` line, or if the `Agent` tool call itself threw an error, treat it as `FAILED — no result emitted`.
|
||
|
||
### 5. Verify no orphaned locks
|
||
|
||
```bash
|
||
ls /workspace/in-use/*.lock 2>/dev/null
|
||
```
|
||
|
||
Each subagent is responsible for releasing its own lock. By this point every subagent has returned, so **any** remaining lock is an orphan from a subagent that exited badly — regardless of whether its recipe was in this run's candidate set. Log a warning naming each leftover lock and remove them all:
|
||
|
||
```bash
|
||
ls /workspace/in-use/*.lock 2>/dev/null && rm -f /workspace/in-use/*.lock
|
||
```
|
||
|
||
This must happen before the step 8 teardown — a surviving lock would protect its recipe from the final `context_reset` and leave it deployed.
|
||
|
||
### 6. Write the summary report
|
||
|
||
Write `/workspace/logs/recipe-upgrade-cron-all-<YYYY-MM-DD>.md` containing:
|
||
|
||
```markdown
|
||
# Weekly Recipe Upgrade Run — <YYYY-MM-DD>
|
||
|
||
## Summary
|
||
|
||
- Considered: N recipes
|
||
- Upgraded successfully (PR opened): N
|
||
- Upgrade succeeded locally but PR failed: N
|
||
- Failed: N
|
||
- Skipped: N
|
||
|
||
## PRs to review
|
||
|
||
- [<recipe>](<gitea-pr-url>) — <old-version> → <new-version>
|
||
- ...
|
||
|
||
## Failed runs (investigate)
|
||
|
||
### <recipe>
|
||
- Step: <step name>
|
||
- Reason: <one-line>
|
||
- Log: logs/recipe-upgrade-full-<recipe>-<date>.md
|
||
|
||
## Skipped
|
||
|
||
| Recipe | Reason |
|
||
|--------|--------|
|
||
| ... | ... |
|
||
```
|
||
|
||
### 7. Print the summary
|
||
|
||
Print the same summary to stdout, leading with the list of PR URLs (this is the primary actionable output of the cron run). End with the report file path.
|
||
|
||
### 8. Final teardown
|
||
|
||
After the summary is written and printed, return the test instance to a bare, infrastructure-only state so it isn't left holding the last upgraded recipe between weekly runs:
|
||
|
||
```bash
|
||
python3 scripts/context_reset.py
|
||
```
|
||
|
||
With no `--recipe`, this undeploys every non-infrastructure app on the active test instance (infra recipes like traefik are always protected). Run this only **after** step 5 confirmed no `in-use/*.lock` files remain — a leftover lock would protect that recipe from teardown.
|
||
|
||
This is best-effort cleanup: if it fails, log the error and the affected domains, but do NOT change the run's overall result or re-print the summary — the upgrade work and PRs from steps 3–7 stand regardless. Note in the log that a manual `/test-context-reset` may be needed.
|
||
|
||
## Operator notes
|
||
|
||
- The cron job that drives this skill is configured separately (e.g. via `/schedule` or a system crontab invoking Claude Code). This skill itself is just the work it does on each run.
|
||
- Re-running is safe: skipped/failed recipes are simply retried next week. Successful recipes will report `up-to-date` and skip until a new upstream version appears.
|
||
- If you want to see what _would_ be upgraded without actually touching anything, run `/recipe-upgrade-cron-all --dry-run`.
|
||
- To upgrade a subset, pass recipe names: `/recipe-upgrade-cron-all keycloak immich`.
|
||
- To run all upgrades concurrently (faster but more server load), pass `--parallel`: `/recipe-upgrade-cron-all --parallel`.
|
||
- Flags and recipe names can be combined freely: `/recipe-upgrade-cron-all keycloak immich --parallel`.
|