autonomic-bot
d95afd934b
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change — open PRs for visibility/historical record and merge directly once verified (invocation = authorization); reports must list merged-PR links + change summaries. (tests-update still never merges the paired recipe upgrade PRs.) - cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy. - AGENTS.md: test-before-switch policy for orchestrator host rebuilds. - NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner server 134487234), with self-update caveats. - JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
111 lines
6.1 KiB
Markdown
111 lines
6.1 KiB
Markdown
---
|
|
name: cc-ci-update
|
|
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
|
|
---
|
|
|
|
# cc-ci-update
|
|
|
|
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
|
|
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
|
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
|
|
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
|
|
|
|
## What this skill does NOT do
|
|
|
|
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This is cc-ci
|
|
**self-maintenance**, not recipe maintenance.
|
|
- **Does NOT merge** any PR — operator merges everything.
|
|
- **Does NOT skip the health gate** — if the server rebuild regresses, stop and roll back; do not
|
|
proceed to the tests sweep on a sick host.
|
|
|
|
## Arguments
|
|
|
|
None. (If you only want one of the two updates, invoke the sub-skill directly:
|
|
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
|
|
|
|
## Preconditions
|
|
|
|
Both sub-skills' preconditions hold:
|
|
- `ssh cc-ci` works (root, tailnet `100.95.31.88`).
|
|
- Operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server (the server-update
|
|
half needs it). Pick a **no-CI window** — no `!testme` / nightly sweep running.
|
|
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (PRs + `!testme`).
|
|
- The shared Swarm is quiescent enough for the tests sweep's verification deploys.
|
|
|
|
## Procedure
|
|
|
|
### 1. Run `/cc-ci-server-update` (host first)
|
|
|
|
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
|
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
|
|
gate** (no failed units, infra up, endpoints 200, disk free). See
|
|
`.claude/skills/cc-ci-server-update/SKILL.md`.
|
|
|
|
**Gate on the result before continuing:**
|
|
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
|
|
- `ROLLED-BACK` / `FAILED` → **STOP.** Do not run the tests sweep on a sick or rolled-back host.
|
|
Report the server-update failure and exit. The operator fixes the server bump first; re-run
|
|
`/cc-ci-update` (or just `/cc-ci-tests-update`) once the server is healthy on the new (or
|
|
reverted) nixpkgs.
|
|
|
|
> **Why server-first?** The tests sweep runs the harness on cc-ci, and the harness depends on the
|
|
> host's python/pytest/playwright/docker toolchain — all from nixpkgs. Updating the host first means
|
|
> the test sweep runs against the **new** toolchain, so a green verification is meaningful for the
|
|
> post-update state (not the pre-update one). It also avoids the edge case where a nixpkgs bump
|
|
> changes a tool the tests depend on, invalidating a test sweep you just did. Server bump is the
|
|
> foundational change; tests ride on top.
|
|
|
|
### 2. Run `/cc-ci-tests-update` (tests second, on the now-updated host)
|
|
|
|
Run the full `/cc-ci-tests-update` procedure: build the stale-test candidate list (carry-over from
|
|
`/upgrade-all` reports + fresh `/ci-test-review` sweep) → classify each failure → for each
|
|
genuinely-stale test, author the minimal update, open a cc-ci test PR, verify via the harness
|
|
(paired with the recipe upgrade PR if carry-over) → tear down dev deploys → report. See
|
|
`.claude/skills/cc-ci-tests-update/SKILL.md`.
|
|
|
|
**No additional gate between recipes** — the tests skill already serializes per-recipe and tears
|
|
down each verify deploy. Just run it to completion.
|
|
|
|
### 3. Combined report
|
|
|
|
After both halves are done, print + write a combined summary to
|
|
`/srv/cc-ci/.cc-ci-logs/cc-ci-update-<YYYY-MM-DD>.md`:
|
|
```markdown
|
|
# cc-ci Update — <YYYY-MM-DD>
|
|
## Server (host dependency bump)
|
|
- nixpkgs: <old-rev[:8]> → <new-rev[:8]> · sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
|
- Rebuild: SUCCESS / ROLLED-BACK / FAILED · host healthy: yes/no
|
|
- Server PR: <url> (NOT merged)
|
|
## Tests (stale-test sweep)
|
|
- Swept: N · Stale found: N · Test PRs opened + verified: N · Failed: N
|
|
- Test PRs: <list of urls> (NOT merged)
|
|
## Overall
|
|
- <one-line headline>
|
|
```
|
|
End with: **nothing was merged** — the server PR + test PRs (+ any paired recipe upgrade PRs) await
|
|
operator review.
|
|
|
|
## Guardrails
|
|
|
|
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
|
|
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
|
|
health gate is a hard go/no-go for step 2.
|
|
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
|
|
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
|
|
concurrent with `/upgrade-all` or active build loops.
|
|
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
|
|
- **Never weaken a test** (tests half) — a stale test is updated to the new correct behavior, not
|
|
loosened.
|
|
- **Stable channel only** for the server bump (never `nixos-unstable`).
|
|
|
|
## Relationship to automatic updates
|
|
|
|
This is the **manual trigger** for what should ideally be two periodic timers:
|
|
- `cc-ci-server-update.timer` (host deps) — e.g. monthly, or on nixpkgs CVE advisories.
|
|
- `cc-ci-tests-update.timer` (stale tests) — e.g. weekly, after the `/upgrade-all` run so it can
|
|
close the loop on that week's commented stale tests.
|
|
|
|
If/when those timers exist, `/cc-ci-update` is the "run both now, in the right order, with the
|
|
gate" skill — either timer can also invoke its single sub-skill directly. The order + the gate
|
|
between them (server healthy before tests) is the only thing this umbrella adds.
|