recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

skills: move cc-ci-update skills to .opencode/ as canonical; .claude/ -> thin wrappers

Browse Source 
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.

Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
This commit is contained in:
autonomic-bot
2026-08-03 19:10:05 +00:00
parent e805baf030
commit 2f22d42d34
6 changed files with 642 additions and 599 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.opencode/skills/cc-ci-server-update/SKILL.md
+269
View File
@@ -0,0 +1,269 @@
---
name: cc-ci-server-update
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
---
# cc-ci-server-update
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
> different machine with its own flake — it is NOT touched here. If you want to update the
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
> operator). This skill is only about the cc-ci **server**.
## What this skill does NOT do
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
- **Does NOT update the orchestrator host** — only the cc-ci server.
- **Does NOT merge** the cc-ci PR — operator merges after review.
## Preconditions / access
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
```
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
```
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
(NOT in git). The rebuild needs them — see the rebuild procedure below.
## Procedure
### 1. Snapshot current state (so you can compare + roll back)
```
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
df -h / | tail -1; nixos-version; readlink /run/current-system'
```
Record: the current `nixos-version`, the current-system store path, the list of active infra
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
failed or an endpoint down gets compared against this.
Also record the **current flake input revs** (the "before" of what you're bumping):
```
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
```
### 2. Prepare a working clone of the cc-ci repo (single-writer)
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
fresh one:
```
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
cd /home/loops/work/cc-ci-server-update
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
```
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
### 3. Bump the flake inputs (deliberate, not drift)
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
blind `nix flake update --relock-all`.
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
**stable channel revision** that's compatible:
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
to staying on the same channel unless the operator asks for the jump.
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
stable nixpkgs. Unstable is for experiments, not production.
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
and check whether `sops-nix` needs a matching bump:
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
host):
```
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
git diff flake.nix flake.lock | head -40
```
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
(meaningfully). The real build happens on the cc-ci host in step 5.
### 4. Commit + open the cc-ci PR (never merge)
```
git add flake.nix flake.lock
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
sops-nix: <old-rev[:8]> → <new-rev[:8]>
<one-line reason — CVE patch / channel bump / sops-nix compat>"
git push -u origin HEAD
```
Open the PR via the helper:
```
set -a; . /srv/cc-ci/.testenv; set +a
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
```
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.")
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
proceed (the rebuild doesn't depend on the lint gate passing).
### 5. Deploy to the live cc-ci host (no-CI window)
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
**(a) Stage the PR branch on the host:**
```
# from the orchestrator, on the PR branch, clean tree:
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
```
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
flake — see (c).)
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
```
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
```
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
**(c) Build first (catch errors before switching):**
```
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
```
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
That's expected; this is a real nixpkgs bump.
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
module option renames across channels (check the release notes), python package removals, a
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
**(d) Switch (the live deploy):**
```
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
```
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
may be transient; a proxy failure is real).
### 6. Verify the server is healthy (health gate — do not skip)
```
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
echo "=== failed units ==="; systemctl --failed --no-legend; \
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
echo "=== disk ==="; df -h / | tail -1'
```
Compare against the step-1 baseline:
- `nixos-version` moved (or the current-system path changed) — the bump landed.
- **No failed units** (the `--failed` output is empty).
- All 7 infra services `active` (or `active (exited)` for the oneshots).
- `docker service ls` shows no `0/N` replicas (everything converged).
- Endpoints `200` (traefik `/api/version`, dashboard root).
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
flake input change on the PR branch is still your source of truth, but the running host reverts
while you fix). Report the rollback in the PR.
### 7. Report
Print + record a one-line result + the PR URL. Write a short note to
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>`
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
- `RESULT: FAILED — <step>: <reason>`
Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator
reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation +
the path to re-apply, not a gate on the deploy).
### 8. Optional cold-boot proof
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
if the operator wants it, and only outside a CI window:
```
ssh cc-ci 'systemctl reboot'
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
```
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
are tracked in its own journal; a note in the server-update report is enough).
## Guardrails
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
step-1 check), but don't gate on a second authorization.
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
`/root/cc-ci-deploy` instead).
- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record.
- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first.
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
to the baseline. Roll back on regression.
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
generation without touching the flake input revs.
## Relationship to an automatic update
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
without a human prompt."
.opencode/skills/cc-ci-tests-update/SKILL.md
+244
View File
@@ -0,0 +1,244 @@
---
name: cc-ci-tests-update
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
---
# cc-ci-tests-update
The fleet-wide **stale-test cleanup** for cc-ci. The weekly `/upgrade-all` cron deliberately does
**not** auto-edit cc-ci tests — it runs in DEFAULT mode and, where a recipe's existing test is
genuinely stale against a new upstream version, it leaves an **explanatory comment** on the recipe
PR and stops (operator decides). Over time those commented stale tests accumulate. This skill
**closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale,
authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via
the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never
weakens a test and never merges.
> **Relationship to the other skills.** This is the fleet-wide version of the per-recipe
> `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification
> branch of `/ci-test-review`. It reuses both skills' discipline + helpers. The difference is
> **scope**: those skills react to a failure they happened to hit; this one **proactively sweeps
> the whole fleet** for stale tests and fixes them in one pass.
## What this skill does NOT do
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This skill fixes
the **tests**, not the recipes. (It may *discover* a stale test by re-running a recipe upgrade
PR's `!testme`, but the recipe bump itself is owned by `/upgrade-all`.)
- **Does NOT touch the cc-ci server host / flake** — that's `/cc-ci-server-update`.
- **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is
broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review`
job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**.
- **Does NOT merge** any PR — operator merges after review.
## Preconditions / access
- `ssh cc-ci` works (root). Same access notes as `/recipe-upgrade` / `/ci-test-review`.
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening cc-ci test PRs + posting `!testme`).
- The shared Swarm is **quiescent enough** for test runs — each stale-test verification deploys a
recipe, so don't run this concurrent with a `/upgrade-all` or active build loops. Serialize.
- abra over ssh needs the pseudo-TTY wrapper (see the box in `/recipe-upgrade`): every `abra` call
is `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
## Procedure
### 1. Build the stale-test candidate list (sweep — deterministic, no AI judgement yet)
The candidates are recipes whose cc-ci test is **potentially stale**. Two sources, merge both:
**(a) Carry-over from `/upgrade-all` — the commented stale tests.** Each weekly run's report has a
"PRs where a test looks stale" section (recipes where `!testme` went RED on a stale test and got a
PR comment). Read the latest report + a few weeks back:
```
grep -A2 "test.*stale\|stale.*test\|SUCCESS-PENDING-TESTS" /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | tail -40
```
Each line names a recipe + the stale test + the PR URL. These are **known stale** (already
diagnosed by the upgrader) — they go straight to the fix list.
**(b) Fresh sweep — re-run the test suite across all maintained recipes.** Use the
`/ci-test-review` deterministic sweep helper to find failures that aren't already on the carry-over
list:
```
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh
```
It runs the full harness per recipe on cc-ci and writes a JSON summary to
`/srv/cc-ci/.cc-ci-logs/ci-test-review-<runid>.json`. Parse it for `fail` verdicts.
> **Concurrency note:** a full sweep is ~20 recipes × full suite — it takes a while and uses the
> shared Swarm. Run it when the host is otherwise idle. If you only want the carry-over stale tests
> (skip the fresh sweep), pass `RECIPES="recipe1 recipe2"` to the helper to limit it.
**(c) Merge + classify.** Combine (a) and (b) into `CANDIDATES`. For each candidate, classify the
failure (AI — this is the `ci-test-review` step-3 diagnosis):
- **RECIPE bug** (the upgrade is broken) → NOT this skill. Hand off to `/recipe-upgrade` (5a) or
note it for the operator. Don't touch the test.
- **CI-SERVER bug** (harness/infra wrong) → NOT this skill (that's a `/ci-test-review` cc-ci-side
fix, not a test-update). Note it.
- **TEST out-of-date** (the recipe legitimately changed upstream and the cc-ci test/overlay asserts
old behavior) → **this skill**. Add to `STALE_TESTS` with: recipe, the failing test name, what
changed upstream, what the test currently asserts.
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
### 2. For each stale test — author the minimal test update (AI; never weaken)
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
Swarm). For each `STALE_TESTS` entry:
**(a) Set up a dedicated cc-ci clone on a branch** (single-writer — never push `main`, never touch
the build loops' clones):
```
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests-<recipe>
cd /home/loops/work/cc-ci-tests-<recipe>
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b test/<recipe>-stale-test-$(date -u +%Y%m%d)
```
(Or reuse `/home/loops/work/cc-ci-fix` on a new branch if it's clean — the rule is a dedicated
branch off `main`, in a clone you're not racing the loops on.)
**(b) Read the failing test + the upstream change.** The test lives in
`tests/<recipe>/` (custom tests, overlays, fixtures). Read:
- the failing test (the sweep / carry-over names it),
- the recipe's current `compose.yml` + upstream changelog (use the per-recipe registry
`cc-ci-plan/upstream/<recipe>.md` for the release-notes URLs — the `/upgrade-all` diagnosis
already identified what changed),
- what the test currently asserts vs what the new version does.
**(c) Author the minimal update.** Change the test/overlay to assert the **new, correct** behavior
— not to weaken it. Examples of a legitimate stale-test fix:
- a config key was renamed upstream → update the overlay to the new key.
- a default flipped (e.g. `allow_uncategorized_topics` false→true in Discourse 3.5) → update the
assertion to the new default, OR make the test explicitly set the old value (preferred — the
test then asserts the behavior it cares about, not the default).
- an auth method was removed (e.g. Bearer-auth in impress v5.4.0) → drop that assertion, or
replace it with the new auth method's assertion (preferred if the new method is the successor).
- a readiness endpoint changed → update the readiness gate to the new endpoint.
**What is NOT allowed:** lowering an assertion's strictness to make a red go green, deleting a test
that still applies, skipping a tier that still applies, or changing a `pass` threshold. The fix
must make the test **correct for the new version**, not lenient. If you can't fix it without
weakening, stop and report `FAILED — test cannot be updated without weakening; needs operator
decision` (leave the recipe's stale-test PR comment as-is).
**(d) Lint the test change** (cc-ci has a lint gate):
```
ruff check tests/<recipe>/ && ruff format --check tests/<recipe>/
```
Fix any lint errors you introduce (don't commit red lint).
### 3. Open the cc-ci test PR (never merge)
```
git add tests/<recipe>/
git commit -m "test(<recipe>): update stale test for <upstream-change>
<test-name> asserted <old-behavior>; <recipe> <new-version> now <new-behavior>.
Updated to assert the new behavior. No assertion weakened.
Stale-test fix for recipe PR <url> (carry-over from /upgrade-all <date>)."
git push -u origin HEAD
```
Open the PR via the helper:
```
set -a; . /srv/cc-ci/.testenv; set +a
TITLE="test(<recipe>): update stale test for <upstream-change>" \
BODY_FILE=/tmp/cc-ci-tests-<recipe>-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
```
The PR body must: name the stale test, explain what changed upstream, state that no assertion was
weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe
upgrade is also applied — they're a pair), and say "NOT merged — for operator review."
Capture the `TEST_PR_URL`.
### 4. Verify the updated test passes (deterministic; the standard PR + !testme flow)
A test-update PR is only "working" once cc-ci verifies it green with the new test applied. There
are two verification paths — use whichever applies:
**(a) If the stale test was carry-over from an open recipe upgrade PR** — verify via the paired
`!testme` flow (the `/recipe-upgrade --with-tests` step-5b path). The recipe upgrade PR's `!testme`
uses the **deployed/main** cc-ci tests, so it can't see your unmerged test branch — you need to
run the harness with your cc-ci test branch checked out on cc-ci:
```
# 1. On cc-ci, check out your test branch in a deploy clone (NOT /root/builder-clone — use a
# separate checkout so the running server is unaffected):
ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/<recipe>-stale-test-<date> \
ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \
cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true'
# 2. Run the harness against the recipe upgrade PR head, with your test branch as the cc-ci checkout:
RECIPE=<recipe> REF=<recipe-upgrade-pr-head-branch> REMOTE_ROOT=/root/cc-ci-test-verify \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh
```
Green ⇔ the recipe upgrade passes under your corrected test. This is the **paired** verification
(the test change + the recipe upgrade together go green).
**(b) If the stale test is against a recipe already at its current upstream** (no upgrade PR — the
test is just stale against current main) — verify the test branch directly:
```
RECIPE=<recipe> REMOTE_ROOT=/root/cc-ci-test-verify \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh
```
(No `REF` — runs against the current recipe main, with your updated tests.)
**(c) Cleanup the verify clone on cc-ci** (mandatory — shared host):
```
ssh cc-ci 'rm -rf /root/cc-ci-test-verify'
```
**If verification is RED:** iterate the test fix on the same branch (**bounded — ≤3 attempts**) and
re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green
after 3 verify runs` with the failing evidence. Don't weaken the test to force green.
> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open,
> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests —
> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the
> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is
> open and verified-paired, so the operator knows merging both → green.
### 5. Tear down any dev deploys (defence-in-depth)
If your verification path (or diagnosis) deployed a `dev-<recipe>` stack on cc-ci, tear it down:
```
ssh cc-ci 'D=dev-<recipe>.ci.commoninternet.net; script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe>"' # should be 0
```
(The `/upgrade-all` sweep would reap it next run anyway, but clean up yourself.)
### 6. Report — one line per recipe + summary
Write `/srv/cc-ci/.cc-ci-logs/tests-update-<YYYY-MM-DD>.md` and print, leading with the PR list:
```markdown
# cc-ci Stale-Test Update — <YYYY-MM-DD>
## Summary
- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N
## Test PRs to review (NOT merged)
| recipe | stale test | upstream change | test PR | verified |
|--------|-----------|-----------------|---------|----------|
| <recipe> | <test-name> | <what changed> | <url> | GREEN (paired with recipe PR <url>) |
## Failed (needs work)
- <recipe> — <test-name>: <reason> (test PR <url> left open, red)
## Not-stale (dropped — was flaky / real recipe bug / CI-server bug)
- <recipe> — <test-name>: <classification> (handed off / noted)
```
End with the report path and a reminder that **nothing was merged** — the test PRs (and their
paired recipe upgrade PRs) await operator review.
## Guardrails
- **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream
behavior; if that's not possible without weakening, stop and report.
- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is
operator-merged after review.
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify`
on the host and `rm -rf` it after.
- **Stale ≠ broken.** Only update a test when the **upgrade/recipe is correct** and the **test is
wrong** (asserts old behavior). A real upgrade regression is a `/recipe-upgrade` (5a) job; a
CI-server bug is a `/ci-test-review` job. This skill fixes tests, not recipes or harness.
- **Bounded fixes** — minimal update to the failing assertion/overlay, not a test rewrite.
- **Bounded retries** — ≤3 verify runs per test PR; then leave it open + red with evidence.
- **Flake ≠ stale** — re-run before classifying as stale; don't author a fix for a flake.
- **Serialize on the shared Swarm** — one recipe's verify deploy at a time; tear down before the
next. Don't run concurrent with `/upgrade-all` or active build loops.
- **Real abra path** throughout (no docker-level bypass); pseudo-TTY wrap every abra call.
.opencode/skills/cc-ci-update/SKILL.md
+110
View File
@@ -0,0 +1,110 @@
---
name: cc-ci-update
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update.
---
# cc-ci-update
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
## What this skill does NOT do
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This is cc-ci
**self-maintenance**, not recipe maintenance.
- **Does NOT merge** any PR — operator merges everything.
- **Does NOT skip the health gate** — if the server rebuild regresses, stop and roll back; do not
proceed to the tests sweep on a sick host.
## Arguments
None. (If you only want one of the two updates, invoke the sub-skill directly:
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
## Preconditions
Both sub-skills' preconditions hold:
- `ssh cc-ci` works (root, tailnet `100.95.31.88`).
- Operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server (the server-update
half needs it). Pick a **no-CI window** — no `!testme` / nightly sweep running.
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (PRs + `!testme`).
- The shared Swarm is quiescent enough for the tests sweep's verification deploys.
## Procedure
### 1. Run `/cc-ci-server-update` (host first)
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
gate** (no failed units, infra up, endpoints 200, disk free). See
`.claude/skills/cc-ci-server-update/SKILL.md`.
**Gate on the result before continuing:**
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
- `ROLLED-BACK` / `FAILED` → **STOP.** Do not run the tests sweep on a sick or rolled-back host.
Report the server-update failure and exit. The operator fixes the server bump first; re-run
`/cc-ci-update` (or just `/cc-ci-tests-update`) once the server is healthy on the new (or
reverted) nixpkgs.
> **Why server-first?** The tests sweep runs the harness on cc-ci, and the harness depends on the
> host's python/pytest/playwright/docker toolchain — all from nixpkgs. Updating the host first means
> the test sweep runs against the **new** toolchain, so a green verification is meaningful for the
> post-update state (not the pre-update one). It also avoids the edge case where a nixpkgs bump
> changes a tool the tests depend on, invalidating a test sweep you just did. Server bump is the
> foundational change; tests ride on top.
### 2. Run `/cc-ci-tests-update` (tests second, on the now-updated host)
Run the full `/cc-ci-tests-update` procedure: build the stale-test candidate list (carry-over from
`/upgrade-all` reports + fresh `/ci-test-review` sweep) → classify each failure → for each
genuinely-stale test, author the minimal update, open a cc-ci test PR, verify via the harness
(paired with the recipe upgrade PR if carry-over) → tear down dev deploys → report. See
`.claude/skills/cc-ci-tests-update/SKILL.md`.
**No additional gate between recipes** — the tests skill already serializes per-recipe and tears
down each verify deploy. Just run it to completion.
### 3. Combined report
After both halves are done, print + write a combined summary to
`/srv/cc-ci/.cc-ci-logs/cc-ci-update-<YYYY-MM-DD>.md`:
```markdown
# cc-ci Update — <YYYY-MM-DD>
## Server (host dependency bump)
- nixpkgs: <old-rev[:8]> → <new-rev[:8]> · sops-nix: <old-rev[:8]> → <new-rev[:8]>
- Rebuild: SUCCESS / ROLLED-BACK / FAILED · host healthy: yes/no
- Server PR: <url> (NOT merged)
## Tests (stale-test sweep)
- Swept: N · Stale found: N · Test PRs opened + verified: N · Failed: N
- Test PRs: <list of urls> (NOT merged)
## Overall
- <one-line headline>
```
End with: **nothing was merged** — the server PR + test PRs (+ any paired recipe upgrade PRs) await
operator review.
## Guardrails
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
health gate is a hard go/no-go for step 2.
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
concurrent with `/upgrade-all` or active build loops.
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
- **Never weaken a test** (tests half) — a stale test is updated to the new correct behavior, not
loosened.
- **Stable channel only** for the server bump (never `nixos-unstable`).
## Relationship to automatic updates
This is the **manual trigger** for what should ideally be two periodic timers:
- `cc-ci-server-update.timer` (host deps) — e.g. monthly, or on nixpkgs CVE advisories.
- `cc-ci-tests-update.timer` (stale tests) — e.g. weekly, after the `/upgrade-all` run so it can
close the loop on that week's commented stale tests.
If/when those timers exist, `/cc-ci-update` is the "run both now, in the right order, with the
gate" skill — either timer can also invoke its single sub-skill directly. The order + the gate
between them (server healthy before tests) is the only thing this umbrella adds.