skills: move cc-ci-update skills to .opencode/ as canonical; .claude/ -> thin wrappers
Reverse the skill location: the full definitions now live in .opencode/skills/<name>/SKILL.md (opencode's native project-skill location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin pointer wrappers that keep the frontmatter for discoverability and redirect to the .opencode/ canonical file for the procedure. Also drops the 'operator-authorized mid-run' gating from /cc-ci-server-update: the operator's choice to invoke the skill IS the authorization for the live nixos-rebuild switch — no second check to wait for. The quiescent-moment preference stays (don't disrupt an in-flight !testme), but it's not a gate.
This commit is contained in:
autonomic-bot
@@ -1,268 +1,14 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (operator-authorized; picked a no-CI window), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update
|
||||
# cc-ci-server-update (thin wrapper)
|
||||
|
||||
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
|
||||
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
|
||||
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
|
||||
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
|
||||
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
|
||||
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
|
||||
> different machine with its own flake — it is NOT touched here. If you want to update the
|
||||
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
|
||||
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
|
||||
> operator). This skill is only about the cc-ci **server**.
|
||||
**`.opencode/skills/cc-ci-server-update/SKILL.md`**
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
|
||||
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||
- **Does NOT merge** the cc-ci PR — operator merges after review.
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
|
||||
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
|
||||
- The operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server. A switch
|
||||
cycles the abra reconcile oneshots (`deploy-proxy`, `warm-keycloak`, …) — pick a **no-CI window**
|
||||
(no `!testme` / nightly sweep running). Check first:
|
||||
```
|
||||
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
|
||||
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
|
||||
```
|
||||
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
|
||||
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
|
||||
(NOT in git). The rebuild needs them — see the rebuild procedure below.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot current state (so you can compare + roll back)
|
||||
|
||||
```
|
||||
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
|
||||
df -h / | tail -1; nixos-version; readlink /run/current-system'
|
||||
```
|
||||
Record: the current `nixos-version`, the current-system store path, the list of active infra
|
||||
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
|
||||
failed or an endpoint down gets compared against this.
|
||||
|
||||
Also record the **current flake input revs** (the "before" of what you're bumping):
|
||||
```
|
||||
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
|
||||
```
|
||||
|
||||
### 2. Prepare a working clone of the cc-ci repo (single-writer)
|
||||
|
||||
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
|
||||
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
|
||||
fresh one:
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
|
||||
cd /home/loops/work/cc-ci-server-update
|
||||
git fetch origin && git checkout main && git reset --hard origin/main
|
||||
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
|
||||
```
|
||||
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
|
||||
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
|
||||
|
||||
### 3. Bump the flake inputs (deliberate, not drift)
|
||||
|
||||
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
|
||||
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
|
||||
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
|
||||
blind `nix flake update --relock-all`.
|
||||
|
||||
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
|
||||
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
|
||||
**stable channel revision** that's compatible:
|
||||
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
|
||||
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
|
||||
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
|
||||
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
|
||||
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
|
||||
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
|
||||
to staying on the same channel unless the operator asks for the jump.
|
||||
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
|
||||
stable nixpkgs. Unstable is for experiments, not production.
|
||||
|
||||
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
|
||||
and check whether `sops-nix` needs a matching bump:
|
||||
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
|
||||
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
|
||||
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
|
||||
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
|
||||
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
|
||||
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
|
||||
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
|
||||
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
|
||||
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
|
||||
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
|
||||
|
||||
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
|
||||
host):
|
||||
```
|
||||
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
|
||||
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
git diff flake.nix flake.lock | head -40
|
||||
```
|
||||
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
|
||||
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||
|
||||
### 4. Commit + open the cc-ci PR (never merge)
|
||||
|
||||
```
|
||||
git add flake.nix flake.lock
|
||||
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
|
||||
sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||
<one-line reason — CVE patch / channel bump / sops-nix compat>"
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR via the helper:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
|
||||
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||
breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.
|
||||
Live rebuild to follow on operator go-ahead.")
|
||||
|
||||
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||
proceed (the rebuild doesn't depend on the lint gate passing).
|
||||
|
||||
### 5. Deploy to the live cc-ci host (operator-authorized, no-CI window)
|
||||
|
||||
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
|
||||
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
|
||||
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
|
||||
|
||||
**(a) Stage the PR branch on the host:**
|
||||
```
|
||||
# from the orchestrator, on the PR branch, clean tree:
|
||||
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
|
||||
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
|
||||
```
|
||||
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
|
||||
flake — see (c).)
|
||||
|
||||
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
|
||||
```
|
||||
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
|
||||
```
|
||||
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
|
||||
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
|
||||
|
||||
**(c) Build first (catch errors before switching):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
|
||||
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
|
||||
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
|
||||
That's expected; this is a real nixpkgs bump.
|
||||
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
|
||||
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
|
||||
module option renames across channels (check the release notes), python package removals, a
|
||||
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||
|
||||
**(d) Switch (the live deploy):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
|
||||
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
|
||||
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
|
||||
may be transient; a proxy failure is real).
|
||||
|
||||
### 6. Verify the server is healthy (health gate — do not skip)
|
||||
|
||||
```
|
||||
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
|
||||
echo "=== failed units ==="; systemctl --failed --no-legend; \
|
||||
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
|
||||
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
|
||||
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
|
||||
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
|
||||
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
|
||||
echo "=== disk ==="; df -h / | tail -1'
|
||||
```
|
||||
Compare against the step-1 baseline:
|
||||
- `nixos-version` moved (or the current-system path changed) — the bump landed.
|
||||
- **No failed units** (the `--failed` output is empty).
|
||||
- All 7 infra services `active` (or `active (exited)` for the oneshots).
|
||||
- `docker service ls` shows no `0/N` replicas (everything converged).
|
||||
- Endpoints `200` (traefik `/api/version`, dashboard root).
|
||||
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
|
||||
|
||||
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
|
||||
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
|
||||
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||
while you fix). Report the rollback in the PR.
|
||||
|
||||
### 7. Report
|
||||
|
||||
Print + record a one-line result + the PR URL. Write a short note to
|
||||
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>`
|
||||
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||
- `RESULT: FAILED — <step>: <reason>`
|
||||
|
||||
Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator
|
||||
reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation +
|
||||
the path to re-apply, not a gate on the deploy).
|
||||
|
||||
### 8. Optional cold-boot proof
|
||||
|
||||
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
|
||||
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
|
||||
if the operator wants it, and only outside a CI window:
|
||||
```
|
||||
ssh cc-ci 'systemctl reboot'
|
||||
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
|
||||
```
|
||||
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
|
||||
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
|
||||
are tracked in its own journal; a note in the server-update report is enough).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Operator-authorized live rebuild, no-CI window.** A `nixos-rebuild switch` cycles reconcile
|
||||
oneshots and can disrupt an in-flight `!testme`. Check the host is quiescent first (step 1) and
|
||||
get the operator's go-ahead.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||
`/root/cc-ci-deploy` instead).
|
||||
- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record.
|
||||
- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first.
|
||||
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||
to the baseline. Roll back on regression.
|
||||
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
|
||||
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
|
||||
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
|
||||
generation without touching the flake input revs.
|
||||
|
||||
## Relationship to an automatic update
|
||||
|
||||
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
|
||||
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
|
||||
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
|
||||
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
|
||||
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
|
||||
without a human prompt."
|
||||
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
|
||||
@@ -3,242 +3,12 @@ name: cc-ci-tests-update
|
||||
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
|
||||
---
|
||||
|
||||
# cc-ci-tests-update
|
||||
# cc-ci-tests-update (thin wrapper)
|
||||
|
||||
The fleet-wide **stale-test cleanup** for cc-ci. The weekly `/upgrade-all` cron deliberately does
|
||||
**not** auto-edit cc-ci tests — it runs in DEFAULT mode and, where a recipe's existing test is
|
||||
genuinely stale against a new upstream version, it leaves an **explanatory comment** on the recipe
|
||||
PR and stops (operator decides). Over time those commented stale tests accumulate. This skill
|
||||
**closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale,
|
||||
authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via
|
||||
the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never
|
||||
weakens a test and never merges.
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
> **Relationship to the other skills.** This is the fleet-wide version of the per-recipe
|
||||
> `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification
|
||||
> branch of `/ci-test-review`. It reuses both skills' discipline + helpers. The difference is
|
||||
> **scope**: those skills react to a failure they happened to hit; this one **proactively sweeps
|
||||
> the whole fleet** for stale tests and fixes them in one pass.
|
||||
**`.opencode/skills/cc-ci-tests-update/SKILL.md`**
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This skill fixes
|
||||
the **tests**, not the recipes. (It may *discover* a stale test by re-running a recipe upgrade
|
||||
PR's `!testme`, but the recipe bump itself is owned by `/upgrade-all`.)
|
||||
- **Does NOT touch the cc-ci server host / flake** — that's `/cc-ci-server-update`.
|
||||
- **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is
|
||||
broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review`
|
||||
job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**.
|
||||
- **Does NOT merge** any PR — operator merges after review.
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
- `ssh cc-ci` works (root). Same access notes as `/recipe-upgrade` / `/ci-test-review`.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening cc-ci test PRs + posting `!testme`).
|
||||
- The shared Swarm is **quiescent enough** for test runs — each stale-test verification deploys a
|
||||
recipe, so don't run this concurrent with a `/upgrade-all` or active build loops. Serialize.
|
||||
- abra over ssh needs the pseudo-TTY wrapper (see the box in `/recipe-upgrade`): every `abra` call
|
||||
is `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Build the stale-test candidate list (sweep — deterministic, no AI judgement yet)
|
||||
|
||||
The candidates are recipes whose cc-ci test is **potentially stale**. Two sources, merge both:
|
||||
|
||||
**(a) Carry-over from `/upgrade-all` — the commented stale tests.** Each weekly run's report has a
|
||||
"PRs where a test looks stale" section (recipes where `!testme` went RED on a stale test and got a
|
||||
PR comment). Read the latest report + a few weeks back:
|
||||
```
|
||||
grep -A2 "test.*stale\|stale.*test\|SUCCESS-PENDING-TESTS" /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | tail -40
|
||||
```
|
||||
Each line names a recipe + the stale test + the PR URL. These are **known stale** (already
|
||||
diagnosed by the upgrader) — they go straight to the fix list.
|
||||
|
||||
**(b) Fresh sweep — re-run the test suite across all maintained recipes.** Use the
|
||||
`/ci-test-review` deterministic sweep helper to find failures that aren't already on the carry-over
|
||||
list:
|
||||
```
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh
|
||||
```
|
||||
It runs the full harness per recipe on cc-ci and writes a JSON summary to
|
||||
`/srv/cc-ci/.cc-ci-logs/ci-test-review-<runid>.json`. Parse it for `fail` verdicts.
|
||||
|
||||
> **Concurrency note:** a full sweep is ~20 recipes × full suite — it takes a while and uses the
|
||||
> shared Swarm. Run it when the host is otherwise idle. If you only want the carry-over stale tests
|
||||
> (skip the fresh sweep), pass `RECIPES="recipe1 recipe2"` to the helper to limit it.
|
||||
|
||||
**(c) Merge + classify.** Combine (a) and (b) into `CANDIDATES`. For each candidate, classify the
|
||||
failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
||||
- **RECIPE bug** (the upgrade is broken) → NOT this skill. Hand off to `/recipe-upgrade` (5a) or
|
||||
note it for the operator. Don't touch the test.
|
||||
- **CI-SERVER bug** (harness/infra wrong) → NOT this skill (that's a `/ci-test-review` cc-ci-side
|
||||
fix, not a test-update). Note it.
|
||||
- **TEST out-of-date** (the recipe legitimately changed upstream and the cc-ci test/overlay asserts
|
||||
old behavior) → **this skill**. Add to `STALE_TESTS` with: recipe, the failing test name, what
|
||||
changed upstream, what the test currently asserts.
|
||||
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
|
||||
|
||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||
|
||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||
Swarm). For each `STALE_TESTS` entry:
|
||||
|
||||
**(a) Set up a dedicated cc-ci clone on a branch** (single-writer — never push `main`, never touch
|
||||
the build loops' clones):
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests-<recipe>
|
||||
cd /home/loops/work/cc-ci-tests-<recipe>
|
||||
git fetch origin && git checkout main && git reset --hard origin/main
|
||||
git checkout -b test/<recipe>-stale-test-$(date -u +%Y%m%d)
|
||||
```
|
||||
(Or reuse `/home/loops/work/cc-ci-fix` on a new branch if it's clean — the rule is a dedicated
|
||||
branch off `main`, in a clone you're not racing the loops on.)
|
||||
|
||||
**(b) Read the failing test + the upstream change.** The test lives in
|
||||
`tests/<recipe>/` (custom tests, overlays, fixtures). Read:
|
||||
- the failing test (the sweep / carry-over names it),
|
||||
- the recipe's current `compose.yml` + upstream changelog (use the per-recipe registry
|
||||
`cc-ci-plan/upstream/<recipe>.md` for the release-notes URLs — the `/upgrade-all` diagnosis
|
||||
already identified what changed),
|
||||
- what the test currently asserts vs what the new version does.
|
||||
|
||||
**(c) Author the minimal update.** Change the test/overlay to assert the **new, correct** behavior
|
||||
— not to weaken it. Examples of a legitimate stale-test fix:
|
||||
- a config key was renamed upstream → update the overlay to the new key.
|
||||
- a default flipped (e.g. `allow_uncategorized_topics` false→true in Discourse 3.5) → update the
|
||||
assertion to the new default, OR make the test explicitly set the old value (preferred — the
|
||||
test then asserts the behavior it cares about, not the default).
|
||||
- an auth method was removed (e.g. Bearer-auth in impress v5.4.0) → drop that assertion, or
|
||||
replace it with the new auth method's assertion (preferred if the new method is the successor).
|
||||
- a readiness endpoint changed → update the readiness gate to the new endpoint.
|
||||
|
||||
**What is NOT allowed:** lowering an assertion's strictness to make a red go green, deleting a test
|
||||
that still applies, skipping a tier that still applies, or changing a `pass` threshold. The fix
|
||||
must make the test **correct for the new version**, not lenient. If you can't fix it without
|
||||
weakening, stop and report `FAILED — test cannot be updated without weakening; needs operator
|
||||
decision` (leave the recipe's stale-test PR comment as-is).
|
||||
|
||||
**(d) Lint the test change** (cc-ci has a lint gate):
|
||||
```
|
||||
ruff check tests/<recipe>/ && ruff format --check tests/<recipe>/
|
||||
```
|
||||
Fix any lint errors you introduce (don't commit red lint).
|
||||
|
||||
### 3. Open the cc-ci test PR (never merge)
|
||||
|
||||
```
|
||||
git add tests/<recipe>/
|
||||
git commit -m "test(<recipe>): update stale test for <upstream-change>
|
||||
|
||||
<test-name> asserted <old-behavior>; <recipe> <new-version> now <new-behavior>.
|
||||
Updated to assert the new behavior. No assertion weakened.
|
||||
|
||||
Stale-test fix for recipe PR <url> (carry-over from /upgrade-all <date>)."
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR via the helper:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
TITLE="test(<recipe>): update stale test for <upstream-change>" \
|
||||
BODY_FILE=/tmp/cc-ci-tests-<recipe>-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
The PR body must: name the stale test, explain what changed upstream, state that no assertion was
|
||||
weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe
|
||||
upgrade is also applied — they're a pair), and say "NOT merged — for operator review."
|
||||
|
||||
Capture the `TEST_PR_URL`.
|
||||
|
||||
### 4. Verify the updated test passes (deterministic; the standard PR + !testme flow)
|
||||
|
||||
A test-update PR is only "working" once cc-ci verifies it green with the new test applied. There
|
||||
are two verification paths — use whichever applies:
|
||||
|
||||
**(a) If the stale test was carry-over from an open recipe upgrade PR** — verify via the paired
|
||||
`!testme` flow (the `/recipe-upgrade --with-tests` step-5b path). The recipe upgrade PR's `!testme`
|
||||
uses the **deployed/main** cc-ci tests, so it can't see your unmerged test branch — you need to
|
||||
run the harness with your cc-ci test branch checked out on cc-ci:
|
||||
```
|
||||
# 1. On cc-ci, check out your test branch in a deploy clone (NOT /root/builder-clone — use a
|
||||
# separate checkout so the running server is unaffected):
|
||||
ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/<recipe>-stale-test-<date> \
|
||||
ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \
|
||||
cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true'
|
||||
# 2. Run the harness against the recipe upgrade PR head, with your test branch as the cc-ci checkout:
|
||||
RECIPE=<recipe> REF=<recipe-upgrade-pr-head-branch> REMOTE_ROOT=/root/cc-ci-test-verify \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh
|
||||
```
|
||||
Green ⇔ the recipe upgrade passes under your corrected test. This is the **paired** verification
|
||||
(the test change + the recipe upgrade together go green).
|
||||
|
||||
**(b) If the stale test is against a recipe already at its current upstream** (no upgrade PR — the
|
||||
test is just stale against current main) — verify the test branch directly:
|
||||
```
|
||||
RECIPE=<recipe> REMOTE_ROOT=/root/cc-ci-test-verify \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh
|
||||
```
|
||||
(No `REF` — runs against the current recipe main, with your updated tests.)
|
||||
|
||||
**(c) Cleanup the verify clone on cc-ci** (mandatory — shared host):
|
||||
```
|
||||
ssh cc-ci 'rm -rf /root/cc-ci-test-verify'
|
||||
```
|
||||
|
||||
**If verification is RED:** iterate the test fix on the same branch (**bounded — ≤3 attempts**) and
|
||||
re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green
|
||||
after 3 verify runs` with the failing evidence. Don't weaken the test to force green.
|
||||
|
||||
> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open,
|
||||
> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests —
|
||||
> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the
|
||||
> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is
|
||||
> open and verified-paired, so the operator knows merging both → green.
|
||||
|
||||
### 5. Tear down any dev deploys (defence-in-depth)
|
||||
|
||||
If your verification path (or diagnosis) deployed a `dev-<recipe>` stack on cc-ci, tear it down:
|
||||
```
|
||||
ssh cc-ci 'D=dev-<recipe>.ci.commoninternet.net; script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
|
||||
script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
|
||||
ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe>"' # should be 0
|
||||
```
|
||||
(The `/upgrade-all` sweep would reap it next run anyway, but clean up yourself.)
|
||||
|
||||
### 6. Report — one line per recipe + summary
|
||||
|
||||
Write `/srv/cc-ci/.cc-ci-logs/tests-update-<YYYY-MM-DD>.md` and print, leading with the PR list:
|
||||
```markdown
|
||||
# cc-ci Stale-Test Update — <YYYY-MM-DD>
|
||||
## Summary
|
||||
- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N
|
||||
## Test PRs to review (NOT merged)
|
||||
| recipe | stale test | upstream change | test PR | verified |
|
||||
|--------|-----------|-----------------|---------|----------|
|
||||
| <recipe> | <test-name> | <what changed> | <url> | GREEN (paired with recipe PR <url>) |
|
||||
## Failed (needs work)
|
||||
- <recipe> — <test-name>: <reason> (test PR <url> left open, red)
|
||||
## Not-stale (dropped — was flaky / real recipe bug / CI-server bug)
|
||||
- <recipe> — <test-name>: <classification> (handed off / noted)
|
||||
```
|
||||
End with the report path and a reminder that **nothing was merged** — the test PRs (and their
|
||||
paired recipe upgrade PRs) await operator review.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream
|
||||
behavior; if that's not possible without weakening, stop and report.
|
||||
- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is
|
||||
operator-merged after review.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify`
|
||||
on the host and `rm -rf` it after.
|
||||
- **Stale ≠ broken.** Only update a test when the **upgrade/recipe is correct** and the **test is
|
||||
wrong** (asserts old behavior). A real upgrade regression is a `/recipe-upgrade` (5a) job; a
|
||||
CI-server bug is a `/ci-test-review` job. This skill fixes tests, not recipes or harness.
|
||||
- **Bounded fixes** — minimal update to the failing assertion/overlay, not a test rewrite.
|
||||
- **Bounded retries** — ≤3 verify runs per test PR; then leave it open + red with evidence.
|
||||
- **Flake ≠ stale** — re-run before classifying as stale; don't author a fix for a flake.
|
||||
- **Serialize on the shared Swarm** — one recipe's verify deploy at a time; tear down before the
|
||||
next. Don't run concurrent with `/upgrade-all` or active build loops.
|
||||
- **Real abra path** throughout (no docker-level bypass); pseudo-TTY wrap every abra call.
|
||||
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
|
||||
@@ -3,108 +3,12 @@ name: cc-ci-update
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update.
|
||||
---
|
||||
|
||||
# cc-ci-update
|
||||
# cc-ci-update (thin wrapper)
|
||||
|
||||
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
|
||||
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
||||
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
|
||||
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
|
||||
The canonical definition of this skill lives in the **opencode** position:
|
||||
|
||||
## What this skill does NOT do
|
||||
**`.opencode/skills/cc-ci-update/SKILL.md`**
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This is cc-ci
|
||||
**self-maintenance**, not recipe maintenance.
|
||||
- **Does NOT merge** any PR — operator merges everything.
|
||||
- **Does NOT skip the health gate** — if the server rebuild regresses, stop and roll back; do not
|
||||
proceed to the tests sweep on a sick host.
|
||||
|
||||
## Arguments
|
||||
|
||||
None. (If you only want one of the two updates, invoke the sub-skill directly:
|
||||
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
|
||||
|
||||
## Preconditions
|
||||
|
||||
Both sub-skills' preconditions hold:
|
||||
- `ssh cc-ci` works (root, tailnet `100.95.31.88`).
|
||||
- Operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server (the server-update
|
||||
half needs it). Pick a **no-CI window** — no `!testme` / nightly sweep running.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (PRs + `!testme`).
|
||||
- The shared Swarm is quiescent enough for the tests sweep's verification deploys.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Run `/cc-ci-server-update` (host first)
|
||||
|
||||
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
||||
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
|
||||
gate** (no failed units, infra up, endpoints 200, disk free). See
|
||||
`.claude/skills/cc-ci-server-update/SKILL.md`.
|
||||
|
||||
**Gate on the result before continuing:**
|
||||
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
|
||||
- `ROLLED-BACK` / `FAILED` → **STOP.** Do not run the tests sweep on a sick or rolled-back host.
|
||||
Report the server-update failure and exit. The operator fixes the server bump first; re-run
|
||||
`/cc-ci-update` (or just `/cc-ci-tests-update`) once the server is healthy on the new (or
|
||||
reverted) nixpkgs.
|
||||
|
||||
> **Why server-first?** The tests sweep runs the harness on cc-ci, and the harness depends on the
|
||||
> host's python/pytest/playwright/docker toolchain — all from nixpkgs. Updating the host first means
|
||||
> the test sweep runs against the **new** toolchain, so a green verification is meaningful for the
|
||||
> post-update state (not the pre-update one). It also avoids the edge case where a nixpkgs bump
|
||||
> changes a tool the tests depend on, invalidating a test sweep you just did. Server bump is the
|
||||
> foundational change; tests ride on top.
|
||||
|
||||
### 2. Run `/cc-ci-tests-update` (tests second, on the now-updated host)
|
||||
|
||||
Run the full `/cc-ci-tests-update` procedure: build the stale-test candidate list (carry-over from
|
||||
`/upgrade-all` reports + fresh `/ci-test-review` sweep) → classify each failure → for each
|
||||
genuinely-stale test, author the minimal update, open a cc-ci test PR, verify via the harness
|
||||
(paired with the recipe upgrade PR if carry-over) → tear down dev deploys → report. See
|
||||
`.claude/skills/cc-ci-tests-update/SKILL.md`.
|
||||
|
||||
**No additional gate between recipes** — the tests skill already serializes per-recipe and tears
|
||||
down each verify deploy. Just run it to completion.
|
||||
|
||||
### 3. Combined report
|
||||
|
||||
After both halves are done, print + write a combined summary to
|
||||
`/srv/cc-ci/.cc-ci-logs/cc-ci-update-<YYYY-MM-DD>.md`:
|
||||
```markdown
|
||||
# cc-ci Update — <YYYY-MM-DD>
|
||||
## Server (host dependency bump)
|
||||
- nixpkgs: <old-rev[:8]> → <new-rev[:8]> · sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||
- Rebuild: SUCCESS / ROLLED-BACK / FAILED · host healthy: yes/no
|
||||
- Server PR: <url> (NOT merged)
|
||||
## Tests (stale-test sweep)
|
||||
- Swept: N · Stale found: N · Test PRs opened + verified: N · Failed: N
|
||||
- Test PRs: <list of urls> (NOT merged)
|
||||
## Overall
|
||||
- <one-line headline>
|
||||
```
|
||||
End with: **nothing was merged** — the server PR + test PRs (+ any paired recipe upgrade PRs) await
|
||||
operator review.
|
||||
|
||||
## Guardrails
|
||||
|
||||
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
|
||||
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
|
||||
health gate is a hard go/no-go for step 2.
|
||||
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
|
||||
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
|
||||
concurrent with `/upgrade-all` or active build loops.
|
||||
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
|
||||
- **Never weaken a test** (tests half) — a stale test is updated to the new correct behavior, not
|
||||
loosened.
|
||||
- **Stable channel only** for the server bump (never `nixos-unstable`).
|
||||
|
||||
## Relationship to automatic updates
|
||||
|
||||
This is the **manual trigger** for what should ideally be two periodic timers:
|
||||
- `cc-ci-server-update.timer` (host deps) — e.g. monthly, or on nixpkgs CVE advisories.
|
||||
- `cc-ci-tests-update.timer` (stale tests) — e.g. weekly, after the `/upgrade-all` run so it can
|
||||
close the loop on that week's commented stale tests.
|
||||
|
||||
If/when those timers exist, `/cc-ci-update` is the "run both now, in the right order, with the
|
||||
gate" skill — either timer can also invoke its single sub-skill directly. The order + the gate
|
||||
between them (server healthy before tests) is the only thing this umbrella adds.
|
||||
Read that file for the full procedure. This `.claude/skills/` copy is kept as a
|
||||
thin pointer for Claude Code compatibility; opencode loads the canonical
|
||||
definition from `.opencode/skills/` directly.
|
||||
|
||||
@@ -0,0 +1,269 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update
|
||||
|
||||
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
|
||||
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
|
||||
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
|
||||
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
|
||||
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
|
||||
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
|
||||
> different machine with its own flake — it is NOT touched here. If you want to update the
|
||||
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
|
||||
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
|
||||
> operator). This skill is only about the cc-ci **server**.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
|
||||
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||
- **Does NOT merge** the cc-ci PR — operator merges after review.
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
|
||||
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
|
||||
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
|
||||
is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
|
||||
oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
|
||||
nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
|
||||
```
|
||||
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
|
||||
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
|
||||
```
|
||||
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
|
||||
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
|
||||
(NOT in git). The rebuild needs them — see the rebuild procedure below.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot current state (so you can compare + roll back)
|
||||
|
||||
```
|
||||
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
|
||||
df -h / | tail -1; nixos-version; readlink /run/current-system'
|
||||
```
|
||||
Record: the current `nixos-version`, the current-system store path, the list of active infra
|
||||
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
|
||||
failed or an endpoint down gets compared against this.
|
||||
|
||||
Also record the **current flake input revs** (the "before" of what you're bumping):
|
||||
```
|
||||
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
|
||||
```
|
||||
|
||||
### 2. Prepare a working clone of the cc-ci repo (single-writer)
|
||||
|
||||
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
|
||||
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
|
||||
fresh one:
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
|
||||
cd /home/loops/work/cc-ci-server-update
|
||||
git fetch origin && git checkout main && git reset --hard origin/main
|
||||
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
|
||||
```
|
||||
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
|
||||
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
|
||||
|
||||
### 3. Bump the flake inputs (deliberate, not drift)
|
||||
|
||||
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
|
||||
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
|
||||
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
|
||||
blind `nix flake update --relock-all`.
|
||||
|
||||
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
|
||||
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
|
||||
**stable channel revision** that's compatible:
|
||||
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
|
||||
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
|
||||
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
|
||||
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
|
||||
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
|
||||
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
|
||||
to staying on the same channel unless the operator asks for the jump.
|
||||
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
|
||||
stable nixpkgs. Unstable is for experiments, not production.
|
||||
|
||||
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
|
||||
and check whether `sops-nix` needs a matching bump:
|
||||
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
|
||||
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
|
||||
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
|
||||
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
|
||||
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
|
||||
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
|
||||
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
|
||||
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
|
||||
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
|
||||
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
|
||||
|
||||
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
|
||||
host):
|
||||
```
|
||||
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
|
||||
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
git diff flake.nix flake.lock | head -40
|
||||
```
|
||||
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
|
||||
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||
|
||||
### 4. Commit + open the cc-ci PR (never merge)
|
||||
|
||||
```
|
||||
git add flake.nix flake.lock
|
||||
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
|
||||
sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||
<one-line reason — CVE patch / channel bump / sops-nix compat>"
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR via the helper:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
|
||||
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||
breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.")
|
||||
|
||||
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||
proceed (the rebuild doesn't depend on the lint gate passing).
|
||||
|
||||
### 5. Deploy to the live cc-ci host (no-CI window)
|
||||
|
||||
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
|
||||
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
|
||||
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
|
||||
|
||||
**(a) Stage the PR branch on the host:**
|
||||
```
|
||||
# from the orchestrator, on the PR branch, clean tree:
|
||||
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
|
||||
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
|
||||
```
|
||||
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
|
||||
flake — see (c).)
|
||||
|
||||
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
|
||||
```
|
||||
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
|
||||
```
|
||||
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
|
||||
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
|
||||
|
||||
**(c) Build first (catch errors before switching):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
|
||||
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
|
||||
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
|
||||
That's expected; this is a real nixpkgs bump.
|
||||
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
|
||||
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
|
||||
module option renames across channels (check the release notes), python package removals, a
|
||||
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||
|
||||
**(d) Switch (the live deploy):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
|
||||
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
|
||||
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
|
||||
may be transient; a proxy failure is real).
|
||||
|
||||
### 6. Verify the server is healthy (health gate — do not skip)
|
||||
|
||||
```
|
||||
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
|
||||
echo "=== failed units ==="; systemctl --failed --no-legend; \
|
||||
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
|
||||
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
|
||||
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
|
||||
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
|
||||
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
|
||||
echo "=== disk ==="; df -h / | tail -1'
|
||||
```
|
||||
Compare against the step-1 baseline:
|
||||
- `nixos-version` moved (or the current-system path changed) — the bump landed.
|
||||
- **No failed units** (the `--failed` output is empty).
|
||||
- All 7 infra services `active` (or `active (exited)` for the oneshots).
|
||||
- `docker service ls` shows no `0/N` replicas (everything converged).
|
||||
- Endpoints `200` (traefik `/api/version`, dashboard root).
|
||||
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
|
||||
|
||||
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
|
||||
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
|
||||
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||
while you fix). Report the rollback in the PR.
|
||||
|
||||
### 7. Report
|
||||
|
||||
Print + record a one-line result + the PR URL. Write a short note to
|
||||
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>`
|
||||
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||
- `RESULT: FAILED — <step>: <reason>`
|
||||
|
||||
Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator
|
||||
reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation +
|
||||
the path to re-apply, not a gate on the deploy).
|
||||
|
||||
### 8. Optional cold-boot proof
|
||||
|
||||
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
|
||||
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
|
||||
if the operator wants it, and only outside a CI window:
|
||||
```
|
||||
ssh cc-ci 'systemctl reboot'
|
||||
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
|
||||
```
|
||||
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
|
||||
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
|
||||
are tracked in its own journal; a note in the server-update report is enough).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
|
||||
go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
|
||||
reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
|
||||
step-1 check), but don't gate on a second authorization.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||
`/root/cc-ci-deploy` instead).
|
||||
- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record.
|
||||
- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first.
|
||||
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||
to the baseline. Roll back on regression.
|
||||
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
|
||||
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
|
||||
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
|
||||
generation without touching the flake input revs.
|
||||
|
||||
## Relationship to an automatic update
|
||||
|
||||
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
|
||||
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
|
||||
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
|
||||
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
|
||||
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
|
||||
without a human prompt."
|
||||
@@ -0,0 +1,244 @@
|
||||
---
|
||||
name: cc-ci-tests-update
|
||||
description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update.
|
||||
---
|
||||
|
||||
# cc-ci-tests-update
|
||||
|
||||
The fleet-wide **stale-test cleanup** for cc-ci. The weekly `/upgrade-all` cron deliberately does
|
||||
**not** auto-edit cc-ci tests — it runs in DEFAULT mode and, where a recipe's existing test is
|
||||
genuinely stale against a new upstream version, it leaves an **explanatory comment** on the recipe
|
||||
PR and stops (operator decides). Over time those commented stale tests accumulate. This skill
|
||||
**closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale,
|
||||
authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via
|
||||
the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never
|
||||
weakens a test and never merges.
|
||||
|
||||
> **Relationship to the other skills.** This is the fleet-wide version of the per-recipe
|
||||
> `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification
|
||||
> branch of `/ci-test-review`. It reuses both skills' discipline + helpers. The difference is
|
||||
> **scope**: those skills react to a failure they happened to hit; this one **proactively sweeps
|
||||
> the whole fleet** for stale tests and fixes them in one pass.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This skill fixes
|
||||
the **tests**, not the recipes. (It may *discover* a stale test by re-running a recipe upgrade
|
||||
PR's `!testme`, but the recipe bump itself is owned by `/upgrade-all`.)
|
||||
- **Does NOT touch the cc-ci server host / flake** — that's `/cc-ci-server-update`.
|
||||
- **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is
|
||||
broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review`
|
||||
job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**.
|
||||
- **Does NOT merge** any PR — operator merges after review.
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
- `ssh cc-ci` works (root). Same access notes as `/recipe-upgrade` / `/ci-test-review`.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening cc-ci test PRs + posting `!testme`).
|
||||
- The shared Swarm is **quiescent enough** for test runs — each stale-test verification deploys a
|
||||
recipe, so don't run this concurrent with a `/upgrade-all` or active build loops. Serialize.
|
||||
- abra over ssh needs the pseudo-TTY wrapper (see the box in `/recipe-upgrade`): every `abra` call
|
||||
is `ssh cc-ci 'script -qec "abra <args> -n" /dev/null'`.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Build the stale-test candidate list (sweep — deterministic, no AI judgement yet)
|
||||
|
||||
The candidates are recipes whose cc-ci test is **potentially stale**. Two sources, merge both:
|
||||
|
||||
**(a) Carry-over from `/upgrade-all` — the commented stale tests.** Each weekly run's report has a
|
||||
"PRs where a test looks stale" section (recipes where `!testme` went RED on a stale test and got a
|
||||
PR comment). Read the latest report + a few weeks back:
|
||||
```
|
||||
grep -A2 "test.*stale\|stale.*test\|SUCCESS-PENDING-TESTS" /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | tail -40
|
||||
```
|
||||
Each line names a recipe + the stale test + the PR URL. These are **known stale** (already
|
||||
diagnosed by the upgrader) — they go straight to the fix list.
|
||||
|
||||
**(b) Fresh sweep — re-run the test suite across all maintained recipes.** Use the
|
||||
`/ci-test-review` deterministic sweep helper to find failures that aren't already on the carry-over
|
||||
list:
|
||||
```
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh
|
||||
```
|
||||
It runs the full harness per recipe on cc-ci and writes a JSON summary to
|
||||
`/srv/cc-ci/.cc-ci-logs/ci-test-review-<runid>.json`. Parse it for `fail` verdicts.
|
||||
|
||||
> **Concurrency note:** a full sweep is ~20 recipes × full suite — it takes a while and uses the
|
||||
> shared Swarm. Run it when the host is otherwise idle. If you only want the carry-over stale tests
|
||||
> (skip the fresh sweep), pass `RECIPES="recipe1 recipe2"` to the helper to limit it.
|
||||
|
||||
**(c) Merge + classify.** Combine (a) and (b) into `CANDIDATES`. For each candidate, classify the
|
||||
failure (AI — this is the `ci-test-review` step-3 diagnosis):
|
||||
- **RECIPE bug** (the upgrade is broken) → NOT this skill. Hand off to `/recipe-upgrade` (5a) or
|
||||
note it for the operator. Don't touch the test.
|
||||
- **CI-SERVER bug** (harness/infra wrong) → NOT this skill (that's a `/ci-test-review` cc-ci-side
|
||||
fix, not a test-update). Note it.
|
||||
- **TEST out-of-date** (the recipe legitimately changed upstream and the cc-ci test/overlay asserts
|
||||
old behavior) → **this skill**. Add to `STALE_TESTS` with: recipe, the failing test name, what
|
||||
changed upstream, what the test currently asserts.
|
||||
- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky).
|
||||
|
||||
### 2. For each stale test — author the minimal test update (AI; never weaken)
|
||||
|
||||
Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared
|
||||
Swarm). For each `STALE_TESTS` entry:
|
||||
|
||||
**(a) Set up a dedicated cc-ci clone on a branch** (single-writer — never push `main`, never touch
|
||||
the build loops' clones):
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests-<recipe>
|
||||
cd /home/loops/work/cc-ci-tests-<recipe>
|
||||
git fetch origin && git checkout main && git reset --hard origin/main
|
||||
git checkout -b test/<recipe>-stale-test-$(date -u +%Y%m%d)
|
||||
```
|
||||
(Or reuse `/home/loops/work/cc-ci-fix` on a new branch if it's clean — the rule is a dedicated
|
||||
branch off `main`, in a clone you're not racing the loops on.)
|
||||
|
||||
**(b) Read the failing test + the upstream change.** The test lives in
|
||||
`tests/<recipe>/` (custom tests, overlays, fixtures). Read:
|
||||
- the failing test (the sweep / carry-over names it),
|
||||
- the recipe's current `compose.yml` + upstream changelog (use the per-recipe registry
|
||||
`cc-ci-plan/upstream/<recipe>.md` for the release-notes URLs — the `/upgrade-all` diagnosis
|
||||
already identified what changed),
|
||||
- what the test currently asserts vs what the new version does.
|
||||
|
||||
**(c) Author the minimal update.** Change the test/overlay to assert the **new, correct** behavior
|
||||
— not to weaken it. Examples of a legitimate stale-test fix:
|
||||
- a config key was renamed upstream → update the overlay to the new key.
|
||||
- a default flipped (e.g. `allow_uncategorized_topics` false→true in Discourse 3.5) → update the
|
||||
assertion to the new default, OR make the test explicitly set the old value (preferred — the
|
||||
test then asserts the behavior it cares about, not the default).
|
||||
- an auth method was removed (e.g. Bearer-auth in impress v5.4.0) → drop that assertion, or
|
||||
replace it with the new auth method's assertion (preferred if the new method is the successor).
|
||||
- a readiness endpoint changed → update the readiness gate to the new endpoint.
|
||||
|
||||
**What is NOT allowed:** lowering an assertion's strictness to make a red go green, deleting a test
|
||||
that still applies, skipping a tier that still applies, or changing a `pass` threshold. The fix
|
||||
must make the test **correct for the new version**, not lenient. If you can't fix it without
|
||||
weakening, stop and report `FAILED — test cannot be updated without weakening; needs operator
|
||||
decision` (leave the recipe's stale-test PR comment as-is).
|
||||
|
||||
**(d) Lint the test change** (cc-ci has a lint gate):
|
||||
```
|
||||
ruff check tests/<recipe>/ && ruff format --check tests/<recipe>/
|
||||
```
|
||||
Fix any lint errors you introduce (don't commit red lint).
|
||||
|
||||
### 3. Open the cc-ci test PR (never merge)
|
||||
|
||||
```
|
||||
git add tests/<recipe>/
|
||||
git commit -m "test(<recipe>): update stale test for <upstream-change>
|
||||
|
||||
<test-name> asserted <old-behavior>; <recipe> <new-version> now <new-behavior>.
|
||||
Updated to assert the new behavior. No assertion weakened.
|
||||
|
||||
Stale-test fix for recipe PR <url> (carry-over from /upgrade-all <date>)."
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR via the helper:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
TITLE="test(<recipe>): update stale test for <upstream-change>" \
|
||||
BODY_FILE=/tmp/cc-ci-tests-<recipe>-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
The PR body must: name the stale test, explain what changed upstream, state that no assertion was
|
||||
weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe
|
||||
upgrade is also applied — they're a pair), and say "NOT merged — for operator review."
|
||||
|
||||
Capture the `TEST_PR_URL`.
|
||||
|
||||
### 4. Verify the updated test passes (deterministic; the standard PR + !testme flow)
|
||||
|
||||
A test-update PR is only "working" once cc-ci verifies it green with the new test applied. There
|
||||
are two verification paths — use whichever applies:
|
||||
|
||||
**(a) If the stale test was carry-over from an open recipe upgrade PR** — verify via the paired
|
||||
`!testme` flow (the `/recipe-upgrade --with-tests` step-5b path). The recipe upgrade PR's `!testme`
|
||||
uses the **deployed/main** cc-ci tests, so it can't see your unmerged test branch — you need to
|
||||
run the harness with your cc-ci test branch checked out on cc-ci:
|
||||
```
|
||||
# 1. On cc-ci, check out your test branch in a deploy clone (NOT /root/builder-clone — use a
|
||||
# separate checkout so the running server is unaffected):
|
||||
ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/<recipe>-stale-test-<date> \
|
||||
ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \
|
||||
cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true'
|
||||
# 2. Run the harness against the recipe upgrade PR head, with your test branch as the cc-ci checkout:
|
||||
RECIPE=<recipe> REF=<recipe-upgrade-pr-head-branch> REMOTE_ROOT=/root/cc-ci-test-verify \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh
|
||||
```
|
||||
Green ⇔ the recipe upgrade passes under your corrected test. This is the **paired** verification
|
||||
(the test change + the recipe upgrade together go green).
|
||||
|
||||
**(b) If the stale test is against a recipe already at its current upstream** (no upgrade PR — the
|
||||
test is just stale against current main) — verify the test branch directly:
|
||||
```
|
||||
RECIPE=<recipe> REMOTE_ROOT=/root/cc-ci-test-verify \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh
|
||||
```
|
||||
(No `REF` — runs against the current recipe main, with your updated tests.)
|
||||
|
||||
**(c) Cleanup the verify clone on cc-ci** (mandatory — shared host):
|
||||
```
|
||||
ssh cc-ci 'rm -rf /root/cc-ci-test-verify'
|
||||
```
|
||||
|
||||
**If verification is RED:** iterate the test fix on the same branch (**bounded — ≤3 attempts**) and
|
||||
re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green
|
||||
after 3 verify runs` with the failing evidence. Don't weaken the test to force green.
|
||||
|
||||
> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open,
|
||||
> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests —
|
||||
> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the
|
||||
> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is
|
||||
> open and verified-paired, so the operator knows merging both → green.
|
||||
|
||||
### 5. Tear down any dev deploys (defence-in-depth)
|
||||
|
||||
If your verification path (or diagnosis) deployed a `dev-<recipe>` stack on cc-ci, tear it down:
|
||||
```
|
||||
ssh cc-ci 'D=dev-<recipe>.ci.commoninternet.net; script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \
|
||||
script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2'
|
||||
ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-<recipe>"' # should be 0
|
||||
```
|
||||
(The `/upgrade-all` sweep would reap it next run anyway, but clean up yourself.)
|
||||
|
||||
### 6. Report — one line per recipe + summary
|
||||
|
||||
Write `/srv/cc-ci/.cc-ci-logs/tests-update-<YYYY-MM-DD>.md` and print, leading with the PR list:
|
||||
```markdown
|
||||
# cc-ci Stale-Test Update — <YYYY-MM-DD>
|
||||
## Summary
|
||||
- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N
|
||||
## Test PRs to review (NOT merged)
|
||||
| recipe | stale test | upstream change | test PR | verified |
|
||||
|--------|-----------|-----------------|---------|----------|
|
||||
| <recipe> | <test-name> | <what changed> | <url> | GREEN (paired with recipe PR <url>) |
|
||||
## Failed (needs work)
|
||||
- <recipe> — <test-name>: <reason> (test PR <url> left open, red)
|
||||
## Not-stale (dropped — was flaky / real recipe bug / CI-server bug)
|
||||
- <recipe> — <test-name>: <classification> (handed off / noted)
|
||||
```
|
||||
End with the report path and a reminder that **nothing was merged** — the test PRs (and their
|
||||
paired recipe upgrade PRs) await operator review.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream
|
||||
behavior; if that's not possible without weakening, stop and report.
|
||||
- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is
|
||||
operator-merged after review.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify`
|
||||
on the host and `rm -rf` it after.
|
||||
- **Stale ≠ broken.** Only update a test when the **upgrade/recipe is correct** and the **test is
|
||||
wrong** (asserts old behavior). A real upgrade regression is a `/recipe-upgrade` (5a) job; a
|
||||
CI-server bug is a `/ci-test-review` job. This skill fixes tests, not recipes or harness.
|
||||
- **Bounded fixes** — minimal update to the failing assertion/overlay, not a test rewrite.
|
||||
- **Bounded retries** — ≤3 verify runs per test PR; then leave it open + red with evidence.
|
||||
- **Flake ≠ stale** — re-run before classifying as stale; don't author a fix for a flake.
|
||||
- **Serialize on the shared Swarm** — one recipe's verify deploy at a time; tear down before the
|
||||
next. Don't run concurrent with `/upgrade-all` or active build loops.
|
||||
- **Real abra path** throughout (no docker-level bypass); pseudo-TTY wrap every abra call.
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: cc-ci-update
|
||||
description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update.
|
||||
---
|
||||
|
||||
# cc-ci-update
|
||||
|
||||
The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
|
||||
rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
|
||||
is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
|
||||
SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This is cc-ci
|
||||
**self-maintenance**, not recipe maintenance.
|
||||
- **Does NOT merge** any PR — operator merges everything.
|
||||
- **Does NOT skip the health gate** — if the server rebuild regresses, stop and roll back; do not
|
||||
proceed to the tests sweep on a sick host.
|
||||
|
||||
## Arguments
|
||||
|
||||
None. (If you only want one of the two updates, invoke the sub-skill directly:
|
||||
`/cc-ci-server-update` or `/cc-ci-tests-update`.)
|
||||
|
||||
## Preconditions
|
||||
|
||||
Both sub-skills' preconditions hold:
|
||||
- `ssh cc-ci` works (root, tailnet `100.95.31.88`).
|
||||
- Operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server (the server-update
|
||||
half needs it). Pick a **no-CI window** — no `!testme` / nightly sweep running.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (PRs + `!testme`).
|
||||
- The shared Swarm is quiescent enough for the tests sweep's verification deploys.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Run `/cc-ci-server-update` (host first)
|
||||
|
||||
Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
|
||||
sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
|
||||
gate** (no failed units, infra up, endpoints 200, disk free). See
|
||||
`.claude/skills/cc-ci-server-update/SKILL.md`.
|
||||
|
||||
**Gate on the result before continuing:**
|
||||
- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
|
||||
- `ROLLED-BACK` / `FAILED` → **STOP.** Do not run the tests sweep on a sick or rolled-back host.
|
||||
Report the server-update failure and exit. The operator fixes the server bump first; re-run
|
||||
`/cc-ci-update` (or just `/cc-ci-tests-update`) once the server is healthy on the new (or
|
||||
reverted) nixpkgs.
|
||||
|
||||
> **Why server-first?** The tests sweep runs the harness on cc-ci, and the harness depends on the
|
||||
> host's python/pytest/playwright/docker toolchain — all from nixpkgs. Updating the host first means
|
||||
> the test sweep runs against the **new** toolchain, so a green verification is meaningful for the
|
||||
> post-update state (not the pre-update one). It also avoids the edge case where a nixpkgs bump
|
||||
> changes a tool the tests depend on, invalidating a test sweep you just did. Server bump is the
|
||||
> foundational change; tests ride on top.
|
||||
|
||||
### 2. Run `/cc-ci-tests-update` (tests second, on the now-updated host)
|
||||
|
||||
Run the full `/cc-ci-tests-update` procedure: build the stale-test candidate list (carry-over from
|
||||
`/upgrade-all` reports + fresh `/ci-test-review` sweep) → classify each failure → for each
|
||||
genuinely-stale test, author the minimal update, open a cc-ci test PR, verify via the harness
|
||||
(paired with the recipe upgrade PR if carry-over) → tear down dev deploys → report. See
|
||||
`.claude/skills/cc-ci-tests-update/SKILL.md`.
|
||||
|
||||
**No additional gate between recipes** — the tests skill already serializes per-recipe and tears
|
||||
down each verify deploy. Just run it to completion.
|
||||
|
||||
### 3. Combined report
|
||||
|
||||
After both halves are done, print + write a combined summary to
|
||||
`/srv/cc-ci/.cc-ci-logs/cc-ci-update-<YYYY-MM-DD>.md`:
|
||||
```markdown
|
||||
# cc-ci Update — <YYYY-MM-DD>
|
||||
## Server (host dependency bump)
|
||||
- nixpkgs: <old-rev[:8]> → <new-rev[:8]> · sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||
- Rebuild: SUCCESS / ROLLED-BACK / FAILED · host healthy: yes/no
|
||||
- Server PR: <url> (NOT merged)
|
||||
## Tests (stale-test sweep)
|
||||
- Swept: N · Stale found: N · Test PRs opened + verified: N · Failed: N
|
||||
- Test PRs: <list of urls> (NOT merged)
|
||||
## Overall
|
||||
- <one-line headline>
|
||||
```
|
||||
End with: **nothing was merged** — the server PR + test PRs (+ any paired recipe upgrade PRs) await
|
||||
operator review.
|
||||
|
||||
## Guardrails
|
||||
|
||||
Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
|
||||
- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
|
||||
health gate is a hard go/no-go for step 2.
|
||||
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
|
||||
- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
|
||||
concurrent with `/upgrade-all` or active build loops.
|
||||
- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
|
||||
- **Never weaken a test** (tests half) — a stale test is updated to the new correct behavior, not
|
||||
loosened.
|
||||
- **Stable channel only** for the server bump (never `nixos-unstable`).
|
||||
|
||||
## Relationship to automatic updates
|
||||
|
||||
This is the **manual trigger** for what should ideally be two periodic timers:
|
||||
- `cc-ci-server-update.timer` (host deps) — e.g. monthly, or on nixpkgs CVE advisories.
|
||||
- `cc-ci-tests-update.timer` (stale tests) — e.g. weekly, after the `/upgrade-all` run so it can
|
||||
close the loop on that week's commented stale tests.
|
||||
|
||||
If/when those timers exist, `/cc-ci-update` is the "run both now, in the right order, with the
|
||||
gate" skill — either timer can also invoke its single sub-skill directly. The order + the gate
|
||||
between them (server healthy before tests) is the only thing this umbrella adds.
|
||||
Reference in New Issue
Block a user