diff --git a/.claude/skills/cc-ci-server-update/SKILL.md b/.claude/skills/cc-ci-server-update/SKILL.md index e042e29..7fbba78 100644 --- a/.claude/skills/cc-ci-server-update/SKILL.md +++ b/.claude/skills/cc-ci-server-update/SKILL.md @@ -1,268 +1,14 @@ --- name: cc-ci-server-update -description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (operator-authorized; picked a no-CI window), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. +description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. --- -# cc-ci-server-update +# cc-ci-server-update (thin wrapper) -Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their -latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and -confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update` -+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift. +The canonical definition of this skill lives in the **opencode** position: -> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake -> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`). -> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a -> different machine with its own flake — it is NOT touched here. If you want to update the -> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` -> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the -> operator). This skill is only about the cc-ci **server**. +**`.opencode/skills/cc-ci-server-update/SKILL.md`** -## What this skill does NOT do - -- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. -- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or - `/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`). -- **Does NOT update the orchestrator host** — only the cc-ci server. -- **Does NOT merge** the cc-ci PR — operator merges after review. - -## Preconditions / access - -- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL - / `cc-ci-tailscaled` proxy) — do NOT proceed. -- The operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server. A switch - cycles the abra reconcile oneshots (`deploy-proxy`, `warm-keycloak`, …) — pick a **no-CI window** - (no `!testme` / nightly sweep running). Check first: - ``` - ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task? - ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight? - ``` - If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI. -- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR). -- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml` - (NOT in git). The rebuild needs them — see the rebuild procedure below. - -## Procedure - -### 1. Snapshot current state (so you can compare + roll back) - -``` -ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \ - deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \ - df -h / | tail -1; nixos-version; readlink /run/current-system' -``` -Record: the current `nixos-version`, the current-system store path, the list of active infra -services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit -failed or an endpoint down gets compared against this. - -Also record the **current flake input revs** (the "before" of what you're bumping): -``` -ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head -``` - -### 2. Prepare a working clone of the cc-ci repo (single-writer) - -Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops' -working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a -fresh one: -``` -git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update -cd /home/loops/work/cc-ci-server-update -git fetch origin && git checkout main && git reset --hard origin/main -git checkout -b chore/server-flake-update-$(date -u +%Y%m%d) -``` -(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new -branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.) - -### 3. Bump the flake inputs (deliberate, not drift) - -The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in -`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that -nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a -blind `nix flake update --relock-all`. - -**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's -runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest -**stable channel revision** that's compatible: -- Default: move to the latest revision of the **same channel** the orchestrator host tracks - (`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current - tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref). -- If a newer stable channel is out and the operator wants it, use that — but treat a **channel - jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes - (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default - to staying on the same channel unless the operator asks for the jump. -- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a - stable nixpkgs. Unstable is for experiments, not production. - -**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`), -and check whether `sops-nix` needs a matching bump: -- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own - rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g. - the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After - editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or - `nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the - lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that - still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects). -- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned. -- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment - style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel"). - -**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the -host): -``` -cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch) -nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock -nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' -git diff flake.nix flake.lock | head -40 -``` -Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the -orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here -(meaningfully). The real build happens on the cc-ci host in step 5. - -### 4. Commit + open the cc-ci PR (never merge) - -``` -git add flake.nix flake.lock -git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d)) - -nixpkgs: → (nixos-) -sops-nix: → -" -git push -u origin HEAD -``` -Open the PR via the helper: -``` -set -a; . /srv/cc-ci/.testenv; set +a -TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \ -BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \ -bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh -``` -(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any -breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review. -Live rebuild to follow on operator go-ahead.") - -Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a -flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and -proceed (the rebuild doesn't depend on the lint gate passing). - -### 5. Deploy to the live cc-ci host (operator-authorized, no-CI window) - -This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server -has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The -deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it. - -**(a) Stage the PR branch on the host:** -``` -# from the orchestrator, on the PR branch, clean tree: -rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/ -ssh cc-ci 'chown -R root:root /root/cc-ci-deploy' -``` -(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path -flake — see (c).) - -**(b) Copy the operator-held sops secrets** (NOT in git, required for the build): -``` -ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml' -``` -(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds. -Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.) - -**(c) Build first (catch errors before switching):** -``` -ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30 -``` -- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`). -- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a - longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs). - That's expected; this is a real nixpkgs bump. -- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed - package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS - module option renames across channels (check the release notes), python package removals, a - sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev). - -**(d) Switch (the live deploy):** -``` -ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30 -``` -This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots -(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that -fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip -may be transient; a proxy failure is real). - -### 6. Verify the server is healthy (health gate — do not skip) - -``` -ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \ - echo "=== failed units ==="; systemctl --failed --no-legend; \ - echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \ - deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \ - echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \ - echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \ - echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \ - curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \ - echo "=== disk ==="; df -h / | tail -1' -``` -Compare against the step-1 baseline: -- `nixos-version` moved (or the current-system path changed) — the bump landed. -- **No failed units** (the `--failed` output is empty). -- All 7 infra services `active` (or `active (exited)` for the oneshots). -- `docker service ls` shows no `0/N` replicas (everything converged). -- Endpoints `200` (traefik `/api/version`, dashboard root). -- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually). - -If **anything regressed**: diagnose. A real regression → roll back to the previous generation: -`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the -flake input change on the PR branch is still your source of truth, but the running host reverts -while you fix). Report the rollback in the PR. - -### 7. Report - -Print + record a one-line result + the PR URL. Write a short note to -`/srv/cc-ci/.cc-ci-logs/server-update-.md`: -- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs , no failed units, endpoints 200, PR: ` -- `RESULT: ROLLED-BACK — nixpkgs rebuild failed/ regressed (); host reverted to ; PR left open` -- `RESULT: FAILED — : ` - -Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator -reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation + -the path to re-apply, not a gate on the deploy). - -### 8. Optional cold-boot proof - -A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves -it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this -if the operator wants it, and only outside a CI window: -``` -ssh cc-ci 'systemctl reboot' -# wait ~60s, re-check ssh cc-ci + the step-6 health gate -``` -Record the new `boot_id` if you do this (it's a genuine reboot — log it to -`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots -are tracked in its own journal; a note in the server-update report is enough). - -## Guardrails - -- **Operator-authorized live rebuild, no-CI window.** A `nixos-rebuild switch` cycles reconcile - oneshots and can disrupt an in-flight `!testme`. Check the host is quiescent first (step 1) and - get the operator's go-ahead. -- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the - build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to - `/root/cc-ci-deploy` instead). -- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record. -- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first. -- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare - to the baseline. Roll back on regression. -- **Stable channel only** for the cc-ci server (never `nixos-unstable`). -- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them. -- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only. -- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior - generation without touching the flake input revs. - -## Relationship to an automatic update - -This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci -flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added -(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke -this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The -skill stays as the authoritative procedure either way — automatic just means "run on a schedule -without a human prompt." +Read that file for the full procedure. This `.claude/skills/` copy is kept as a +thin pointer for Claude Code compatibility; opencode loads the canonical +definition from `.opencode/skills/` directly. diff --git a/.claude/skills/cc-ci-tests-update/SKILL.md b/.claude/skills/cc-ci-tests-update/SKILL.md index 3addcd5..2e8f71d 100644 --- a/.claude/skills/cc-ci-tests-update/SKILL.md +++ b/.claude/skills/cc-ci-tests-update/SKILL.md @@ -3,242 +3,12 @@ name: cc-ci-tests-update description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update. --- -# cc-ci-tests-update +# cc-ci-tests-update (thin wrapper) -The fleet-wide **stale-test cleanup** for cc-ci. The weekly `/upgrade-all` cron deliberately does -**not** auto-edit cc-ci tests — it runs in DEFAULT mode and, where a recipe's existing test is -genuinely stale against a new upstream version, it leaves an **explanatory comment** on the recipe -PR and stops (operator decides). Over time those commented stale tests accumulate. This skill -**closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale, -authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via -the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never -weakens a test and never merges. +The canonical definition of this skill lives in the **opencode** position: -> **Relationship to the other skills.** This is the fleet-wide version of the per-recipe -> `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification -> branch of `/ci-test-review`. It reuses both skills' discipline + helpers. The difference is -> **scope**: those skills react to a failure they happened to hit; this one **proactively sweeps -> the whole fleet** for stale tests and fixes them in one pass. +**`.opencode/skills/cc-ci-tests-update/SKILL.md`** -## What this skill does NOT do - -- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This skill fixes - the **tests**, not the recipes. (It may *discover* a stale test by re-running a recipe upgrade - PR's `!testme`, but the recipe bump itself is owned by `/upgrade-all`.) -- **Does NOT touch the cc-ci server host / flake** — that's `/cc-ci-server-update`. -- **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is - broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review` - job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**. -- **Does NOT merge** any PR — operator merges after review. - -## Preconditions / access - -- `ssh cc-ci` works (root). Same access notes as `/recipe-upgrade` / `/ci-test-review`. -- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening cc-ci test PRs + posting `!testme`). -- The shared Swarm is **quiescent enough** for test runs — each stale-test verification deploys a - recipe, so don't run this concurrent with a `/upgrade-all` or active build loops. Serialize. -- abra over ssh needs the pseudo-TTY wrapper (see the box in `/recipe-upgrade`): every `abra` call - is `ssh cc-ci 'script -qec "abra -n" /dev/null'`. - -## Procedure - -### 1. Build the stale-test candidate list (sweep — deterministic, no AI judgement yet) - -The candidates are recipes whose cc-ci test is **potentially stale**. Two sources, merge both: - -**(a) Carry-over from `/upgrade-all` — the commented stale tests.** Each weekly run's report has a -"PRs where a test looks stale" section (recipes where `!testme` went RED on a stale test and got a -PR comment). Read the latest report + a few weeks back: -``` -grep -A2 "test.*stale\|stale.*test\|SUCCESS-PENDING-TESTS" /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | tail -40 -``` -Each line names a recipe + the stale test + the PR URL. These are **known stale** (already -diagnosed by the upgrader) — they go straight to the fix list. - -**(b) Fresh sweep — re-run the test suite across all maintained recipes.** Use the -`/ci-test-review` deterministic sweep helper to find failures that aren't already on the carry-over -list: -``` -bash /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh -``` -It runs the full harness per recipe on cc-ci and writes a JSON summary to -`/srv/cc-ci/.cc-ci-logs/ci-test-review-.json`. Parse it for `fail` verdicts. - -> **Concurrency note:** a full sweep is ~20 recipes × full suite — it takes a while and uses the -> shared Swarm. Run it when the host is otherwise idle. If you only want the carry-over stale tests -> (skip the fresh sweep), pass `RECIPES="recipe1 recipe2"` to the helper to limit it. - -**(c) Merge + classify.** Combine (a) and (b) into `CANDIDATES`. For each candidate, classify the -failure (AI — this is the `ci-test-review` step-3 diagnosis): -- **RECIPE bug** (the upgrade is broken) → NOT this skill. Hand off to `/recipe-upgrade` (5a) or - note it for the operator. Don't touch the test. -- **CI-SERVER bug** (harness/infra wrong) → NOT this skill (that's a `/ci-test-review` cc-ci-side - fix, not a test-update). Note it. -- **TEST out-of-date** (the recipe legitimately changed upstream and the cc-ci test/overlay asserts - old behavior) → **this skill**. Add to `STALE_TESTS` with: recipe, the failing test name, what - changed upstream, what the test currently asserts. -- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky). - -### 2. For each stale test — author the minimal test update (AI; never weaken) - -Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared -Swarm). For each `STALE_TESTS` entry: - -**(a) Set up a dedicated cc-ci clone on a branch** (single-writer — never push `main`, never touch -the build loops' clones): -``` -git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests- -cd /home/loops/work/cc-ci-tests- -git fetch origin && git checkout main && git reset --hard origin/main -git checkout -b test/-stale-test-$(date -u +%Y%m%d) -``` -(Or reuse `/home/loops/work/cc-ci-fix` on a new branch if it's clean — the rule is a dedicated -branch off `main`, in a clone you're not racing the loops on.) - -**(b) Read the failing test + the upstream change.** The test lives in -`tests//` (custom tests, overlays, fixtures). Read: -- the failing test (the sweep / carry-over names it), -- the recipe's current `compose.yml` + upstream changelog (use the per-recipe registry - `cc-ci-plan/upstream/.md` for the release-notes URLs — the `/upgrade-all` diagnosis - already identified what changed), -- what the test currently asserts vs what the new version does. - -**(c) Author the minimal update.** Change the test/overlay to assert the **new, correct** behavior -— not to weaken it. Examples of a legitimate stale-test fix: -- a config key was renamed upstream → update the overlay to the new key. -- a default flipped (e.g. `allow_uncategorized_topics` false→true in Discourse 3.5) → update the - assertion to the new default, OR make the test explicitly set the old value (preferred — the - test then asserts the behavior it cares about, not the default). -- an auth method was removed (e.g. Bearer-auth in impress v5.4.0) → drop that assertion, or - replace it with the new auth method's assertion (preferred if the new method is the successor). -- a readiness endpoint changed → update the readiness gate to the new endpoint. - -**What is NOT allowed:** lowering an assertion's strictness to make a red go green, deleting a test -that still applies, skipping a tier that still applies, or changing a `pass` threshold. The fix -must make the test **correct for the new version**, not lenient. If you can't fix it without -weakening, stop and report `FAILED — test cannot be updated without weakening; needs operator -decision` (leave the recipe's stale-test PR comment as-is). - -**(d) Lint the test change** (cc-ci has a lint gate): -``` -ruff check tests// && ruff format --check tests// -``` -Fix any lint errors you introduce (don't commit red lint). - -### 3. Open the cc-ci test PR (never merge) - -``` -git add tests// -git commit -m "test(): update stale test for - - asserted ; now . -Updated to assert the new behavior. No assertion weakened. - -Stale-test fix for recipe PR (carry-over from /upgrade-all )." -git push -u origin HEAD -``` -Open the PR via the helper: -``` -set -a; . /srv/cc-ci/.testenv; set +a -TITLE="test(): update stale test for " \ -BODY_FILE=/tmp/cc-ci-tests--pr-body.md \ -bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh -``` -The PR body must: name the stale test, explain what changed upstream, state that no assertion was -weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe -upgrade is also applied — they're a pair), and say "NOT merged — for operator review." - -Capture the `TEST_PR_URL`. - -### 4. Verify the updated test passes (deterministic; the standard PR + !testme flow) - -A test-update PR is only "working" once cc-ci verifies it green with the new test applied. There -are two verification paths — use whichever applies: - -**(a) If the stale test was carry-over from an open recipe upgrade PR** — verify via the paired -`!testme` flow (the `/recipe-upgrade --with-tests` step-5b path). The recipe upgrade PR's `!testme` -uses the **deployed/main** cc-ci tests, so it can't see your unmerged test branch — you need to -run the harness with your cc-ci test branch checked out on cc-ci: -``` -# 1. On cc-ci, check out your test branch in a deploy clone (NOT /root/builder-clone — use a -# separate checkout so the running server is unaffected): -ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/-stale-test- \ - ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \ - cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true' -# 2. Run the harness against the recipe upgrade PR head, with your test branch as the cc-ci checkout: -RECIPE= REF= REMOTE_ROOT=/root/cc-ci-test-verify \ - bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh -``` -Green ⇔ the recipe upgrade passes under your corrected test. This is the **paired** verification -(the test change + the recipe upgrade together go green). - -**(b) If the stale test is against a recipe already at its current upstream** (no upgrade PR — the -test is just stale against current main) — verify the test branch directly: -``` -RECIPE= REMOTE_ROOT=/root/cc-ci-test-verify \ - bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh -``` -(No `REF` — runs against the current recipe main, with your updated tests.) - -**(c) Cleanup the verify clone on cc-ci** (mandatory — shared host): -``` -ssh cc-ci 'rm -rf /root/cc-ci-test-verify' -``` - -**If verification is RED:** iterate the test fix on the same branch (**bounded — ≤3 attempts**) and -re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green -after 3 verify runs` with the failing evidence. Don't weaken the test to force green. - -> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open, -> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests — -> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the -> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is -> open and verified-paired, so the operator knows merging both → green. - -### 5. Tear down any dev deploys (defence-in-depth) - -If your verification path (or diagnosis) deployed a `dev-` stack on cc-ci, tear it down: -``` -ssh cc-ci 'D=dev-.ci.commoninternet.net; script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \ - script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2' -ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-"' # should be 0 -``` -(The `/upgrade-all` sweep would reap it next run anyway, but clean up yourself.) - -### 6. Report — one line per recipe + summary - -Write `/srv/cc-ci/.cc-ci-logs/tests-update-.md` and print, leading with the PR list: -```markdown -# cc-ci Stale-Test Update — -## Summary -- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N -## Test PRs to review (NOT merged) -| recipe | stale test | upstream change | test PR | verified | -|--------|-----------|-----------------|---------|----------| -| | | | | GREEN (paired with recipe PR ) | -## Failed (needs work) -- — : (test PR left open, red) -## Not-stale (dropped — was flaky / real recipe bug / CI-server bug) -- — : (handed off / noted) -``` -End with the report path and a reminder that **nothing was merged** — the test PRs (and their -paired recipe upgrade PRs) await operator review. - -## Guardrails - -- **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream - behavior; if that's not possible without weakening, stop and report. -- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is - operator-merged after review. -- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the - build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify` - on the host and `rm -rf` it after. -- **Stale ≠ broken.** Only update a test when the **upgrade/recipe is correct** and the **test is - wrong** (asserts old behavior). A real upgrade regression is a `/recipe-upgrade` (5a) job; a - CI-server bug is a `/ci-test-review` job. This skill fixes tests, not recipes or harness. -- **Bounded fixes** — minimal update to the failing assertion/overlay, not a test rewrite. -- **Bounded retries** — ≤3 verify runs per test PR; then leave it open + red with evidence. -- **Flake ≠ stale** — re-run before classifying as stale; don't author a fix for a flake. -- **Serialize on the shared Swarm** — one recipe's verify deploy at a time; tear down before the - next. Don't run concurrent with `/upgrade-all` or active build loops. -- **Real abra path** throughout (no docker-level bypass); pseudo-TTY wrap every abra call. +Read that file for the full procedure. This `.claude/skills/` copy is kept as a +thin pointer for Claude Code compatibility; opencode loads the canonical +definition from `.opencode/skills/` directly. diff --git a/.claude/skills/cc-ci-update/SKILL.md b/.claude/skills/cc-ci-update/SKILL.md index fedf491..67a6ced 100644 --- a/.claude/skills/cc-ci-update/SKILL.md +++ b/.claude/skills/cc-ci-update/SKILL.md @@ -3,108 +3,12 @@ name: cc-ci-update description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update. --- -# cc-ci-update +# cc-ci-update (thin wrapper) -The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump + -rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This -is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's -SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order. +The canonical definition of this skill lives in the **opencode** position: -## What this skill does NOT do +**`.opencode/skills/cc-ci-update/SKILL.md`** -- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This is cc-ci - **self-maintenance**, not recipe maintenance. -- **Does NOT merge** any PR — operator merges everything. -- **Does NOT skip the health gate** — if the server rebuild regresses, stop and roll back; do not - proceed to the tests sweep on a sick host. - -## Arguments - -None. (If you only want one of the two updates, invoke the sub-skill directly: -`/cc-ci-server-update` or `/cc-ci-tests-update`.) - -## Preconditions - -Both sub-skills' preconditions hold: -- `ssh cc-ci` works (root, tailnet `100.95.31.88`). -- Operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server (the server-update - half needs it). Pick a **no-CI window** — no `!testme` / nightly sweep running. -- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (PRs + `!testme`). -- The shared Swarm is quiescent enough for the tests sweep's verification deploys. - -## Procedure - -### 1. Run `/cc-ci-server-update` (host first) - -Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs + -sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health -gate** (no failed units, infra up, endpoints 200, disk free). See -`.claude/skills/cc-ci-server-update/SKILL.md`. - -**Gate on the result before continuing:** -- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2. -- `ROLLED-BACK` / `FAILED` → **STOP.** Do not run the tests sweep on a sick or rolled-back host. - Report the server-update failure and exit. The operator fixes the server bump first; re-run - `/cc-ci-update` (or just `/cc-ci-tests-update`) once the server is healthy on the new (or - reverted) nixpkgs. - -> **Why server-first?** The tests sweep runs the harness on cc-ci, and the harness depends on the -> host's python/pytest/playwright/docker toolchain — all from nixpkgs. Updating the host first means -> the test sweep runs against the **new** toolchain, so a green verification is meaningful for the -> post-update state (not the pre-update one). It also avoids the edge case where a nixpkgs bump -> changes a tool the tests depend on, invalidating a test sweep you just did. Server bump is the -> foundational change; tests ride on top. - -### 2. Run `/cc-ci-tests-update` (tests second, on the now-updated host) - -Run the full `/cc-ci-tests-update` procedure: build the stale-test candidate list (carry-over from -`/upgrade-all` reports + fresh `/ci-test-review` sweep) → classify each failure → for each -genuinely-stale test, author the minimal update, open a cc-ci test PR, verify via the harness -(paired with the recipe upgrade PR if carry-over) → tear down dev deploys → report. See -`.claude/skills/cc-ci-tests-update/SKILL.md`. - -**No additional gate between recipes** — the tests skill already serializes per-recipe and tears -down each verify deploy. Just run it to completion. - -### 3. Combined report - -After both halves are done, print + write a combined summary to -`/srv/cc-ci/.cc-ci-logs/cc-ci-update-.md`: -```markdown -# cc-ci Update — -## Server (host dependency bump) -- nixpkgs: → · sops-nix: → -- Rebuild: SUCCESS / ROLLED-BACK / FAILED · host healthy: yes/no -- Server PR: (NOT merged) -## Tests (stale-test sweep) -- Swept: N · Stale found: N · Test PRs opened + verified: N · Failed: N -- Test PRs: (NOT merged) -## Overall -- -``` -End with: **nothing was merged** — the server PR + test PRs (+ any paired recipe upgrade PRs) await -operator review. - -## Guardrails - -Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones: -- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update - health gate is a hard go/no-go for step 2. -- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`. -- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it - concurrent with `/upgrade-all` or active build loops. -- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged. -- **Never weaken a test** (tests half) — a stale test is updated to the new correct behavior, not - loosened. -- **Stable channel only** for the server bump (never `nixos-unstable`). - -## Relationship to automatic updates - -This is the **manual trigger** for what should ideally be two periodic timers: -- `cc-ci-server-update.timer` (host deps) — e.g. monthly, or on nixpkgs CVE advisories. -- `cc-ci-tests-update.timer` (stale tests) — e.g. weekly, after the `/upgrade-all` run so it can - close the loop on that week's commented stale tests. - -If/when those timers exist, `/cc-ci-update` is the "run both now, in the right order, with the -gate" skill — either timer can also invoke its single sub-skill directly. The order + the gate -between them (server healthy before tests) is the only thing this umbrella adds. +Read that file for the full procedure. This `.claude/skills/` copy is kept as a +thin pointer for Claude Code compatibility; opencode loads the canonical +definition from `.opencode/skills/` directly. diff --git a/.opencode/skills/cc-ci-server-update/SKILL.md b/.opencode/skills/cc-ci-server-update/SKILL.md new file mode 100644 index 0000000..16da17e --- /dev/null +++ b/.opencode/skills/cc-ci-server-update/SKILL.md @@ -0,0 +1,269 @@ +--- +name: cc-ci-server-update +description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update. +--- + +# cc-ci-server-update + +Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their +latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and +confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update` ++ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift. + +> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake +> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`). +> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a +> different machine with its own flake — it is NOT touched here. If you want to update the +> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` +> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the +> operator). This skill is only about the cc-ci **server**. + +## What this skill does NOT do + +- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. +- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or + `/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`). +- **Does NOT update the orchestrator host** — only the cc-ci server. +- **Does NOT merge** the cc-ci PR — operator merges after review. + +## Preconditions / access + +- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL + / `cc-ci-tailscaled` proxy) — do NOT proceed. +- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there + is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile + oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` / + nightly sweep running) to avoid disrupting an in-flight CI run. Check first: + ``` + ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task? + ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight? + ``` + If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI. +- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR). +- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml` + (NOT in git). The rebuild needs them — see the rebuild procedure below. + +## Procedure + +### 1. Snapshot current state (so you can compare + roll back) + +``` +ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \ + deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \ + df -h / | tail -1; nixos-version; readlink /run/current-system' +``` +Record: the current `nixos-version`, the current-system store path, the list of active infra +services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit +failed or an endpoint down gets compared against this. + +Also record the **current flake input revs** (the "before" of what you're bumping): +``` +ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head +``` + +### 2. Prepare a working clone of the cc-ci repo (single-writer) + +Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops' +working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a +fresh one: +``` +git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update +cd /home/loops/work/cc-ci-server-update +git fetch origin && git checkout main && git reset --hard origin/main +git checkout -b chore/server-flake-update-$(date -u +%Y%m%d) +``` +(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new +branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.) + +### 3. Bump the flake inputs (deliberate, not drift) + +The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in +`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that +nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a +blind `nix flake update --relock-all`. + +**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's +runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest +**stable channel revision** that's compatible: +- Default: move to the latest revision of the **same channel** the orchestrator host tracks + (`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current + tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref). +- If a newer stable channel is out and the operator wants it, use that — but treat a **channel + jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes + (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default + to staying on the same channel unless the operator asks for the jump. +- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a + stable nixpkgs. Unstable is for experiments, not production. + +**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`), +and check whether `sops-nix` needs a matching bump: +- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own + rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g. + the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After + editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or + `nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the + lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that + still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects). +- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned. +- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment + style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel"). + +**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the +host): +``` +cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch) +nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock +nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"' +git diff flake.nix flake.lock | head -40 +``` +Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the +orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here +(meaningfully). The real build happens on the cc-ci host in step 5. + +### 4. Commit + open the cc-ci PR (never merge) + +``` +git add flake.nix flake.lock +git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d)) + +nixpkgs: → (nixos-) +sops-nix: → +" +git push -u origin HEAD +``` +Open the PR via the helper: +``` +set -a; . /srv/cc-ci/.testenv; set +a +TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \ +BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \ +bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh +``` +(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any +breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.") + +Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a +flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and +proceed (the rebuild doesn't depend on the lint gate passing). + +### 5. Deploy to the live cc-ci host (no-CI window) + +This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server +has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The +deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it. + +**(a) Stage the PR branch on the host:** +``` +# from the orchestrator, on the PR branch, clean tree: +rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/ +ssh cc-ci 'chown -R root:root /root/cc-ci-deploy' +``` +(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path +flake — see (c).) + +**(b) Copy the operator-held sops secrets** (NOT in git, required for the build): +``` +ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml' +``` +(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds. +Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.) + +**(c) Build first (catch errors before switching):** +``` +ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30 +``` +- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`). +- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a + longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs). + That's expected; this is a real nixpkgs bump. +- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed + package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS + module option renames across channels (check the release notes), python package removals, a + sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev). + +**(d) Switch (the live deploy):** +``` +ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30 +``` +This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots +(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that +fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip +may be transient; a proxy failure is real). + +### 6. Verify the server is healthy (health gate — do not skip) + +``` +ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \ + echo "=== failed units ==="; systemctl --failed --no-legend; \ + echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \ + deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \ + echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \ + echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \ + echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \ + curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \ + echo "=== disk ==="; df -h / | tail -1' +``` +Compare against the step-1 baseline: +- `nixos-version` moved (or the current-system path changed) — the bump landed. +- **No failed units** (the `--failed` output is empty). +- All 7 infra services `active` (or `active (exited)` for the oneshots). +- `docker service ls` shows no `0/N` replicas (everything converged). +- Endpoints `200` (traefik `/api/version`, dashboard root). +- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually). + +If **anything regressed**: diagnose. A real regression → roll back to the previous generation: +`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the +flake input change on the PR branch is still your source of truth, but the running host reverts +while you fix). Report the rollback in the PR. + +### 7. Report + +Print + record a one-line result + the PR URL. Write a short note to +`/srv/cc-ci/.cc-ci-logs/server-update-.md`: +- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs , no failed units, endpoints 200, PR: ` +- `RESULT: ROLLED-BACK — nixpkgs rebuild failed/ regressed (); host reverted to ; PR left open` +- `RESULT: FAILED — : ` + +Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator +reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation + +the path to re-apply, not a gate on the deploy). + +### 8. Optional cold-boot proof + +A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves +it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this +if the operator wants it, and only outside a CI window: +``` +ssh cc-ci 'systemctl reboot' +# wait ~60s, re-check ssh cc-ci + the step-6 health gate +``` +Record the new `boot_id` if you do this (it's a genuine reboot — log it to +`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots +are tracked in its own journal; a note in the server-update report is enough). + +## Guardrails + +- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the + go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles + reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the + step-1 check), but don't gate on a second authorization. +- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the + build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to + `/root/cc-ci-deploy` instead). +- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record. +- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first. +- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare + to the baseline. Roll back on regression. +- **Stable channel only** for the cc-ci server (never `nixos-unstable`). +- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them. +- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only. +- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior + generation without touching the flake input revs. + +## Relationship to an automatic update + +This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci +flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added +(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke +this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The +skill stays as the authoritative procedure either way — automatic just means "run on a schedule +without a human prompt." diff --git a/.opencode/skills/cc-ci-tests-update/SKILL.md b/.opencode/skills/cc-ci-tests-update/SKILL.md new file mode 100644 index 0000000..3addcd5 --- /dev/null +++ b/.opencode/skills/cc-ci-tests-update/SKILL.md @@ -0,0 +1,244 @@ +--- +name: cc-ci-tests-update +description: Sweep all maintained recipes for stale cc-ci tests and update the stale ones so they pass again, verified via the standard PR + !testme -> CI flow. Runs the deterministic test suite across every enrolled recipe, identifies failures caused by a genuinely-stale test (the recipe legitimately changed upstream and the cc-ci test/overlay still asserts the old behavior), authors the minimal test update, opens a cc-ci test PR per stale recipe, and verifies each by running !testme on the corresponding recipe upgrade PR with the test change applied (or a direct harness run with the cc-ci branch checked out). NEVER weakens a test to force green; never merges. The fleet-wide stale-test cleanup that the weekly /upgrade-all deliberately defers (it comments-only on stale tests; this skill closes the loop). Invoke as /cc-ci-tests-update. +--- + +# cc-ci-tests-update + +The fleet-wide **stale-test cleanup** for cc-ci. The weekly `/upgrade-all` cron deliberately does +**not** auto-edit cc-ci tests — it runs in DEFAULT mode and, where a recipe's existing test is +genuinely stale against a new upstream version, it leaves an **explanatory comment** on the recipe +PR and stops (operator decides). Over time those commented stale tests accumulate. This skill +**closes that loop**: it sweeps every maintained recipe, finds the ones whose cc-ci test is stale, +authors the minimal test update, opens a cc-ci test PR per stale recipe, and **verifies** each via +the standard `!testme` → CI flow (or a direct harness run with the test branch applied). It never +weakens a test and never merges. + +> **Relationship to the other skills.** This is the fleet-wide version of the per-recipe +> `--with-tests` path in `/recipe-upgrade` (step 5b), and the "TEST out-of-date" classification +> branch of `/ci-test-review`. It reuses both skills' discipline + helpers. The difference is +> **scope**: those skills react to a failure they happened to hit; this one **proactively sweeps +> the whole fleet** for stale tests and fixes them in one pass. + +## What this skill does NOT do + +- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This skill fixes + the **tests**, not the recipes. (It may *discover* a stale test by re-running a recipe upgrade + PR's `!testme`, but the recipe bump itself is owned by `/upgrade-all`.) +- **Does NOT touch the cc-ci server host / flake** — that's `/cc-ci-server-update`. +- **Does NOT fix real upgrade regressions** — if a recipe's `!testme` is red because the upgrade is + broken (not because the test is stale), that's a `/recipe-upgrade` (step 5a) or `/ci-test-review` + job. This skill only touches a test when the **upgrade is correct** and the **test is wrong**. +- **Does NOT merge** any PR — operator merges after review. + +## Preconditions / access + +- `ssh cc-ci` works (root). Same access notes as `/recipe-upgrade` / `/ci-test-review`. +- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening cc-ci test PRs + posting `!testme`). +- The shared Swarm is **quiescent enough** for test runs — each stale-test verification deploys a + recipe, so don't run this concurrent with a `/upgrade-all` or active build loops. Serialize. +- abra over ssh needs the pseudo-TTY wrapper (see the box in `/recipe-upgrade`): every `abra` call + is `ssh cc-ci 'script -qec "abra -n" /dev/null'`. + +## Procedure + +### 1. Build the stale-test candidate list (sweep — deterministic, no AI judgement yet) + +The candidates are recipes whose cc-ci test is **potentially stale**. Two sources, merge both: + +**(a) Carry-over from `/upgrade-all` — the commented stale tests.** Each weekly run's report has a +"PRs where a test looks stale" section (recipes where `!testme` went RED on a stale test and got a +PR comment). Read the latest report + a few weeks back: +``` +grep -A2 "test.*stale\|stale.*test\|SUCCESS-PENDING-TESTS" /srv/cc-ci/.cc-ci-logs/upgrades/upgrade-all-*.md | tail -40 +``` +Each line names a recipe + the stale test + the PR URL. These are **known stale** (already +diagnosed by the upgrader) — they go straight to the fix list. + +**(b) Fresh sweep — re-run the test suite across all maintained recipes.** Use the +`/ci-test-review` deterministic sweep helper to find failures that aren't already on the carry-over +list: +``` +bash /srv/cc-ci-orch/.claude/skills/ci-test-review/run-all-recipes.sh +``` +It runs the full harness per recipe on cc-ci and writes a JSON summary to +`/srv/cc-ci/.cc-ci-logs/ci-test-review-.json`. Parse it for `fail` verdicts. + +> **Concurrency note:** a full sweep is ~20 recipes × full suite — it takes a while and uses the +> shared Swarm. Run it when the host is otherwise idle. If you only want the carry-over stale tests +> (skip the fresh sweep), pass `RECIPES="recipe1 recipe2"` to the helper to limit it. + +**(c) Merge + classify.** Combine (a) and (b) into `CANDIDATES`. For each candidate, classify the +failure (AI — this is the `ci-test-review` step-3 diagnosis): +- **RECIPE bug** (the upgrade is broken) → NOT this skill. Hand off to `/recipe-upgrade` (5a) or + note it for the operator. Don't touch the test. +- **CI-SERVER bug** (harness/infra wrong) → NOT this skill (that's a `/ci-test-review` cc-ci-side + fix, not a test-update). Note it. +- **TEST out-of-date** (the recipe legitimately changed upstream and the cc-ci test/overlay asserts + old behavior) → **this skill**. Add to `STALE_TESTS` with: recipe, the failing test name, what + changed upstream, what the test currently asserts. +- **FLAKY** → re-run once or twice; if it passes, drop it (not stale, just flaky). + +### 2. For each stale test — author the minimal test update (AI; never weaken) + +Work on **one recipe at a time** (serialize — each verification deploys a recipe on the shared +Swarm). For each `STALE_TESTS` entry: + +**(a) Set up a dedicated cc-ci clone on a branch** (single-writer — never push `main`, never touch +the build loops' clones): +``` +git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-tests- +cd /home/loops/work/cc-ci-tests- +git fetch origin && git checkout main && git reset --hard origin/main +git checkout -b test/-stale-test-$(date -u +%Y%m%d) +``` +(Or reuse `/home/loops/work/cc-ci-fix` on a new branch if it's clean — the rule is a dedicated +branch off `main`, in a clone you're not racing the loops on.) + +**(b) Read the failing test + the upstream change.** The test lives in +`tests//` (custom tests, overlays, fixtures). Read: +- the failing test (the sweep / carry-over names it), +- the recipe's current `compose.yml` + upstream changelog (use the per-recipe registry + `cc-ci-plan/upstream/.md` for the release-notes URLs — the `/upgrade-all` diagnosis + already identified what changed), +- what the test currently asserts vs what the new version does. + +**(c) Author the minimal update.** Change the test/overlay to assert the **new, correct** behavior +— not to weaken it. Examples of a legitimate stale-test fix: +- a config key was renamed upstream → update the overlay to the new key. +- a default flipped (e.g. `allow_uncategorized_topics` false→true in Discourse 3.5) → update the + assertion to the new default, OR make the test explicitly set the old value (preferred — the + test then asserts the behavior it cares about, not the default). +- an auth method was removed (e.g. Bearer-auth in impress v5.4.0) → drop that assertion, or + replace it with the new auth method's assertion (preferred if the new method is the successor). +- a readiness endpoint changed → update the readiness gate to the new endpoint. + +**What is NOT allowed:** lowering an assertion's strictness to make a red go green, deleting a test +that still applies, skipping a tier that still applies, or changing a `pass` threshold. The fix +must make the test **correct for the new version**, not lenient. If you can't fix it without +weakening, stop and report `FAILED — test cannot be updated without weakening; needs operator +decision` (leave the recipe's stale-test PR comment as-is). + +**(d) Lint the test change** (cc-ci has a lint gate): +``` +ruff check tests// && ruff format --check tests// +``` +Fix any lint errors you introduce (don't commit red lint). + +### 3. Open the cc-ci test PR (never merge) + +``` +git add tests// +git commit -m "test(): update stale test for + + asserted ; now . +Updated to assert the new behavior. No assertion weakened. + +Stale-test fix for recipe PR (carry-over from /upgrade-all )." +git push -u origin HEAD +``` +Open the PR via the helper: +``` +set -a; . /srv/cc-ci/.testenv; set +a +TITLE="test(): update stale test for " \ +BODY_FILE=/tmp/cc-ci-tests--pr-body.md \ +bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh +``` +The PR body must: name the stale test, explain what changed upstream, state that no assertion was +weakened, link the **dependent recipe upgrade PR** (the test change only goes green once the recipe +upgrade is also applied — they're a pair), and say "NOT merged — for operator review." + +Capture the `TEST_PR_URL`. + +### 4. Verify the updated test passes (deterministic; the standard PR + !testme flow) + +A test-update PR is only "working" once cc-ci verifies it green with the new test applied. There +are two verification paths — use whichever applies: + +**(a) If the stale test was carry-over from an open recipe upgrade PR** — verify via the paired +`!testme` flow (the `/recipe-upgrade --with-tests` step-5b path). The recipe upgrade PR's `!testme` +uses the **deployed/main** cc-ci tests, so it can't see your unmerged test branch — you need to +run the harness with your cc-ci test branch checked out on cc-ci: +``` +# 1. On cc-ci, check out your test branch in a deploy clone (NOT /root/builder-clone — use a +# separate checkout so the running server is unaffected): +ssh cc-ci 'rm -rf /root/cc-ci-test-verify && git clone --branch test/-stale-test- \ + ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /root/cc-ci-test-verify && \ + cd /root/cc-ci-test-verify && git submodule update --init secrets 2>/dev/null || true' +# 2. Run the harness against the recipe upgrade PR head, with your test branch as the cc-ci checkout: +RECIPE= REF= REMOTE_ROOT=/root/cc-ci-test-verify \ + bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh +``` +Green ⇔ the recipe upgrade passes under your corrected test. This is the **paired** verification +(the test change + the recipe upgrade together go green). + +**(b) If the stale test is against a recipe already at its current upstream** (no upgrade PR — the +test is just stale against current main) — verify the test branch directly: +``` +RECIPE= REMOTE_ROOT=/root/cc-ci-test-verify \ + bash /srv/cc-ci-orch/.claude/skills/ci-test-review/verify-pr.sh +``` +(No `REF` — runs against the current recipe main, with your updated tests.) + +**(c) Cleanup the verify clone on cc-ci** (mandatory — shared host): +``` +ssh cc-ci 'rm -rf /root/cc-ci-test-verify' +``` + +**If verification is RED:** iterate the test fix on the same branch (**bounded — ≤3 attempts**) and +re-verify. If still red after 3, leave the test PR open and report `FAILED — test update not green +after 3 verify runs` with the failing evidence. Don't weaken the test to force green. + +> **Re-verify with a real `!testme` on the recipe PR too?** If the recipe upgrade PR is still open, +> posting a fresh `!testme` on it after the test PR merges would re-run with the merged tests — +> but you can't merge the test PR (operator does). So the harness-branch verification in (a) is the +> authoritative pre-merge proof. Leave a comment on the recipe upgrade PR noting the test PR is +> open and verified-paired, so the operator knows merging both → green. + +### 5. Tear down any dev deploys (defence-in-depth) + +If your verification path (or diagnosis) deployed a `dev-` stack on cc-ci, tear it down: +``` +ssh cc-ci 'D=dev-.ci.commoninternet.net; script -qec "abra app undeploy $D -n" /dev/null 2>&1 | tail -2; \ + script -qec "abra app rm $D -n --no-input" /dev/null 2>&1 | tail -2' +ssh cc-ci 'docker stack ls --format "{{.Name}}" | grep -c "^dev-"' # should be 0 +``` +(The `/upgrade-all` sweep would reap it next run anyway, but clean up yourself.) + +### 6. Report — one line per recipe + summary + +Write `/srv/cc-ci/.cc-ci-logs/tests-update-.md` and print, leading with the PR list: +```markdown +# cc-ci Stale-Test Update — +## Summary +- Swept: N recipes · Stale tests found: N · Test PRs opened + verified: N · Failed: N · Not-stale (dropped): N +## Test PRs to review (NOT merged) +| recipe | stale test | upstream change | test PR | verified | +|--------|-----------|-----------------|---------|----------| +| | | | | GREEN (paired with recipe PR ) | +## Failed (needs work) +- — : (test PR left open, red) +## Not-stale (dropped — was flaky / real recipe bug / CI-server bug) +- — : (handed off / noted) +``` +End with the report path and a reminder that **nothing was merged** — the test PRs (and their +paired recipe upgrade PRs) await operator review. + +## Guardrails + +- **Never weaken a test** to turn a red green. The fix makes the test correct for the new upstream + behavior; if that's not possible without weakening, stop and report. +- **Create + verify, NEVER merge.** Each test PR (and any paired recipe upgrade PR) is + operator-merged after review. +- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the + build loops' `/cc-ci` `/cc-ci-adv` clones or `/root/builder-clone`. Use `/root/cc-ci-test-verify` + on the host and `rm -rf` it after. +- **Stale ≠ broken.** Only update a test when the **upgrade/recipe is correct** and the **test is + wrong** (asserts old behavior). A real upgrade regression is a `/recipe-upgrade` (5a) job; a + CI-server bug is a `/ci-test-review` job. This skill fixes tests, not recipes or harness. +- **Bounded fixes** — minimal update to the failing assertion/overlay, not a test rewrite. +- **Bounded retries** — ≤3 verify runs per test PR; then leave it open + red with evidence. +- **Flake ≠ stale** — re-run before classifying as stale; don't author a fix for a flake. +- **Serialize on the shared Swarm** — one recipe's verify deploy at a time; tear down before the + next. Don't run concurrent with `/upgrade-all` or active build loops. +- **Real abra path** throughout (no docker-level bypass); pseudo-TTY wrap every abra call. diff --git a/.opencode/skills/cc-ci-update/SKILL.md b/.opencode/skills/cc-ci-update/SKILL.md new file mode 100644 index 0000000..fedf491 --- /dev/null +++ b/.opencode/skills/cc-ci-update/SKILL.md @@ -0,0 +1,110 @@ +--- +name: cc-ci-update +description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with verified, ready-to-merge PRs and a healthy server at the end. NEVER merges anything. Invoke as /cc-ci-update. +--- + +# cc-ci-update + +The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump + +rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This +is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's +SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order. + +## What this skill does NOT do + +- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`. This is cc-ci + **self-maintenance**, not recipe maintenance. +- **Does NOT merge** any PR — operator merges everything. +- **Does NOT skip the health gate** — if the server rebuild regresses, stop and roll back; do not + proceed to the tests sweep on a sick host. + +## Arguments + +None. (If you only want one of the two updates, invoke the sub-skill directly: +`/cc-ci-server-update` or `/cc-ci-tests-update`.) + +## Preconditions + +Both sub-skills' preconditions hold: +- `ssh cc-ci` works (root, tailnet `100.95.31.88`). +- Operator's go-ahead for a **live `nixos-rebuild switch`** on the cc-ci server (the server-update + half needs it). Pick a **no-CI window** — no `!testme` / nightly sweep running. +- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (PRs + `!testme`). +- The shared Swarm is quiescent enough for the tests sweep's verification deploys. + +## Procedure + +### 1. Run `/cc-ci-server-update` (host first) + +Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs + +sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health +gate** (no failed units, infra up, endpoints 200, disk free). See +`.claude/skills/cc-ci-server-update/SKILL.md`. + +**Gate on the result before continuing:** +- `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2. +- `ROLLED-BACK` / `FAILED` → **STOP.** Do not run the tests sweep on a sick or rolled-back host. + Report the server-update failure and exit. The operator fixes the server bump first; re-run + `/cc-ci-update` (or just `/cc-ci-tests-update`) once the server is healthy on the new (or + reverted) nixpkgs. + +> **Why server-first?** The tests sweep runs the harness on cc-ci, and the harness depends on the +> host's python/pytest/playwright/docker toolchain — all from nixpkgs. Updating the host first means +> the test sweep runs against the **new** toolchain, so a green verification is meaningful for the +> post-update state (not the pre-update one). It also avoids the edge case where a nixpkgs bump +> changes a tool the tests depend on, invalidating a test sweep you just did. Server bump is the +> foundational change; tests ride on top. + +### 2. Run `/cc-ci-tests-update` (tests second, on the now-updated host) + +Run the full `/cc-ci-tests-update` procedure: build the stale-test candidate list (carry-over from +`/upgrade-all` reports + fresh `/ci-test-review` sweep) → classify each failure → for each +genuinely-stale test, author the minimal update, open a cc-ci test PR, verify via the harness +(paired with the recipe upgrade PR if carry-over) → tear down dev deploys → report. See +`.claude/skills/cc-ci-tests-update/SKILL.md`. + +**No additional gate between recipes** — the tests skill already serializes per-recipe and tears +down each verify deploy. Just run it to completion. + +### 3. Combined report + +After both halves are done, print + write a combined summary to +`/srv/cc-ci/.cc-ci-logs/cc-ci-update-.md`: +```markdown +# cc-ci Update — +## Server (host dependency bump) +- nixpkgs: → · sops-nix: → +- Rebuild: SUCCESS / ROLLED-BACK / FAILED · host healthy: yes/no +- Server PR: (NOT merged) +## Tests (stale-test sweep) +- Swept: N · Stale found: N · Test PRs opened + verified: N · Failed: N +- Test PRs: (NOT merged) +## Overall +- +``` +End with: **nothing was merged** — the server PR + test PRs (+ any paired recipe upgrade PRs) await +operator review. + +## Guardrails + +Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones: +- **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update + health gate is a hard go/no-go for step 2. +- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`. +- **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it + concurrent with `/upgrade-all` or active build loops. +- **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged. +- **Never weaken a test** (tests half) — a stale test is updated to the new correct behavior, not + loosened. +- **Stable channel only** for the server bump (never `nixos-unstable`). + +## Relationship to automatic updates + +This is the **manual trigger** for what should ideally be two periodic timers: +- `cc-ci-server-update.timer` (host deps) — e.g. monthly, or on nixpkgs CVE advisories. +- `cc-ci-tests-update.timer` (stale tests) — e.g. weekly, after the `/upgrade-all` run so it can + close the loop on that week's commented stale tests. + +If/when those timers exist, `/cc-ci-update` is the "run both now, in the right order, with the +gate" skill — either timer can also invoke its single sub-skill directly. The order + the gate +between them (server healthy before tests) is the only thing this umbrella adds.