skills: move cc-ci-update skills to .opencode/ as canonical; .claude/ -> thin wrappers
Reverse the skill location: the full definitions now live in .opencode/skills/<name>/SKILL.md (opencode's native project-skill location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin pointer wrappers that keep the frontmatter for discoverability and redirect to the .opencode/ canonical file for the procedure. Also drops the 'operator-authorized mid-run' gating from /cc-ci-server-update: the operator's choice to invoke the skill IS the authorization for the live nixos-rebuild switch — no second check to wait for. The quiescent-moment preference stays (don't disrupt an in-flight !testme), but it's not a gate.
This commit is contained in:
autonomic-bot
@@ -0,0 +1,269 @@
|
||||
---
|
||||
name: cc-ci-server-update
|
||||
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
|
||||
---
|
||||
|
||||
# cc-ci-server-update
|
||||
|
||||
Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
|
||||
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
|
||||
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
|
||||
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
|
||||
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
|
||||
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
|
||||
> different machine with its own flake — it is NOT touched here. If you want to update the
|
||||
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
|
||||
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
|
||||
> operator). This skill is only about the cc-ci **server**.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
|
||||
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
|
||||
`/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
|
||||
- **Does NOT update the orchestrator host** — only the cc-ci server.
|
||||
- **Does NOT merge** the cc-ci PR — operator merges after review.
|
||||
|
||||
## Preconditions / access
|
||||
|
||||
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
|
||||
/ `cc-ci-tailscaled` proxy) — do NOT proceed.
|
||||
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
|
||||
is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
|
||||
oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
|
||||
nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
|
||||
```
|
||||
ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"' # any unhealthy task?
|
||||
ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep' # a CI run in flight?
|
||||
```
|
||||
If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
|
||||
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
|
||||
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
|
||||
(NOT in git). The rebuild needs them — see the rebuild procedure below.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot current state (so you can compare + roll back)
|
||||
|
||||
```
|
||||
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
|
||||
df -h / | tail -1; nixos-version; readlink /run/current-system'
|
||||
```
|
||||
Record: the current `nixos-version`, the current-system store path, the list of active infra
|
||||
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
|
||||
failed or an endpoint down gets compared against this.
|
||||
|
||||
Also record the **current flake input revs** (the "before" of what you're bumping):
|
||||
```
|
||||
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
|
||||
```
|
||||
|
||||
### 2. Prepare a working clone of the cc-ci repo (single-writer)
|
||||
|
||||
Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
|
||||
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
|
||||
fresh one:
|
||||
```
|
||||
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
|
||||
cd /home/loops/work/cc-ci-server-update
|
||||
git fetch origin && git checkout main && git reset --hard origin/main
|
||||
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
|
||||
```
|
||||
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
|
||||
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
|
||||
|
||||
### 3. Bump the flake inputs (deliberate, not drift)
|
||||
|
||||
The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
|
||||
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
|
||||
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
|
||||
blind `nix flake update --relock-all`.
|
||||
|
||||
**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
|
||||
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
|
||||
**stable channel revision** that's compatible:
|
||||
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
|
||||
(`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
|
||||
tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
|
||||
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
|
||||
jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
|
||||
(docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
|
||||
to staying on the same channel unless the operator asks for the jump.
|
||||
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
|
||||
stable nixpkgs. Unstable is for experiments, not production.
|
||||
|
||||
**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
|
||||
and check whether `sops-nix` needs a matching bump:
|
||||
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
|
||||
rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
|
||||
the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
|
||||
editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
|
||||
`nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
|
||||
lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
|
||||
still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
|
||||
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
|
||||
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
|
||||
style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
|
||||
|
||||
**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
|
||||
host):
|
||||
```
|
||||
cd /home/loops/work/cc-ci-server-update # (or cc-ci-fix on the new branch)
|
||||
nix flake lock --update-input nixpkgs --update-input sops-nix # rewrites flake.lock
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
git diff flake.nix flake.lock | head -40
|
||||
```
|
||||
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
|
||||
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
|
||||
(meaningfully). The real build happens on the cc-ci host in step 5.
|
||||
|
||||
### 4. Commit + open the cc-ci PR (never merge)
|
||||
|
||||
```
|
||||
git add flake.nix flake.lock
|
||||
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
|
||||
sops-nix: <old-rev[:8]> → <new-rev[:8]>
|
||||
<one-line reason — CVE patch / channel bump / sops-nix compat>"
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR via the helper:
|
||||
```
|
||||
set -a; . /srv/cc-ci/.testenv; set +a
|
||||
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
|
||||
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
|
||||
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
|
||||
```
|
||||
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
|
||||
breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review.")
|
||||
|
||||
Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
|
||||
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
|
||||
proceed (the rebuild doesn't depend on the lint gate passing).
|
||||
|
||||
### 5. Deploy to the live cc-ci host (no-CI window)
|
||||
|
||||
This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
|
||||
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
|
||||
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
|
||||
|
||||
**(a) Stage the PR branch on the host:**
|
||||
```
|
||||
# from the orchestrator, on the PR branch, clean tree:
|
||||
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
|
||||
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
|
||||
```
|
||||
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
|
||||
flake — see (c).)
|
||||
|
||||
**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
|
||||
```
|
||||
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
|
||||
```
|
||||
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
|
||||
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
|
||||
|
||||
**(c) Build first (catch errors before switching):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
|
||||
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
|
||||
longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
|
||||
That's expected; this is a real nixpkgs bump.
|
||||
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
|
||||
package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
|
||||
module option renames across channels (check the release notes), python package removals, a
|
||||
sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
|
||||
|
||||
**(d) Switch (the live deploy):**
|
||||
```
|
||||
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
|
||||
```
|
||||
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
|
||||
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
|
||||
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
|
||||
may be transient; a proxy failure is real).
|
||||
|
||||
### 6. Verify the server is healthy (health gate — do not skip)
|
||||
|
||||
```
|
||||
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
|
||||
echo "=== failed units ==="; systemctl --failed --no-legend; \
|
||||
echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
|
||||
deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
|
||||
echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
|
||||
echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
|
||||
echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
|
||||
curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
|
||||
echo "=== disk ==="; df -h / | tail -1'
|
||||
```
|
||||
Compare against the step-1 baseline:
|
||||
- `nixos-version` moved (or the current-system path changed) — the bump landed.
|
||||
- **No failed units** (the `--failed` output is empty).
|
||||
- All 7 infra services `active` (or `active (exited)` for the oneshots).
|
||||
- `docker service ls` shows no `0/N` replicas (everything converged).
|
||||
- Endpoints `200` (traefik `/api/version`, dashboard root).
|
||||
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
|
||||
|
||||
If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
|
||||
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
|
||||
flake input change on the PR branch is still your source of truth, but the running host reverts
|
||||
while you fix). Report the rollback in the PR.
|
||||
|
||||
### 7. Report
|
||||
|
||||
Print + record a one-line result + the PR URL. Write a short note to
|
||||
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
|
||||
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>`
|
||||
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
|
||||
- `RESULT: FAILED — <step>: <reason>`
|
||||
|
||||
Always state that **the cc-ci PR is NOT merged** — it's the record of the bump; the operator
|
||||
reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation +
|
||||
the path to re-apply, not a gate on the deploy).
|
||||
|
||||
### 8. Optional cold-boot proof
|
||||
|
||||
A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
|
||||
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
|
||||
if the operator wants it, and only outside a CI window:
|
||||
```
|
||||
ssh cc-ci 'systemctl reboot'
|
||||
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
|
||||
```
|
||||
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
|
||||
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
|
||||
are tracked in its own journal; a note in the server-update report is enough).
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
|
||||
go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
|
||||
reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
|
||||
step-1 check), but don't gate on a second authorization.
|
||||
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
|
||||
build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
|
||||
`/root/cc-ci-deploy` instead).
|
||||
- **Never merge the cc-ci PR.** The live rebuild is the deploy; the PR is the reviewable record.
|
||||
- **Build before switch.** Never `nixos-rebuild switch` a build that didn't build cleanly first.
|
||||
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
|
||||
to the baseline. Roll back on regression.
|
||||
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
|
||||
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
|
||||
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
|
||||
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
|
||||
generation without touching the flake input revs.
|
||||
|
||||
## Relationship to an automatic update
|
||||
|
||||
This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
|
||||
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
|
||||
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
|
||||
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
|
||||
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
|
||||
without a human prompt."
|
||||
Reference in New Issue
Block a user