cc-ci-orchestrator/.claude/skills/cc-ci-server-update/SKILL.md

name, description

name

description

cc-ci-server-update

Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (operator-authorized; picked a no-CI window), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main; never merges the cc-ci PR. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.

cc-ci-server-update

Manually update the cc-ci CI server host's NixOS dependencies (nixpkgs + sops-nix) to their latest pinned revisions, ship the bump via a verified cc-ci PR + a live nixos-rebuild switch, and confirm the server is healthy afterward. This is the cc-ci-server analogue of a nix flake update

• rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.

Two hosts, two flakes — don't confuse them. This skill updates the cc-ci server flake (ssh cc-ci, repo recipe-maintainers/cc-ci, host target .#cc-ci = nix/hosts/cc-ci-hetzner/). The orchestrator host (/srv/cc-ci-orch, flake host .#cc-ci-orchestrator-hetzner) is a different machine with its own flake — it is NOT touched here. If you want to update the orchestrator host, that's a separate nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner from /srv/cc-ci-orch (the orchestrator flake already tracks nixos-26.05 and is rebuilt by the operator). This skill is only about the cc-ci server.

What this skill does NOT do

• Does NOT bump recipe image tags — that's /upgrade-all / /recipe-upgrade.
• Does NOT touch cc-ci tests / harness / runner — that's /cc-ci-tests-update or /ci-dev-workflow. This is a flake-input bump only (flake.nix input revs + flake.lock).
• Does NOT update the orchestrator host — only the cc-ci server.
• Does NOT merge the cc-ci PR — operator merges after review.

Preconditions / access

• ssh cc-ci works (root, tailnet 100.95.31.88). If it doesn't, fix access first (Tailscale ACL / cc-ci-tailscaled proxy) — do NOT proceed.
• The operator's go-ahead for a live nixos-rebuild switch on the cc-ci server. A switch cycles the abra reconcile oneshots (deploy-proxy, warm-keycloak, …) — pick a no-CI window (no !testme / nightly sweep running). Check first:

  ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"'   # any unhealthy task?
  ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep'                                 # a CI run in flight?
  

  If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
• /srv/cc-ci/.testenv has the GITEA_* creds (for opening the cc-ci PR).
• The operator-held sops secrets are on the cc-ci host at /etc/cc-ci/secrets/secrets.yaml (NOT in git). The rebuild needs them — see the rebuild procedure below.

Procedure

1. Snapshot current state (so you can compare + roll back)

ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
  deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
  df -h / | tail -1; nixos-version; readlink /run/current-system'

Record: the current nixos-version, the current-system store path, the list of active infra services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit failed or an endpoint down gets compared against this.

Also record the current flake input revs (the "before" of what you're bumping):

ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head

2. Prepare a working clone of the cc-ci repo (single-writer)

Work in a separate clone on the orchestrator — never push main, never touch the build loops' working clones. Use the existing /home/loops/work/cc-ci-fix clone if present and clean, or make a fresh one:

git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
cd /home/loops/work/cc-ci-server-update
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)

(If /home/loops/work/cc-ci-fix already exists and is on a stale branch, either reuse it on a new branch or make the fresh clone above — the key rule is the branch is dedicated + off main.)

3. Bump the flake inputs (deliberate, not drift)

The cc-ci server flake intentionally hard-pins nixpkgs to a specific revision (the comment in flake.nix says "Bump deliberately, not drift") and sops-nix to a commit compatible with that nixpkgs. So the bump is a considered move to a newer nixpkgs + a compatible sops-nix, not a blind nix flake update --relock-all.

(a) Decide the new nixpkgs. The cc-ci server runs recipe CI; nixpkgs provides docker, abra's runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest stable channel revision that's compatible:

• Default: move to the latest revision of the same channel the orchestrator host tracks (nixos-26.05), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current tip: git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 (the SHA after the ref).
• If a newer stable channel is out and the operator wants it, use that — but treat a channel jump (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default to staying on the same channel unless the operator asks for the jump.
• Never jump to nixos-unstable for the cc-ci server — recipe CI reproducibility depends on a stable nixpkgs. Unstable is for experiments, not production.

(b) Update flake.nix with the new nixpkgs rev (replace the rev = … in the nixpkgs.url), and check whether sops-nix needs a matching bump:

• sops-nix inputs.nixpkgs.follows = "nixpkgs", so it tracks nixpkgs automatically — but its own rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g. the buildGoModule → buildGo125Module gotcha the current pin's comment calls out). After editing flake.nix, run nix flake lock --update-input nixpkgs --update-input sops-nix (or nix flake update if both follow) in the clone and let nix resolve a compatible sops-nix. If the lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that still builds against your chosen nixpkgs (its flake.nix declares the nixpkgs it expects).
• Keep the sops-nix.inputs.nixpkgs.follows = "nixpkgs" line — that's what keeps them aligned.
• Update the comment in flake.nix to reflect the new pin date + reason (the existing comment style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").

(c) Re-lock + sanity-build locally first (cheap, catches a broken input before touching the host):

cd /home/loops/work/cc-ci-server-update   # (or cc-ci-fix on the new branch)
nix flake lock --update-input nixpkgs --update-input sops-nix   # rewrites flake.lock
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
git diff flake.nix flake.lock | head -40

Verify the lock moved to the revs you intended. Do NOT nixos-rebuild build from the orchestrator — the orchestrator's nixpkgs is different and the cc-ci host target won't build here (meaningfully). The real build happens on the cc-ci host in step 5.

4. Commit + open the cc-ci PR (never merge)

git add flake.nix flake.lock
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))

nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
sops-nix: <old-rev[:8]> → <new-rev[:8]>
<one-line reason — CVE patch / channel bump / sops-nix compat>"
git push -u origin HEAD

Open the PR via the helper:

set -a; . /srv/cc-ci/.testenv; set +a
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh

(Write the PR body to /tmp/cc-ci-server-update-pr-body.md first: old→new revs, the channel, any breaking changes from the release notes, the rebuild plan, and "NOT merged — for operator review. Live rebuild to follow on operator go-ahead.")

Capture the PR_URL. The cc-ci push-build lint gate will run on the branch — if it's red on a flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and proceed (the rebuild doesn't depend on the lint gate passing).

5. Deploy to the live cc-ci host (operator-authorized, no-CI window)

This is the host-rebuild procedure from memory cc-ci-host-rebuild-procedure — the cc-ci server has no self-service rebuild path, so the orchestrator stages the repo + rebuilds it. The deploy clone on the host is /root/builder-clone; staging a fresh copy avoids disturbing it.

(a) Stage the PR branch on the host:

# from the orchestrator, on the PR branch, clean tree:
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'

(A --delete rsync from a clean tree = exactly the PR head. Excluding .git keeps it a plain path flake — see (c).)

(b) Copy the operator-held sops secrets (NOT in git, required for the build):

ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'

(The age key is at /var/lib/sops-nix/key.txt on the host — already in place from prior rebuilds. Without secrets/secrets.yaml the build FATAs secrets/secrets.yaml does not exist.)

(c) Build first (catch errors before switching):

ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30

• Target is .#cc-ci (= .#cc-ci-hetzner = nix/hosts/cc-ci-hetzner/configuration.nix).
• nixpkgs is now your bumped pin, so this will rebuild more than a same-pin no-op — expect a longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs). That's expected; this is a real nixpkgs bump.
• If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed package), re-push, re-stage, re-build. Do NOT switch a failed build. Common breakages: NixOS module option renames across channels (check the release notes), python package removals, a sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).

(d) Switch (the live deploy):

ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30

This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots (deploy-proxy, warm-keycloak, …) re-run as part of the switch — watch the tail for any that fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip may be transient; a proxy failure is real).

6. Verify the server is healthy (health gate — do not skip)

ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
  echo "=== failed units ==="; systemctl --failed --no-legend; \
  echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
    deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
  echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
  echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
  echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
  curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
  echo "=== disk ==="; df -h / | tail -1'

Compare against the step-1 baseline:

• nixos-version moved (or the current-system path changed) — the bump landed.
• No failed units (the --failed output is empty).
• All 7 infra services active (or active (exited) for the oneshots).
• docker service ls shows no 0/N replicas (everything converged).
• Endpoints 200 (traefik /api/version, dashboard root).
• Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).

If anything regressed: diagnose. A real regression → roll back to the previous generation: ssh cc-ci 'nixos-rebuild switch --rollback' (this switches to the pre-bump system path; the flake input change on the PR branch is still your source of truth, but the running host reverts while you fix). Report the rollback in the PR.

7. Report

Print + record a one-line result + the PR URL. Write a short note to /srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md:

• RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR: <url>
• RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open
• RESULT: FAILED — <step>: <reason>

Always state that the cc-ci PR is NOT merged — it's the record of the bump; the operator reviews/merges at their discretion (the live rebuild already happened, so the PR is documentation + the path to re-apply, not a gate on the deploy).

8. Optional cold-boot proof

A nixos-rebuild switch proves the new generation works on the running kernel. A reboot proves it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this if the operator wants it, and only outside a CI window:

ssh cc-ci 'systemctl reboot'
# wait ~60s, re-check ssh cc-ci + the step-6 health gate

Record the new boot_id if you do this (it's a genuine reboot — log it to cc-ci-plan/REBOOTS.md? No — that file is for the orchestrator host. The cc-ci server's reboots are tracked in its own journal; a note in the server-update report is enough).

Guardrails

• Operator-authorized live rebuild, no-CI window. A nixos-rebuild switch cycles reconcile oneshots and can disrupt an in-flight !testme. Check the host is quiescent first (step 1) and get the operator's go-ahead.
• Single-writer: dedicated branch in a separate clone; never push main; never touch the build loops' /cc-ci / /cc-ci-adv clones or /root/builder-clone (stage to /root/cc-ci-deploy instead).
• Never merge the cc-ci PR. The live rebuild is the deploy; the PR is the reviewable record.
• Build before switch. Never nixos-rebuild switch a build that didn't build cleanly first.
• Health gate is mandatory. No "it switched, ship it" — run the step-6 verification and compare to the baseline. Roll back on regression.
• Stable channel only for the cc-ci server (never nixos-unstable).
• Keep sops-nix.follows so the two inputs stay aligned; don't fork them.
• Secrets never in git. The operator-held secrets/secrets.yaml is copied on the host only.
• Rollback is always available: nixos-rebuild switch --rollback reverts to the prior generation without touching the flake input revs.

Relationship to an automatic update

This skill is the manual trigger for what should ideally run on a periodic timer: bump the cc-ci flake inputs, build, switch, verify. If/when an automatic cc-ci-server-update.timer is added (NixOS systemd timer on the orchestrator, analog of cc-ci-upgrade-all.timer), it should invoke this skill via a dedicated one-shot agent (the way cc-ci-upgrader runs /upgrade-all). The skill stays as the authoritative procedure either way — automatic just means "run on a schedule without a human prompt."