cc-ci-orchestrator/.opencode/skills/cc-ci-server-update/SKILL.md

name, description

name

description

cc-ci-server-update

Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.

cc-ci-server-update

Manually update the cc-ci CI server host's NixOS dependencies (nixpkgs + sops-nix) to their latest pinned revisions, ship the bump via a verified cc-ci PR + a live nixos-rebuild switch, and confirm the server is healthy afterward. This is the cc-ci-server analogue of a nix flake update

• rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.

Two hosts, two flakes — don't confuse them. This skill updates the cc-ci server flake (ssh cc-ci, repo recipe-maintainers/cc-ci, host target .#cc-ci = nix/hosts/cc-ci-hetzner/). The orchestrator host (/srv/cc-ci-orch, flake host .#cc-ci-orchestrator-hetzner) is a different machine with its own flake — it is NOT touched here. If you want to update the orchestrator host, that's a separate nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner from /srv/cc-ci-orch (the orchestrator flake already tracks nixos-26.05 and is rebuilt by the operator). This skill is only about the cc-ci server.

What this skill does NOT do

• Does NOT bump recipe image tags — that's /upgrade-all / /recipe-upgrade.
• Does NOT touch cc-ci tests / harness / runner — that's /cc-ci-tests-update or /ci-dev-workflow. This is a flake-input bump only (flake.nix input revs + flake.lock).
• Does NOT update the orchestrator host — only the cc-ci server.
• Does merge the cc-ci PR itself — the PR exists for visibility and historical record, and is merged directly at the end of a verified run (the skill invocation is the authorization; the operator reviews the merged PR afterwards via the link in the report).

Preconditions / access

• ssh cc-ci works (root, tailnet 100.95.31.88). If it doesn't, fix access first (Tailscale ACL / cc-ci-tailscaled proxy) — do NOT proceed.
• The operator's choice to invoke this skill IS the authorization for the live rebuild — there is no mid-run operator check to wait for. A nixos-rebuild switch cycles the abra reconcile oneshots (deploy-proxy, warm-keycloak, …), so prefer a quiescent moment (no !testme / nightly sweep running) to avoid disrupting an in-flight CI run. Check first:

  ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"'   # any unhealthy task?
  ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep'                                 # a CI run in flight?
  

  If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
• /srv/cc-ci/.testenv has the GITEA_* creds (for opening the cc-ci PR).
• The operator-held sops secrets are on the cc-ci host at /etc/cc-ci/secrets/secrets.yaml (NOT in git). The rebuild needs them — see the rebuild procedure below.

Procedure

1. Snapshot current state (so you can compare + roll back)

ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
  deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
  df -h / | tail -1; nixos-version; readlink /run/current-system'

Record: the current nixos-version, the current-system store path, the list of active infra services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit failed or an endpoint down gets compared against this.

Also record the current flake input revs (the "before" of what you're bumping):

ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head

2. Prepare a working clone of the cc-ci repo (single-writer)

Work in a separate clone on the orchestrator — never push main, never touch the build loops' working clones. Use the existing /home/loops/work/cc-ci-fix clone if present and clean, or make a fresh one:

git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
cd /home/loops/work/cc-ci-server-update
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)

(If /home/loops/work/cc-ci-fix already exists and is on a stale branch, either reuse it on a new branch or make the fresh clone above — the key rule is the branch is dedicated + off main.)

3. Bump the flake inputs (deliberate, not drift)

The cc-ci server flake intentionally hard-pins nixpkgs to a specific revision (the comment in flake.nix says "Bump deliberately, not drift") and sops-nix to a commit compatible with that nixpkgs. So the bump is a considered move to a newer nixpkgs + a compatible sops-nix, not a blind nix flake update --relock-all.

(a) Decide the new nixpkgs. The cc-ci server runs recipe CI; nixpkgs provides docker, abra's runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest stable channel revision that's compatible:

• Default: move to the latest revision of the same channel the orchestrator host tracks (nixos-26.05), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current tip: git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 (the SHA after the ref).
• If a newer stable channel is out and the operator wants it, use that — but treat a channel jump (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default to staying on the same channel unless the operator asks for the jump.
• Never jump to nixos-unstable for the cc-ci server — recipe CI reproducibility depends on a stable nixpkgs. Unstable is for experiments, not production.

(b) Update flake.nix with the new nixpkgs rev (replace the rev = … in the nixpkgs.url), and check whether sops-nix needs a matching bump:

• sops-nix inputs.nixpkgs.follows = "nixpkgs", so it tracks nixpkgs automatically — but its own rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g. the buildGoModule → buildGo125Module gotcha the current pin's comment calls out). After editing flake.nix, run nix flake lock --update-input nixpkgs --update-input sops-nix (or nix flake update if both follow) in the clone and let nix resolve a compatible sops-nix. If the lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that still builds against your chosen nixpkgs (its flake.nix declares the nixpkgs it expects).
• Keep the sops-nix.inputs.nixpkgs.follows = "nixpkgs" line — that's what keeps them aligned.
• Update the comment in flake.nix to reflect the new pin date + reason (the existing comment style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").

(c) Re-lock + sanity-build locally first (cheap, catches a broken input before touching the host):

cd /home/loops/work/cc-ci-server-update   # (or cc-ci-fix on the new branch)
nix flake lock --update-input nixpkgs --update-input sops-nix   # rewrites flake.lock
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
git diff flake.nix flake.lock | head -40

Verify the lock moved to the revs you intended. Do NOT nixos-rebuild build from the orchestrator — the orchestrator's nixpkgs is different and the cc-ci host target won't build here (meaningfully). The real build happens on the cc-ci host in step 5.

4. Commit + open the cc-ci PR (merged in step 7 after the health gate)

git add flake.nix flake.lock
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))

nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
sops-nix: <old-rev[:8]> → <new-rev[:8]>
<one-line reason — CVE patch / channel bump / sops-nix compat>"
git push -u origin HEAD

Open the PR via the helper:

set -a; . /srv/cc-ci/.testenv; set +a
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh

(Write the PR body to /tmp/cc-ci-server-update-pr-body.md first: old→new revs, the channel, any breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and historical record; it gets merged directly in step 7 once the deploy passes the health gate.)

Capture the PR_URL. The cc-ci push-build lint gate will run on the branch — if it's red on a flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and proceed (the rebuild doesn't depend on the lint gate passing).

5. Deploy to the live cc-ci host (no-CI window)

This is the host-rebuild procedure from memory cc-ci-host-rebuild-procedure — the cc-ci server has no self-service rebuild path, so the orchestrator stages the repo + rebuilds it. The deploy clone on the host is /root/builder-clone; staging a fresh copy avoids disturbing it.

(a) Stage the PR branch on the host:

# from the orchestrator, on the PR branch, clean tree:
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'

(A --delete rsync from a clean tree = exactly the PR head. Excluding .git keeps it a plain path flake — see (c).)

(b) Copy the operator-held sops secrets (NOT in git, required for the build):

ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'

(The age key is at /var/lib/sops-nix/key.txt on the host — already in place from prior rebuilds. Without secrets/secrets.yaml the build FATAs secrets/secrets.yaml does not exist.)

(c) Build first (catch errors before switching):

ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30

• Target is .#cc-ci (= .#cc-ci-hetzner = nix/hosts/cc-ci-hetzner/configuration.nix).
• nixpkgs is now your bumped pin, so this will rebuild more than a same-pin no-op — expect a longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs). That's expected; this is a real nixpkgs bump.
• If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed package), re-push, re-stage, re-build. Do NOT switch a failed build. Common breakages: NixOS module option renames across channels (check the release notes), python package removals, a sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).

(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator host): nixos-rebuild test before any switch:

# detach it so a dropped SSH can't kill the activation mid-flight:
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
  > /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
# poll until reachable again, then check the essentials survived:
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'

test activates the new configuration without touching the bootloader or the system profile, so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good generation. Only proceed to switch once the test activation is up, reachable, and healthy. Note: the detached test leaves a transient nixos-rebuild-switch-to-configuration unit; if the follow-up switch fails with "Unit ... was already loaded", wait for it to finish or systemctl stop nixos-rebuild-switch-to-configuration.service first.

(e) Switch (make it permanent — only after (d) verified):

ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30

This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots (deploy-proxy, warm-keycloak, …) re-run as part of the switch — watch the tail for any that fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip may be transient; a proxy failure is real).

6. Verify the server is healthy (health gate — do not skip)

ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
  echo "=== failed units ==="; systemctl --failed --no-legend; \
  echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
    deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
  echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
  echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
  echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
  curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
  echo "=== disk ==="; df -h / | tail -1'

Compare against the step-1 baseline:

• nixos-version moved (or the current-system path changed) — the bump landed.
• No failed units (the --failed output is empty).
• All 7 infra services active (or active (exited) for the oneshots).
• docker service ls shows no 0/N replicas (everything converged).
• Endpoints 200 (traefik /api/version, dashboard root).
• Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).

If anything regressed: diagnose. A real regression → roll back to the previous generation: ssh cc-ci 'nixos-rebuild switch --rollback' (this switches to the pre-bump system path; the flake input change on the PR branch is still your source of truth, but the running host reverts while you fix). Report the rollback in the PR.

7. Merge the PR + report

On a verified run (health gate green): post a PR comment summarizing what was deployed (revs, any incidents, verification results), then merge the PR directly — the skill invocation is the authorization; the PR exists for visibility and historical record. On a rolled-back or failed run, leave the PR open with a comment explaining the state.

Print + record a one-line result + the PR URL. Write a short note to /srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md:

• RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>
• RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open
• RESULT: FAILED — <step>: <reason>

The end-of-run report to the operator MUST include links to every merged PR plus a summary of what changed in each, so the operator can review them after the fact if they want to.

8. Optional cold-boot proof

A nixos-rebuild switch proves the new generation works on the running kernel. A reboot proves it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this if the operator wants it, and only outside a CI window:

ssh cc-ci 'systemctl reboot'
# wait ~60s, re-check ssh cc-ci + the step-6 health gate

Record the new boot_id if you do this (it's a genuine reboot — log it to cc-ci-plan/REBOOTS.md? No — that file is for the orchestrator host. The cc-ci server's reboots are tracked in its own journal; a note in the server-update report is enough).

Guardrails

• Live rebuild is authorized by invocation. The operator's choice to run this skill IS the go-ahead for the nixos-rebuild switch — do not stop mid-run to re-confirm. A switch cycles reconcile oneshots and can disrupt an in-flight !testme, so prefer a quiescent moment (the step-1 check), but don't gate on a second authorization.
• Single-writer: dedicated branch in a separate clone; never push main; never touch the build loops' /cc-ci / /cc-ci-adv clones or /root/builder-clone (stage to /root/cc-ci-deploy instead).
• Open the PR for visibility, merge it directly. The live rebuild is the deploy; the PR is the reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
• Build before test, test before switch. Never nixos-rebuild switch a build that didn't build cleanly first, and never switch without a healthy nixos-rebuild test activation first — test leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation. This is general policy for nix deploys to the cc-ci server and the orchestrator host.
• Health gate is mandatory. No "it switched, ship it" — run the step-6 verification and compare to the baseline. Roll back on regression.
• Stable channel only for the cc-ci server (never nixos-unstable).
• Keep sops-nix.follows so the two inputs stay aligned; don't fork them.
• Secrets never in git. The operator-held secrets/secrets.yaml is copied on the host only.
• Rollback is always available: nixos-rebuild switch --rollback reverts to the prior generation without touching the flake input revs.

Relationship to an automatic update

This skill is the manual trigger for what should ideally run on a periodic timer: bump the cc-ci flake inputs, build, switch, verify. If/when an automatic cc-ci-server-update.timer is added (NixOS systemd timer on the orchestrator, analog of cc-ci-upgrade-all.timer), it should invoke this skill via a dedicated one-shot agent (the way cc-ci-upgrader runs /upgrade-all). The skill stays as the authoritative procedure either way — automatic just means "run on a schedule without a human prompt."