cc-ci-orchestrator/.opencode/skills/cc-ci-server-update/SKILL.md

---
name: cc-ci-server-update
description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
---

# cc-ci-server-update

Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.

> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
> different machine with its own flake — it is NOT touched here. If you want to update the
> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
> operator). This skill is only about the cc-ci **server**.

## What this skill does NOT do

- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
  `/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
- **Does NOT update the orchestrator host** — only the cc-ci server.
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
  merged directly at the end of a verified run (the skill invocation is the authorization; the
  operator reviews the merged PR afterwards via the link in the report).

## Preconditions / access

- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
  / `cc-ci-tailscaled` proxy) — do NOT proceed.
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
  is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
  oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
  nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
  ```
  ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"'   # any unhealthy task?
  ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep'                                 # a CI run in flight?
  ```
  If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
  (NOT in git). The rebuild needs them — see the rebuild procedure below.

## Procedure

### 1. Snapshot current state (so you can compare + roll back)

```
ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
  deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
  df -h / | tail -1; nixos-version; readlink /run/current-system'
```
Record: the current `nixos-version`, the current-system store path, the list of active infra
services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
failed or an endpoint down gets compared against this.

Also record the **current flake input revs** (the "before" of what you're bumping):
```
ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
```

### 2. Prepare a working clone of the cc-ci repo (single-writer)

Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
fresh one:
```
git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
cd /home/loops/work/cc-ci-server-update
git fetch origin && git checkout main && git reset --hard origin/main
git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
```
(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)

### 3. Bump the flake inputs (deliberate, not drift)

The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
blind `nix flake update --relock-all`.

**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
**stable channel revision** that's compatible:
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
  (`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
  tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
  jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
  (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
  to staying on the same channel unless the operator asks for the jump.
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
  stable nixpkgs. Unstable is for experiments, not production.

**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
and check whether `sops-nix` needs a matching bump:
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
  rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
  the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
  editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
  `nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
  lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
  still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
  style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").

**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
host):
```
cd /home/loops/work/cc-ci-server-update   # (or cc-ci-fix on the new branch)
nix flake lock --update-input nixpkgs --update-input sops-nix   # rewrites flake.lock
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
git diff flake.nix flake.lock | head -40
```
Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
(meaningfully). The real build happens on the cc-ci host in step 5.

### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)

```
git add flake.nix flake.lock
git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))

nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
sops-nix: <old-rev[:8]> → <new-rev[:8]>
<one-line reason — CVE patch / channel bump / sops-nix compat>"
git push -u origin HEAD
```
Open the PR via the helper:
```
set -a; . /srv/cc-ci/.testenv; set +a
TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
```
(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
historical record; it gets merged directly in step 7 once the deploy passes the health gate.)

Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
proceed (the rebuild doesn't depend on the lint gate passing).

### 5. Deploy to the live cc-ci host (no-CI window)

This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.

**(a) Stage the PR branch on the host:**
```
# from the orchestrator, on the PR branch, clean tree:
rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
```
(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
flake — see (c).)

**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
```
ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
```
(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)

**(c) Build first (catch errors before switching):**
```
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
```
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
  longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
  That's expected; this is a real nixpkgs bump.
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
  package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
  module option renames across channels (check the release notes), python package removals, a
  sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).

**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
host): `nixos-rebuild test` before any `switch`:**
```
# detach it so a dropped SSH can't kill the activation mid-flight:
ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
  > /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
# poll until reachable again, then check the essentials survived:
ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
```
`test` activates the new configuration **without touching the bootloader or the system profile**,
so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
`systemctl stop nixos-rebuild-switch-to-configuration.service` first.

**(e) Switch (make it permanent — only after (d) verified):**
```
ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
```
This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
may be transient; a proxy failure is real).

### 6. Verify the server is healthy (health gate — do not skip)

```
ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
  echo "=== failed units ==="; systemctl --failed --no-legend; \
  echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
    deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
  echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
  echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
  echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
  curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
  echo "=== disk ==="; df -h / | tail -1'
```
Compare against the step-1 baseline:
- `nixos-version` moved (or the current-system path changed) — the bump landed.
- **No failed units** (the `--failed` output is empty).
- All 7 infra services `active` (or `active (exited)` for the oneshots).
- `docker service ls` shows no `0/N` replicas (everything converged).
- Endpoints `200` (traefik `/api/version`, dashboard root).
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).

If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
flake input change on the PR branch is still your source of truth, but the running host reverts
while you fix). Report the rollback in the PR.

### 7. Merge the PR + report

On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
any incidents, verification results), then **merge the PR directly** — the skill invocation is the
authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
leave the PR **open** with a comment explaining the state.

Print + record a one-line result + the PR URL. Write a short note to
`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
- `RESULT: FAILED — <step>: <reason>`

The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
what changed in each, so the operator can review them after the fact if they want to.

### 8. Optional cold-boot proof

A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
if the operator wants it, and only outside a CI window:
```
ssh cc-ci 'systemctl reboot'
# wait ~60s, re-check ssh cc-ci + the step-6 health gate
```
Record the new `boot_id` if you do this (it's a genuine reboot — log it to
`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
are tracked in its own journal; a note in the server-update report is enough).

## Guardrails

- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
  go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
  reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
  step-1 check), but don't gate on a second authorization.
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
  build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
  `/root/cc-ci-deploy` instead).
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
  reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
  leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
  cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
  leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
  This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
  to the baseline. Roll back on regression.
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
  generation without touching the flake input revs.

## Relationship to an automatic update

This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
skill stays as the authoritative procedure either way — automatic just means "run on a schedule
without a human prompt."