---
name: cc-ci-orchestrator-update
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
---

# cc-ci-orchestrator-update

Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
a health gate, not as silent drift.

> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
> machine — that's `/cc-ci-server-update`, NOT this skill.

> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
> reconnects the orchestrator — announce a planned reboot to the operator first.

## What this skill does NOT do

- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
  choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
  release notes, and expect module/option renames.

## Preconditions

- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
  `recipe-maintainers/cc-ci-orchestrator`).
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
  would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
  the orchestrator host restarting mostly risks its own sessions, not the CI server).
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
  the bump commit.

## Procedure

### 1. Snapshot baseline

```
nixos-version; readlink /run/current-system; uname -r
systemctl --failed --no-legend
df -h / | tail -1
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
tmux ls
```
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.

### 2. Branch + bump the lock

Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
until a rebuild):
```
cd /srv/cc-ci-orch
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
nix flake update 2>&1 | tail -5      # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
```
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
deploy).

### 3. Build (catch errors before any activation)

```
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
readlink -f result
```
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
build that didn't build cleanly.

### 4. Test-activate (MANDATORY before switch — general policy)

`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
`hetzner-server-recovery`) — lands back on the last-known-good generation.
```
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
  > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
# after it settles (poll; tailscaled/sshd may blip):
readlink /run/current-system            # should be the new store path
ip route show default                   # default route survived
systemctl --failed --no-legend          # 0 failed
tmux ls                                 # sessions survived
tail -5 /tmp/orchestrator-test-switch.log
```
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
switch.

### 5. Switch (make permanent — only after 4 is healthy)

```
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
```
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)

### 6. Health gate (do not skip)

```
nixos-version; readlink /run/current-system
systemctl --failed --no-legend
tmux ls                                          # orchestrator + loop sessions intact
tailscale status | head -3                       # tailnet up
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
ssh cc-ci 'hostname' 2>&1                        # can still reach the CI server
df -h / | tail -1
```
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.

### 7. Commit + PR + merge + report

```
cd /srv/cc-ci-orch
git add flake.lock                                # flake.nix too if the channel ref changed
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))

nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
git push -u origin HEAD
```
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
explanatory comment.

Back on the host, return the checkout to main so the live tree matches the merged state:
```
git checkout main && git pull
```

Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
the operator with the **merged PR link + a summary of what changed**.

## Guardrails

- **Build before test, test before switch** — never `switch` without a healthy `test` activation
  (general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
  reboot always recovers).
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
  kill a half-applied activation.
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
  switch is fine but expect brief tailscaled/sshd blips.
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
  commit; return the live checkout to `main` after the merge.
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
  report merged-PR links + change summaries.
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
  `hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).