cc-ci-orchestrator/.opencode/skills/cc-ci-orchestrator-update/SKILL.md

name, description

name

description

cc-ci-orchestrator-update

Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).

cc-ci-orchestrator-update

Manually update the orchestrator host's NixOS dependencies (nixpkgs; sops-nix follows nixpkgs) to the current nixos-26.05 channel tip, ship the bump via a verified PR + a live nixos-rebuild test → switch, and confirm the host is healthy afterward. This is the orchestrator-host analogue of /cc-ci-server-update — done deliberately, with a reviewable PR and a health gate, not as silent drift.

Two hosts, two flakes — don't confuse them. This skill updates the orchestrator host: the machine this session runs on (cc-ci-orchestrator-1, Hetzner cpx22 server 134487234, tailnet 100.84.190.30, public 168.119.126.100), flake checkout /srv/cc-ci-orch (repo recipe-maintainers/cc-ci-orchestrator), target .#cc-ci-orchestrator-hetzner. The cc-ci CI server (ssh cc-ci, repo recipe-maintainers/cc-ci, target .#cc-ci) is a different machine — that's /cc-ci-server-update, NOT this skill.

This is a SELF-update. The switch restarts units on the host running this very session. tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but tailscaled/sshd restarts can briefly drop Remote Control / SSH — run activation steps detached (setsid nohup … &) and re-poll. A full reboot DOES end the session: the auto-launch flow (REBOOTS.md + cc-ci-loops.service) restarts the loops, and the operator reconnects the orchestrator — announce a planned reboot to the operator first.

What this skill does NOT do

• Does NOT touch the cc-ci CI server — that's /cc-ci-server-update.
• Does NOT bump recipe images or cc-ci tests — /upgrade-all / /cc-ci-tests-update.
• Does NOT change the channel (nixos-26.05 branch ref in flake.nix) without operator choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the release notes, and expect module/option renames.

Preconditions

• You are on the orchestrator host; /srv/cc-ci-orch is the live checkout (git repo, remote recipe-maintainers/cc-ci-orchestrator).
• Quiescent moment: no /upgrade-all in flight, no active build-loop phase, no cc-ci CI run that would be disturbed if this host blips (pgrep -af run_recipe_ci empty on cc-ci is a bonus, but the orchestrator host restarting mostly risks its own sessions, not the CI server).
• git status in /srv/cc-ci-orch — note pre-existing dirt; don't sweep unrelated changes into the bump commit.

Procedure

1. Snapshot baseline

nixos-version; readlink /run/current-system; uname -r
systemctl --failed --no-legend
df -h / | tail -1
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
tmux ls

Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.

2. Branch + bump the lock

Work directly in /srv/cc-ci-orch on a branch (it's the live checkout; the flake edit is inert until a rebuild):

cd /srv/cc-ci-orch
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
nix flake update 2>&1 | tail -5      # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'

Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to deploy).

3. Build (catch errors before any activation)

cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
readlink -f result

Build failure → fix on the branch (option renames etc.) before going further. Never activate a build that didn't build cleanly.

4. Test-activate (MANDATORY before switch — general policy)

nixos-rebuild test activates without touching the bootloader or system profile, so if the activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain reboot — Hetzner API power-cycle on server 134487234 if SSH is gone (see hetzner-server-recovery) — lands back on the last-known-good generation.

cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
  > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
# after it settles (poll; tailscaled/sshd may blip):
readlink /run/current-system            # should be the new store path
ip route show default                   # default route survived
systemctl --failed --no-legend          # 0 failed
tmux ls                                 # sessions survived
tail -5 /tmp/orchestrator-test-switch.log

Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT switch.

5. Switch (make permanent — only after 4 is healthy)

cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10

(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the detached test's transient unit is still running — wait or systemctl stop it, then retry.)

6. Health gate (do not skip)

nixos-version; readlink /run/current-system
systemctl --failed --no-legend
tmux ls                                          # orchestrator + loop sessions intact
tailscale status | head -3                       # tailnet up
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
ssh cc-ci 'hostname' 2>&1                        # can still reach the CI server
df -h / | tail -1

Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.

7. Commit + PR + merge + report

cd /srv/cc-ci-orch
git add flake.lock                                # flake.nix too if the channel ref changed
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))

nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
git push -u origin HEAD

Open the PR on recipe-maintainers/cc-ci-orchestrator (Gitea API with the GITEA_* creds from /srv/cc-ci/.testenv, same as open-cc-ci-pr.sh does for cc-ci — the helper is hardcoded to the cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what was verified. Then merge it directly — the PR is for visibility/historical record; the skill invocation is the authorization. On a rolled-back/failed run, leave the PR open with an explanatory comment.

Back on the host, return the checkout to main so the live tree matches the merged state:

git checkout main && git pull

Write /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md (RESULT: SUCCESS/ROLLED-BACK/FAILED …), append a line to cc-ci-plan/JOURNAL.md, and report to the operator with the merged PR link + a summary of what changed.

Guardrails

• Build before test, test before switch — never switch without a healthy test activation (general policy for this host and the cc-ci server; test keeps the bootloader untouched so a reboot always recovers).
• Detach activations (setsid nohup … &) — a dropped SSH/Remote-Control connection must not kill a half-applied activation.
• Self-update awareness: announce before any reboot (it ends the operator's session); a switch is fine but expect brief tailscaled/sshd blips.
• Single-writer: dedicated branch; don't sweep unrelated working-tree changes into the bump commit; return the live checkout to main after the merge.
• Stable channel only (nixos-26.05 until the operator chooses a jump).
• PRs for visibility, merged directly once verified; failed runs leave the PR open. Always report merged-PR links + change summaries.
• Rescue path: Hetzner server 134487234 — same rescue-mode procedure as hetzner-server-recovery (grub generations live in a submenu: 1>N ids).