skills: PR-merge policy + test-before-switch + new /cc-ci-orchestrator-update
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change — open PRs for visibility/historical record and merge directly once verified (invocation = authorization); reports must list merged-PR links + change summaries. (tests-update still never merges the paired recipe upgrade PRs.) - cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy. - AGENTS.md: test-before-switch policy for orchestrator host rebuilds. - NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner server 134487234), with self-update caveats. - JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
This commit is contained in:
autonomic-bot
@@ -0,0 +1,164 @@
|
||||
---
|
||||
name: cc-ci-orchestrator-update
|
||||
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
|
||||
---
|
||||
|
||||
# cc-ci-orchestrator-update
|
||||
|
||||
Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
|
||||
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
|
||||
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
|
||||
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
|
||||
a health gate, not as silent drift.
|
||||
|
||||
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
|
||||
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
|
||||
> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
|
||||
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
|
||||
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
|
||||
> machine — that's `/cc-ci-server-update`, NOT this skill.
|
||||
|
||||
> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
|
||||
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
|
||||
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
|
||||
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
|
||||
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
|
||||
> reconnects the orchestrator — announce a planned reboot to the operator first.
|
||||
|
||||
## What this skill does NOT do
|
||||
|
||||
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
|
||||
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
|
||||
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
|
||||
choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
|
||||
release notes, and expect module/option renames.
|
||||
|
||||
## Preconditions
|
||||
|
||||
- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
|
||||
`recipe-maintainers/cc-ci-orchestrator`).
|
||||
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
|
||||
would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
|
||||
the orchestrator host restarting mostly risks its own sessions, not the CI server).
|
||||
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
|
||||
the bump commit.
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Snapshot baseline
|
||||
|
||||
```
|
||||
nixos-version; readlink /run/current-system; uname -r
|
||||
systemctl --failed --no-legend
|
||||
df -h / | tail -1
|
||||
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
tmux ls
|
||||
```
|
||||
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
|
||||
|
||||
### 2. Branch + bump the lock
|
||||
|
||||
Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
|
||||
until a rebuild):
|
||||
```
|
||||
cd /srv/cc-ci-orch
|
||||
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
|
||||
nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
|
||||
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
|
||||
```
|
||||
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
|
||||
deploy).
|
||||
|
||||
### 3. Build (catch errors before any activation)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
|
||||
readlink -f result
|
||||
```
|
||||
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
|
||||
build that didn't build cleanly.
|
||||
|
||||
### 4. Test-activate (MANDATORY before switch — general policy)
|
||||
|
||||
`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
|
||||
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
|
||||
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
|
||||
`hetzner-server-recovery`) — lands back on the last-known-good generation.
|
||||
```
|
||||
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
|
||||
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
|
||||
# after it settles (poll; tailscaled/sshd may blip):
|
||||
readlink /run/current-system # should be the new store path
|
||||
ip route show default # default route survived
|
||||
systemctl --failed --no-legend # 0 failed
|
||||
tmux ls # sessions survived
|
||||
tail -5 /tmp/orchestrator-test-switch.log
|
||||
```
|
||||
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
|
||||
switch.
|
||||
|
||||
### 5. Switch (make permanent — only after 4 is healthy)
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
|
||||
```
|
||||
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
|
||||
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
|
||||
|
||||
### 6. Health gate (do not skip)
|
||||
|
||||
```
|
||||
nixos-version; readlink /run/current-system
|
||||
systemctl --failed --no-legend
|
||||
tmux ls # orchestrator + loop sessions intact
|
||||
tailscale status | head -3 # tailnet up
|
||||
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
|
||||
ssh cc-ci 'hostname' 2>&1 # can still reach the CI server
|
||||
df -h / | tail -1
|
||||
```
|
||||
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
|
||||
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
|
||||
|
||||
### 7. Commit + PR + merge + report
|
||||
|
||||
```
|
||||
cd /srv/cc-ci-orch
|
||||
git add flake.lock # flake.nix too if the channel ref changed
|
||||
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
|
||||
|
||||
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
|
||||
Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
|
||||
git push -u origin HEAD
|
||||
```
|
||||
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
|
||||
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
|
||||
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
|
||||
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
|
||||
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
|
||||
explanatory comment.
|
||||
|
||||
Back on the host, return the checkout to main so the live tree matches the merged state:
|
||||
```
|
||||
git checkout main && git pull
|
||||
```
|
||||
|
||||
Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
|
||||
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
|
||||
the operator with the **merged PR link + a summary of what changed**.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Build before test, test before switch** — never `switch` without a healthy `test` activation
|
||||
(general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
|
||||
reboot always recovers).
|
||||
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
|
||||
kill a half-applied activation.
|
||||
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
|
||||
switch is fine but expect brief tailscaled/sshd blips.
|
||||
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
|
||||
commit; return the live checkout to `main` after the merge.
|
||||
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
|
||||
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
|
||||
report merged-PR links + change summaries.
|
||||
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
|
||||
`hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
|
||||
Reference in New Issue
Block a user