recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

weekly health-gated auto-update of the cc-ci host; skills rewritten for the combined host

Browse Source 
nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday
03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone
builds → skip), `nix flake update` → build → switch-to-configuration test →
cc-ci health checks (sshd, 0 failed units, core units, every swarm service at
replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401)
→ profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci
fast-forwarded; revert + lock restore on failure; one-line state file for
/cc-ci-status.

Skills (.opencode canonical, .claude pointers' descriptions synced):
- cc-ci-orchestrator-update: THE host update — drives the auto-update unit by
  hand; --cc-ci-only for a cc-ci-main-only move; failure playbook.
- cc-ci-server-update: delegates to it and explains why the old procedure
  (rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host.
- cc-ci-update: chains orchestrator-update then tests-update.
- cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot
  generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator
  agent session, secrets inventory; verdict updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 22:22:50 +00:00
co-authored by Claude Fable 5.1
parent 48226dd78f
commit a8af8429f2
11 changed files with 366 additions and 476 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.opencode/skills/cc-ci-orchestrator-update/SKILL.md
+66 -143
View File
@@ -1,164 +1,87 @@
---
name: cc-ci-orchestrator-update
description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
---
# cc-ci-orchestrator-update
Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
a health gate, not as silent drift.
Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet):
the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the
orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.
> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo
> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci
> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
> machine — that's `/cc-ci-server-update`, NOT this skill.
> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in
> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own
> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow
> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host:
> that is the standalone-server config and would replace the combined one (orchestrator half gone).
> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
> reconnects the orchestrator — announce a planned reboot to the operator first.
## What this skill does NOT do
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
release notes, and expect module/option renames.
The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit
the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence —
busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health
check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm
service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) →
profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` →
`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file.
## Preconditions
- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
`recipe-maintainers/cc-ci-orchestrator`).
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
the orchestrator host restarting mostly risks its own sessions, not the CI server).
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
the bump commit.
- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host.
- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone
builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and
`docker service ls` to see what is running. Never kill a run to update a host.
- Note the current generation so you can name the old one in the report:
`ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`.
## Procedure
### 1. Snapshot baseline
```bash
# 1. what will move
ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12
```
nixos-version; readlink /run/current-system; uname -r
systemctl --failed --no-legend
df -h / | tail -1
cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
tmux ls
```
Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update' # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
# or, non-interactively:
ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'
### 2. Branch + bump the lock
Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
until a rebuild):
```
cd /srv/cc-ci-orch
git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
nix flake update 2>&1 | tail -5 # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
```
Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
deploy).
### 3. Build (catch errors before any activation)
```
cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15
readlink -f result
```
Build failure → fix on the branch (option renames etc.) before going further. Never activate a
build that didn't build cleanly.
### 4. Test-activate (MANDATORY before switch — general policy)
`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
`hetzner-server-recovery`) — lands back on the last-known-good generation.
```
cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \
> /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
# after it settles (poll; tailscaled/sshd may blip):
readlink /run/current-system # should be the new store path
ip route show default # default route survived
systemctl --failed --no-legend # 0 failed
tmux ls # sessions survived
tail -5 /tmp/orchestrator-test-switch.log
```
Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
switch.
### 5. Switch (make permanent — only after 4 is healthy)
```
cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10
```
(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
### 6. Health gate (do not skip)
```
nixos-version; readlink /run/current-system
systemctl --failed --no-legend
tmux ls # orchestrator + loop sessions intact
tailscale status | head -3 # tailnet up
systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
ssh cc-ci 'hostname' 2>&1 # can still reach the CI server
df -h / | tail -1
```
Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
### 7. Commit + PR + merge + report
```
cd /srv/cc-ci-orch
git add flake.lock # flake.nix too if the channel ref changed
git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green."
git push -u origin HEAD
```
Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
explanatory comment.
Back on the host, return the checkout to main so the live tree matches the merged state:
```
git checkout main && git pull
# 3. read the outcome
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'
```
Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
the operator with the **merged PR link + a summary of what changed**.
`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now):
```bash
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
# then the same step-3 checks, plus the §6 health commands from /cc-ci-status
```
## When it fails
`result=failed note=<step>` means the unit already re-activated the previous generation and
restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the
`HEALTH FAIL:` / `REVERT:` line:
- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate
against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci`
in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run.
- **swarm services not at their replica count** — one of the reconcile oneshots did not converge
in 10 min: `journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>`, `docker service ps <svc>`.
- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake
update cc-ci` on the host and re-run (README day-2).
- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops.
- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the
unit or the host mid-switch (see the `stopIfChanged` comment in the module).
## Report (end of run)
- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit
on `main`, the health check result, and anything the journal warned about.
- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not.
## Guardrails
- **Build before test, test before switch** — never `switch` without a healthy `test` activation
(general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
reboot always recovers).
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
kill a half-applied activation.
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
switch is fine but expect brief tailscaled/sshd blips.
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
commit; return the live checkout to `main` after the merge.
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
report merged-PR links + change summaries.
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
`hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless
`--cc-ci-only` says so, and then `test` first.
- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ
between Nix versions) — locks for the host are made on the host.
- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate.