cc-ci-orchestrator/.opencode/skills/cc-ci-orchestrator-update/SKILL.md

name, description

name

description

cc-ci-orchestrator-update

Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.

cc-ci-orchestrator-update

Update the cc-ci host — one machine since 2026-09-07 (cc-ci, 195.201.88.249, no tailnet): the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.

One flake, one target. The host is nixosConfigurations.cc-ci in recipe-maintainers/cc-ci-orchestrator (/srv/cc-ci-orch on the host). The cc-ci repo's own flake.nix inputs are IRRELEVANT to the live build — this flake makes the cc-ci input follow its nixpkgs/sops-nix. Never run nixos-rebuild on the cc-ci repo's #cc-ci target on this host: that is the standalone-server config and would replace the combined one (orchestrator half gone).

The work is done by cc-ci-auto-update.service (nix/modules/auto-update.nix), the same unit the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence — busy gate → nix flake update → build → switch-to-configuration test → 45 s settle → health check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) → profile + bootloader → flake.lock committed as auto-update and pushed to main → /etc/cc-ci fast-forwarded — with a journal to read and a one-line state file.

Preconditions

• ssh cc-ci works from the notplants-orchestrator box (root on the host), or you are on the host.
• Not busy. The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone builds → result=skipped note=busy, exit 0). Don't fight it: wait, or tmux ls as loops and docker service ls to see what is running. Never kill a run to update a host.
• Note the current generation so you can name the old one in the report: ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'.

Procedure

# 1. what will move
ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12

# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update'   # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
# or, non-interactively:
ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'

# 3. read the outcome
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'

--cc-ci-only (after merging something in the cc-ci repo that the host should run now):

ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
# then the same step-3 checks, plus the §6 health commands from /cc-ci-status

When it fails

result=failed note=<step> means the unit already re-activated the previous generation and restored flake.lock; the host is as it was. Read journalctl -u cc-ci-auto-update for the HEALTH FAIL: / REVERT: line:

• build failed — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate against the new nixpkgs. Reproduce: sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci in the checkout after nix flake update; fix in the right repo (PR, merge), re-run.
• swarm services not at their replica count — one of the reconcile oneshots did not converge in 10 min: journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>, docker service ps <svc>.
• NAR hash mismatch on the cc-ci input — the lock was made on another machine; nix flake update cc-ci on the host and re-run (README day-2).
• push failed — bookkeeping only (the generation is committed): push the lock by hand as loops.
• A hung run: the unit has a 3 h timeout; systemctl status cc-ci-auto-update. Do NOT restart the unit or the host mid-switch (see the stopIfChanged comment in the module).

Report (end of run)

• old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit on main, the health check result, and anything the journal warned about.
• Then /cc-ci-status §5–§6 should read ok — say if it does not.

Guardrails

• The unit is the only thing that switches here; do not hand-roll nixos-rebuild switch unless --cc-ci-only says so, and then test first.
• Never nix flake update from the notplants box and push a lock for this host (NAR hashes differ between Nix versions) — locks for the host are made on the host.
• Nothing recipe-side: no PRs merged, no tests touched; /cc-ci-tests-update is separate.