---
name: cc-ci-orchestrator-update
description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
---

# cc-ci-orchestrator-update

Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet):
the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the
orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.

> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in
> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own
> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow
> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host:
> that is the standalone-server config and would replace the combined one (orchestrator half gone).

The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit
the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence —
busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health
check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm
service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) →
profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` →
`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file.

## Preconditions

- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host.
- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone
  builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and
  `docker service ls` to see what is running. Never kill a run to update a host.
- Note the current generation so you can name the old one in the report:
  `ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`.

## Procedure

```bash
# 1. what will move
ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12

# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update'   # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
# or, non-interactively:
ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'

# 3. read the outcome
ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'
```

`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now):
```bash
ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
# then the same step-3 checks, plus the §6 health commands from /cc-ci-status
```

## When it fails

`result=failed note=<step>` means the unit already re-activated the previous generation and
restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the
`HEALTH FAIL:` / `REVERT:` line:
- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate
  against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci`
  in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run.
- **swarm services not at their replica count** — one of the reconcile oneshots did not converge
  in 10 min: `journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>`, `docker service ps <svc>`.
- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake
  update cc-ci` on the host and re-run (README day-2).
- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops.
- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the
  unit or the host mid-switch (see the `stopIfChanged` comment in the module).

## Report (end of run)

- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit
  on `main`, the health check result, and anything the journal warned about.
- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not.

## Guardrails

- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless
  `--cc-ci-only` says so, and then `test` first.
- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ
  between Nix versions) — locks for the host are made on the host.
- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate.