recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nix: one Hetzner host for the CI server + the orchestrator (#cc-ci)

Browse Source 
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).

nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.

README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 19:58:33 +00:00
co-authored by Claude Fable 5.1
parent 5bebd13e49
commit 31af820079
32 changed files with 863 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+11 -8
View File
@@ -30,15 +30,18 @@ the orchestrator watches from outside.
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
operator reconnects to it (that's why the startup notification matters). The orchestrator now runs on
a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.30`, public
`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) — the operator reconnects to it (that's
why the startup notification matters). Since 2026-09 the orchestrator runs on the **same Hetzner
host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
The full provisioning + deploy guide is `README.md`; the move is recorded in
`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
shared with notplants) are in `archive/`. Rebuild this host with
`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
before the `switch`** (general policy for nix deploys to this host: `test`
leaves the bootloader and system profile untouched, so a reboot always recovers to the
last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
`.cc-ci-logs/server-update-2026-08-03.md`).