recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nix: one Hetzner host for the CI server + the orchestrator (#cc-ci)

Browse Source 
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).

nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.

README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 19:58:33 +00:00
co-authored by Claude Fable 5.1
parent 5bebd13e49
commit 31af820079
32 changed files with 863 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+278 -36
View File
@@ -1,47 +1,289 @@
# cc-ci-orchestrator
Orchestrator workspace for building the **cc-ci** Co-op Cloud recipe CI server. The plan, launch
tooling, and loop prompts live in [`cc-ci-plan/`](cc-ci-plan/); see [`AGENTS.md`](AGENTS.md) for the
roles and operating model. Secrets (`.testenv`) are gitignored — never commit them.
The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.
## Run the orchestrator in tmux (survives disconnects + closing your laptop)
| | where |
|---|---|
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
Keep this supervising session alive on the host with tmux, and use `--remote-control` so you can
watch/steer it from **claude.ai/code** (or the mobile app).
Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
commit them.
---
# Deploying a cc-ci host from scratch
This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
can repeat it. Read it once before starting; the order matters.
## 0. What you need in hand
- A **Hetzner Cloud** project you can create servers in (console login or an API token).
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
`autonomic-bot-gitea-ed25519`, has it).
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
you are starting fresh you create them (each row says how).
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
## 1. Provision the server on Hetzner (Debian image)
In the Hetzner Cloud console (or with `hcloud server create`):
| setting | value | why |
|---|---|---|
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
| Name | `cc-ci` | becomes the hostname |
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
Check you can log in: `ssh root@<ip> hostname`.
## 2. Convert Debian → NixOS with nixos-infect
[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
reboots. Run it detached so the SSH session dropping does not kill it:
```bash
# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
# /exit (inside Claude) or Ctrl-D
# 1. Start a detachable tmux session on this host
tmux new -s orchestrator
# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
claude --resume autonomous-orchestrator \
--remote-control "autonomous-orchestrator" \
--dangerously-skip-permissions
# - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
# - Or resume by the stable session id (more deterministic in a fresh pane):
# claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
# --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# 3. Detach — the process keeps running: press Ctrl-b, then d
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
export NIX_CHANNEL="nixos-26.05" # must match the nixpkgs channel in flake.nix
export PROVIDER="hetznercloud" # GRUB + Hetzner networking
export NIXOS_IMPORT="" # the real config comes from the flake in §5
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
export NO_SWAP=true
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
EOF
chmod +x /root/infect.sh
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
```
**Reconnect later**
- On this host: `tmux attach -t orchestrator`
- From anywhere: **claude.ai/code** → the `autonomous-orchestrator` session
It downloads Nix, builds a NixOS system (5–10 min; follow with
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.
**Why it survives:** tmux keeps the `claude` process alive across SSH disconnects and your laptop
closing; remote-control runs *outbound* from this host to Anthropic, so it stays connected
regardless of the viewer. After a host reboot, re-run steps 1–2.
> If the box does not come back within ~5 minutes, open the Hetzner console (or rescue mode) and
> look at the boot messages — the two known failure modes are a broken bootloader install and a
> wrong `networking.nix` (see §3, the `defaultGateway.interface` note).
> Two different "names": `--resume <name|id>` selects the *conversation* to restore (shown in the
> `/resume` picker); the `--remote-control "<name>"` value is only the web display label and resumes
> nothing. Resuming reuses the same session id each time (stays `34a8…`) — don't pass
> `--fork-session` unless you intend to branch a new conversation.
>
> Already inside a live session and just want the web surface? Run `/remote-control` — no exit/resume.
## 3. Capture the machine-specific config into this repo
nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
first two matter; the flake replaces `configuration.nix`.
```bash
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
scp root@<ip>:/etc/nixos/networking.nix nix/hosts/cc-ci/networking.nix
```
Then in `nix/hosts/cc-ci/`:
- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
`/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
`networking.defaultGateway` has **both** `address` and `interface = "eth0"` — since NixOS 25.05
omitting the interface leaves the host without a default route after a rebuild (this took the
old CI server offline once; rescue mode fixed it). If the generated IPv6 block has an empty
address, delete the IPv6 parts.
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
is the release you installed (never change it later).
- `ssh-keys`: the root keys.
Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
## 4. Stage the workspace and secrets on the new host
Everything in this section is **outside git**. Do it as root over SSH, in this order.
### 4a. Tailscale
```bash
# a reusable (or fresh) tailnet auth key from the tailscale admin console
install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
```
### 4b. The CI server's checkout and its one out-of-band secret
```bash
# root's deploy key for the private cc-ci-secrets submodule
install -d -m700 /root/.ssh
install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
cat > /root/.ssh/config <<'EOF'
Host git.autonomic.zone
Port 2222
User git
IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
IdentitiesOnly yes
EOF
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
# the master (recovery) age key — the only sops recipient a fresh host can be
install -d -m700 /var/lib/sops-nix
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
```
`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.
### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
run §5 once first if this is a fresh host, then come back)
```bash
sudo -iu loops
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci # every script and unit says /srv/cc-ci
cd /srv/cc-ci-orch
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci # Builder clone
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv # Adversary clone
mkdir -p .cc-ci-logs .sops
```
Then the files below (`install -m600 -o loops -g users`):
| file | what | source |
|---|---|---|
| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them** | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
## 5. Build and activate
From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
```bash
sudo -iu loops
cd /srv/cc-ci-orch && git checkout <branch>
sudo nixos-rebuild test --flake .#cc-ci # activates WITHOUT touching the bootloader
```
`test` first, always: if the activation breaks networking or sshd, a reboot from the Hetzner
console lands on the last known-good generation. Over an SSH session that itself rides tailscale,
run it detached (`systemd-run --unit=ccci-sw --collect nixos-rebuild test --flake .#cc-ci`) so the
tailscale restart during activation does not abort it.
The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
```bash
systemctl is-system-running # running — or list-units --failed and read journalctl -u <unit>
tailscale status | head -3
docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
systemctl list-timers 'cc-ci-*' nightly-sweep
sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active)
```
When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
the new box does not process PR comments or start a second weekly run while the old host is live:
```bash
docker service scale ccci-bridge_app=0
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
```
## 6. Migrating: restore state from the previous host
Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
stopped on the new host while its directory is copied:
| path | holds | notes |
|---|---|---|
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
| `/var/lib/acme-dns` | the acme-dns zone DB | |
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
## 7. Cutover and verification
1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
are. Wait for propagation (`dig +short ci.commoninternet.net`).
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
(acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
(dashboard, valid cert), same for `report.ci` and `drone.ci`.
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
`systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
orchestrator. New host: `docker service scale ccci-bridge_app=1`;
`systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
(or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
## 8. Day 2
- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
`nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
`nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
so the deployed checkout the sweep runs from matches.
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
`switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
`/hetzner-server-recovery`).
---
# Operating the orchestrator session
The steering session is a long-lived interactive Claude Code session under tmux with
`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
(backend + model in `cc-ci-plan/agents.toml`).
```bash
# attach on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator
# or resume the conversation by hand in a fresh tmux pane
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# already inside a live session and just want the web surface? /remote-control
```
`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
only the web display label. Don't pass `--fork-session` unless you mean to branch.
## Kick off / supervise the loops
@@ -53,5 +295,5 @@ cd /srv/cc-ci/cc-ci-plan
./launch.sh stop
```
Full supervision guide, credential map, and the Incus VM fallback are in
[`cc-ci-plan/kickoff.md`](cc-ci-plan/kickoff.md) and [`cc-ci-plan/plan.md`](cc-ci-plan/plan.md) §1.5.
Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.