cc-ci-orchestrator

The cc-ci orchestrator: the agent loops that built the cc-ci Co-op Cloud recipe CI server, the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS host they run on. Since 2026-09 that host is the same Hetzner server as the CI server itself: one nixos-rebuild from this repo builds both, because this flake imports the CI server as a module from the cc-ci repo.

	where
Orchestrator loops, timers (weekly upgrader, hourly supervisor)	nix/modules/cc-ci.nix → nixosModules.cc-ci-orchestrator
The host contract those need (loops user, claude/opencode CLIs, opencode web UI)	nix/modules/orchestrator-host.nix → nixosModules.orchestrator-host
The CI server (swarm, traefik, drone, runner, !testme bridge, dashboard, reports, acme-dns)	cc-ci repo nix/modules/ → nixosModules.cc-ci-server (flake input cc-ci)
The machine: hardware, networking, tailscale, root keys	nix/hosts/cc-ci/ → nixosConfigurations.cc-ci
Plans, launch tooling, loop prompts, journal	cc-ci-plan/ (see AGENTS.md for roles)
Skills the orchestrator runs (/upgrade-all, /recipe-upgrade, /cc-ci-status, …)	.claude/skills/, .opencode/skills/
How it used to be built (Pi → Incus VM → shared Hetzner box)	archive/

Secrets (.testenv, upgrader.env, .sops/, everything under /secrets) are gitignored — never commit them.

---

Deploying a cc-ci host from scratch

This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner server. It was last done on 2026-09-07 for 195.201.88.249 and is written so a person or an LLM can repeat it. Read it once before starting; the order matters.

0. What you need in hand

• A Hetzner Cloud project you can create servers in (console login or an API token).
• SSH keys: yours, and the orchestrator's own key so the automation can reach the box. The public keys that get root are tracked in nix/hosts/cc-ci/ssh-keys (one per line).
• Read access to recipe-maintainers/cc-ci, recipe-maintainers/cc-ci-orchestrator (both public read) and the private recipe-maintainers/cc-ci-secrets (the autonomic-bot deploy key, autonomic-bot-gitea-ed25519, has it).
• The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if you are starting fresh you create them (each row says how).
• Control of the DNS zone (Gandi for commoninternet.net) for the cutover in §7.

1. Provision the server on Hetzner (Debian image)

In the Hetzner Cloud console (or with hcloud server create):

setting	value	why
Image	Debian 13 (any recent Debian/Ubuntu works with nixos-infect)	it is replaced by NixOS in §2
Type	x86, 8 GB RAM, 4 vCPU — e.g. cpx32 (dedicated AMD) or cx33. Never cax* (ARM): the flakes are x86_64-linux.	swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small
Disk	the type's default 150+ GB NVMe	docker layers alone are ~60 GB after a few weeks
Network	public IPv4 required; IPv6 optional (leave enabled or not, NixOS config ignores it)	cc-ci serves 80/443 and DNS on 53 publicly
SSH keys	add every key from nix/hosts/cc-ci/ssh-keys you want to log in with, at least the orchestrator's	nixos-infect carries /root/.ssh/authorized_keys over
Name	cc-ci	becomes the hostname
Firewall	if a Hetzner Cloud Firewall is attached it must allow 22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp in, and ICMP	the NixOS firewall is separate and is configured by the flake

Check you can log in: ssh root@<ip> hostname.

2. Convert Debian → NixOS with nixos-infect

nixos-infect installs NixOS over the running Debian and reboots. Run it detached so the SSH session dropping does not kill it:

ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
export NIXOS_IMPORT=""             # the real config comes from the flake in §5
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
export NO_SWAP=true
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
EOF
chmod +x /root/infect.sh
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'

It downloads Nix, builds a NixOS system (5–10 min; follow with ssh root@<ip> tail -f /var/log/nixos-infect.log), then reboots. The SSH host key changes: ssh-keygen -R <ip> and confirm ssh root@<ip> nixos-version prints a 26.05 version.

2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

All three bit on the first attempt; the script above and §3 already include the fixes, this is so you recognise them if they come back in another form.

1. swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument right at the start, script exits. The Debian 13 cloud image mounts /tmp as tmpfs and a swapfile cannot live there. Fix: NO_SWAP=true (in the script above). An 8 GB box does not need the temporary swap.
2. The box never comes back after the reboot: it boots NixOS, but nearly every unit fails (dbus, systemd-logind, sshd, networking …) with Could not start dynamically linked executable: /usr/bin/dbus-daemon in the journal. nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it to /old-root (/etc/NIXOS_LUSTRATE). With NixOS 26.05's systemd-based initrd that lustration did not happen, so Debian's /etc/systemd/system/*.service files shadowed the NixOS units and started Debian binaries. Fix, from Hetzner rescue mode (enable_rescue + reset in the API/console, ssh in, mount /dev/sda1 /mnt/root): move everything except nix, boot, swapfile, lost+found, var/log, var/empty, etc/nixos, etc/resolv.conf, etc/NIXOS, etc/machine-id, etc/ssh/ssh_host_*, root/.nix-*, root/.ssh into /mnt/root/old-root, delete etc/NIXOS_LUSTRATE, unmount, disable_rescue, reset. (/old-root, ~1 GB, can be deleted once the host is in service.)
3. Boots, units fine, but no network. The generated networking.nix has defaultGateway = "172.31.1.1"; — a bare string. Since NixOS 25.05 that yields no default route. Fix: defaultGateway = { address = "172.31.1.1"; interface = "eth0"; }; (this is what nix/hosts/cc-ci/networking.nix carries). To apply it from rescue mode, chroot into the mounted root and rebuild the boot entry — the nix sandbox cannot pivot_root inside a chroot, so turn it off for that one build:

   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
     ln -sfn /nix/var/nix/profiles/system /run/current-system
     nixos-rebuild boot --option sandbox false'
   

   The journalctl -D /mnt/root/var/log/journal -b 0 trick (reading the dead system's journal from rescue mode) is what told these apart.

Rescue mode without a console: POST /servers/<id>/actions/enable_rescue with your ssh key id, then …/actions/reset; afterwards disable_rescue and check rescue_enabled is false before the next reset, or it boots the rescue image again. scripts/recovery/hetzner.py wraps these (token in /srv/cc-ci/.hcloud-token).

3. Capture the machine-specific config into this repo

nixos-infect wrote /etc/nixos/{hardware-configuration,networking,configuration}.nix. Only the first two matter; the flake replaces configuration.nix.

scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix

Then in nix/hosts/cc-ci/:

• hardware.nix: keep as generated (GRUB EFI with efiInstallAsRemovable, /boot/efi by UUID, /dev/sda1 root). Do not copy another host's file — the UUIDs are per machine.
• networking.nix: keep the static IPv4 + Hetzner gateway 172.31.1.1. Make sure networking.defaultGateway has both address and interface = "eth0" (§2a item 3). If the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address (as on the 2026-09 box) can stay.
• configuration.nix: set cc-ci.publicIPv4 to the server's IPv4 and check system.stateVersion is the release you installed (never change it later).
• ssh-keys: the root keys.

Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.

4. Stage the workspace and secrets on the new host

Everything in this section is outside git. Do it as root over SSH, in this order.

4a. Tailscale

# a reusable (or fresh) tailnet auth key from the tailscale admin console
install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'

4b. The CI server's checkout and its one out-of-band secret

# root's deploy key for the private cc-ci-secrets submodule
install -d -m700 /root/.ssh
install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
cat > /root/.ssh/config <<'EOF'
Host git.autonomic.zone
    Port 2222
    User git
    IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
    IdentitiesOnly yes
EOF
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
# the master (recovery) age key — the only sops recipient a fresh host can be
install -d -m700 /var/lib/sops-nix
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt

/etc/cc-ci/secrets/secrets.yaml is encrypted to the master key and the old host's SSH host key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient: ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub, add it to secrets/.sops.yaml in cc-ci-secrets, sops updatekeys secrets.yaml, push, git -C /etc/cc-ci submodule update --remote.

4c. The orchestrator's workspace (as the loops user — it exists after the first rebuild, so

run §5 once first if this is a fresh host, then come back)

sudo -iu loops
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
cd /srv/cc-ci-orch
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
mkdir -p .cc-ci-logs .sops

Then the files below (install -m600 -o loops -g users):

file	what	source
/srv/cc-ci/.testenv	TS_AUTH_KEY, GITEA_PASSWORD (autonomic-bot), DOCKERHUB_USERNAME/TOKEN, model API keys	old host /secrets/files/cc-ci.testenv; fresh: create each credential
/srv/cc-ci/upgrader.env	LOOP_TIER, LOOP_MODEL, REPORT_MODEL for the weekly run (no secrets)	old host, or copy the example in AGENTS.md
/srv/cc-ci/.sops/master-age.txt	the same master age key as 4b (skills that re-key secrets use it)	old host
~loops/.ssh/cc-ci-root-ed25519 (+.pub)	ssh cc-ci as root — to loopback on this host	old host; fresh: ssh-keygen -t ed25519 and add the pub to nix/hosts/cc-ci/ssh-keys
~loops/.ssh/autonomic-bot-gitea-ed25519 (+.pub)	pushes recipe branches / PRs as autonomic-bot	old host; fresh: new key added to the bot's Gitea account
~loops/.ssh/tangled-ed25519	optional, tangled.org mirrors	old host
~loops/.claude/	Claude Code auth + settings + the orchestrator session history	old host (rsync -a); fresh: claude auth login as loops (device code, interactive)
~loops/.local/share/opencode/auth.json, ~loops/.config/opencode/	opencode provider auth (the weekly upgrader runs on opencode)	old host; fresh: opencode auth login
/etc/nginx/oc-selfsigned.{crt,key}, /etc/nginx/oc-htpasswd	the tailnet-only opencode UI; nginx refuses to start without them	old host, or generate (commands in nix/modules/orchestrator-host.nix)

~loops/.ssh/config is written by the activation script on first rebuild (Host cc-ci → 127.0.0.1, git.autonomic.zone, tangled.org); it is not overwritten if present.

5. Build and activate

From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):

sudo -iu loops
cd /srv/cc-ci-orch && git checkout <branch>
sudo nixos-rebuild test --flake .#cc-ci     # activates WITHOUT touching the bootloader

test first, always: if the activation breaks networking or sshd, a reboot from the Hetzner console lands on the last known-good generation. Over an SSH session that itself rides tailscale, run it detached (systemd-run --unit=ccci-sw --collect nixos-rebuild test --flake .#cc-ci) so the tailscale restart during activation does not abort it.

The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile oneshots (swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard → deploy-reports, deploy-backupbot, warm-keycloak). Verify:

systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
tailscale status | head -3
docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
systemctl list-timers 'cc-ci-*' nightly-sweep
sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)

When it is healthy: sudo nixos-rebuild switch --flake .#cc-ci (same config, now also the boot default). If you are migrating from another host, do §6 before letting it serve anything: right after the first activation scale the !testme bridge to 0 and mask the two orchestrator timers so the new box does not process PR comments or start a second weekly run while the old host is live:

docker service scale ccci-bridge_app=0
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer

6. Migrating: restore state from the previous host

Over tailscale (rsync -aHAX --numeric-ids root@<old>:<path> <path>), with the matching service stopped on the new host while its directory is copied:

path	holds	notes
/var/lib/cc-ci-reports	the published weekly report pages (report.ci…)
/var/lib/cc-ci-runs	per-run artifacts the dashboard shows
/var/lib/ci-warm	warm-canonical state + alerts	recipe warm volumes are caches: not copied, rebuilt by the Sunday sweep / first use
/var/lib/acme	the Let's Encrypt cert + account and acme-dns-accounts.json — the account the permanent _acme-challenge CNAME points at	without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in acme-dns.nix)
/var/lib/acme-dns	the acme-dns zone DB
/var/lib/ci-certs	the copy traefik is handed	then systemctl restart cc-ci-acme-traefik-handoff
/root/.abra	abra's per-app env files for the deployed stacks
Drone data volume /var/lib/docker/volumes/drone_ci_commoninternet_net_data	Drone's DB: the Gitea OAuth grant, repo activation, build history	docker service scale drone_ci_commoninternet_net_app=0 on the new host, copy, scale back to 1. Otherwise run scripts/bootstrap-drone-oauth.sh (cc-ci repo) with the bot password and re-activate repos
/srv/cc-ci-orch/.cc-ci-logs, /srv/cc-ci-orch/cc-ci-plan/upstream/, REBOOTS.md, JOURNAL.md	orchestrator history, the upgrader's per-recipe release-note registry	as loops; do the final sync after stopping the orchestrator on the old host

7. Cutover and verification

1. DNS (operator, Gandi zone commoninternet.net): A records ci, *.ci and ns-acme → the new IPv4. acme NS ns-acme and _acme-challenge.ci CNAME <account>.acme… stay as they are. Wait for propagation (dig +short ci.commoninternet.net).
2. Check the new host answers on the new IP before DNS moves: dig @<new-ip> ns-acme.commoninternet.net (acme-dns), curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/ (dashboard, valid cert), same for report.ci and drone.ci.
3. Old host: docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0; systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer on the old orchestrator. New host: docker service scale ccci-bridge_app=1; systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start both.
4. End to end: post !testme on an open recipe PR and watch it turn green on the new Drone; open https://ci.commoninternet.net and https://report.ci.commoninternet.net.
5. The orchestrator: as loops on the new host cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator (or just systemctl restart cc-ci-orchestrator), attach with claude --resume or from claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.

8. Day 2

• Update the host (nixpkgs bump for both halves): /cc-ci-orchestrator-update, which is nix flake update → nixos-rebuild test → verify → switch → PR. The cc-ci input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
• Update only cc-ci's code (harness/tests/modules): merge in the cc-ci repo, then nix flake update cc-ci here and rebuild; also git -C /etc/cc-ci pull --recurse-submodules so the deployed checkout the sweep runs from matches.
• Something is down: systemctl --failed, journalctl -u deploy-<x>, docker service ps <svc>; the cc-ci repo's docs/runbook.md. Host unreachable: Hetzner console → reboot lands on the last switched generation; rescue mode + nixos-enter for anything worse (skill /hetzner-server-recovery).

---

Operating the orchestrator session

The steering session is a long-lived interactive Claude Code session under tmux with --remote-control, so it can be watched and steered from claude.ai/code (or the mobile app). cc-ci-orchestrator.service relaunches it on boot via cc-ci-plan/agents.py up orchestrator (backend + model in cc-ci-plan/agents.toml).

# attach on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator
# or resume the conversation by hand in a fresh tmux pane
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# already inside a live session and just want the web surface? /remote-control

--resume <name|id> selects the conversation to restore; the --remote-control "<name>" value is only the web display label. Don't pass --fork-session unless you mean to branch.

Kick off / supervise the loops

cd /srv/cc-ci/cc-ci-plan
./launch.sh start                       # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
./launch.sh status                      # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop

Full supervision guide, credential map and history are in cc-ci-plan/kickoff.md, cc-ci-plan/plan.md §1.5 and cc-ci-plan/JOURNAL.md.