cc-ci-orchestrator/README.md

# cc-ci-orchestrator

The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.

| | where |
|---|---|
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |

Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
commit them.

---

# Deploying a cc-ci host from scratch

This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
can repeat it. Read it once before starting; the order matters.

## 0. What you need in hand

- A **Hetzner Cloud** project you can create servers in (console login or an API token).
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
  read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
  `autonomic-bot-gitea-ed25519`, has it).
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
  you are starting fresh you create them (each row says how).
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.

## 1. Provision the server on Hetzner (Debian image)

In the Hetzner Cloud console (or with `hcloud server create`):

| setting | value | why |
|---|---|---|
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
| Name | `cc-ci` | becomes the hostname |
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |

Check you can log in: `ssh root@<ip> hostname`.

## 2. Convert Debian → NixOS with nixos-infect

[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
reboots. Run it detached so the SSH session dropping does not kill it:

```bash
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
export NIXOS_IMPORT=""             # the real config comes from the flake in §5
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
export NO_SWAP=true
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
EOF
chmod +x /root/infect.sh
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
```

It downloads Nix, builds a NixOS system (5–10 min; follow with
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.

> If the box does not come back within ~5 minutes, open the Hetzner console (or rescue mode) and
> look at the boot messages — the two known failure modes are a broken bootloader install and a
> wrong `networking.nix` (see §3, the `defaultGateway.interface` note).

## 3. Capture the machine-specific config into this repo

nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
first two matter; the flake replaces `configuration.nix`.

```bash
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix
```

Then in `nix/hosts/cc-ci/`:

- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
  `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` — since NixOS 25.05
  omitting the interface leaves the host without a default route after a rebuild (this took the
  old CI server offline once; rescue mode fixed it). If the generated IPv6 block has an empty
  address, delete the IPv6 parts.
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
  is the release you installed (never change it later).
- `ssh-keys`: the root keys.

Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.

## 4. Stage the workspace and secrets on the new host

Everything in this section is **outside git**. Do it as root over SSH, in this order.

### 4a. Tailscale

```bash
# a reusable (or fresh) tailnet auth key from the tailscale admin console
install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
```

### 4b. The CI server's checkout and its one out-of-band secret

```bash
# root's deploy key for the private cc-ci-secrets submodule
install -d -m700 /root/.ssh
install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
cat > /root/.ssh/config <<'EOF'
Host git.autonomic.zone
    Port 2222
    User git
    IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
    IdentitiesOnly yes
EOF
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
# the master (recovery) age key — the only sops recipient a fresh host can be
install -d -m700 /var/lib/sops-nix
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
```

`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.

### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
run §5 once first if this is a fresh host, then come back)

```bash
sudo -iu loops
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
cd /srv/cc-ci-orch
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
mkdir -p .cc-ci-logs .sops
```

Then the files below (`install -m600 -o loops -g users`):

| file | what | source |
|---|---|---|
| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them** | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |

`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.

## 5. Build and activate

From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):

```bash
sudo -iu loops
cd /srv/cc-ci-orch && git checkout <branch>
sudo nixos-rebuild test --flake .#cc-ci     # activates WITHOUT touching the bootloader
```

`test` first, always: if the activation breaks networking or sshd, a reboot from the Hetzner
console lands on the last known-good generation. Over an SSH session that itself rides tailscale,
run it detached (`systemd-run --unit=ccci-sw --collect nixos-rebuild test --flake .#cc-ci`) so the
tailscale restart during activation does not abort it.

The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:

```bash
systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
tailscale status | head -3
docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
systemctl list-timers 'cc-ci-*' nightly-sweep
sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
```

When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
the new box does not process PR comments or start a second weekly run while the old host is live:

```bash
docker service scale ccci-bridge_app=0
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
```

## 6. Migrating: restore state from the previous host

Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
stopped on the new host while its directory is copied:

| path | holds | notes |
|---|---|---|
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
| `/var/lib/acme-dns` | the acme-dns zone DB | |
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |

## 7. Cutover and verification

1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
   the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
   are. Wait for propagation (`dig +short ci.commoninternet.net`).
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
   (acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
   (dashboard, valid cert), same for `report.ci` and `drone.ci`.
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
   `systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
   orchestrator. New host: `docker service scale ccci-bridge_app=1`;
   `systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
   open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
   (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
   claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.

## 8. Day 2

- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
  `nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
  this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
  `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
  so the deployed checkout the sweep runs from matches.
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
  the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
  `switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
  `/hetzner-server-recovery`).

---

# Operating the orchestrator session

The steering session is a long-lived interactive Claude Code session under tmux with
`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
(backend + model in `cc-ci-plan/agents.toml`).

```bash
# attach on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator
# or resume the conversation by hand in a fresh tmux pane
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# already inside a live session and just want the web surface? /remote-control
```

`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
only the web display label. Don't pass `--fork-session` unless you mean to branch.

## Kick off / supervise the loops

```bash
cd /srv/cc-ci/cc-ci-plan
./launch.sh start                       # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
./launch.sh status                      # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop
```

Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.