cc-ci-orchestrator/cc-ci-plan/plan-cc-ci-combined-host.md

Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator

Status: IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the cc-ci orchestrator onto one new Hetzner server (195.201.88.249, 8 GB, 150 GB, Debian 13 image), cleanly split off from the shared notplants-orchestrator box, which keeps everything else (lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in recipe-maintainers/cc-ci, the orchestrator's in recipe-maintainers/cc-ci-orchestrator, and the orchestrator flake includes cc-ci's module so one nixos-rebuild produces the combined host. Last step (separate, after everything works on the current names): move both to autonomic.zone subdomains.

Facts (2026-09-07)

	old cc-ci server	old orchestrator host (stays, becomes notplants-only)	new combined host
public IP	91.98.47.73 (fsn1, Hetzner 134485294)	168.119.126.100 (nbg1, Hetzner 134487234)	195.201.88.249
tailnet	cc-ci 100.95.31.88	cc-ci-orchestrator-1 100.84.190.30	cc-ci (new node)
RAM / disk	8 GB / 150 GB (83 GB used, 59 GB docker)	4 GB + 4 GB swap / 75 GB + 250 GB /mnt/data	8 GB / 150 GB, one disk
built by	cc-ci flake #cc-ci (nixpkgs 26.05 rev 531670d)	notplants-nix flake #notplants-orchestrator (26.05 channel), importing cc-ci-orchestrator's nixosModules.cc-ci	cc-ci-orchestrator flake #cc-ci importing cc-ci's nixosModules.cc-ci-server
DNS	ci., *.ci., ns-acme.commoninternet.net → 91.98.47.73 (Gandi, direct, no gateway)	oc.commoninternet.net → 100.84.190.30	operator repoints at cutover

Data on the old cc-ci server that must move: /var/lib/cc-ci-reports (published reports), /var/lib/cc-ci-runs (dashboard artifacts, 1.7 G), /var/lib/ci-warm (1.4 G), /var/lib/acme (LE cert valid to 2026-11-29 + acme-dns account json that the _acme-challenge CNAME points at), /var/lib/acme-dns (the authoritative zone DB), /var/lib/ci-certs, /root/.abra (app env files), /etc/cc-ci (deployed checkout the Sunday sweep runs from), Drone's drone_ci_commoninternet_net_data volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile oneshots recreate them from sops.

Out-of-band secrets the new host needs (never in git): /var/lib/sops-nix/key.txt (= the master age key, /srv/cc-ci/.sops/master-age.txt here — the new host's SSH host key is not a sops recipient), /etc/ts-auth-key, /srv/cc-ci/.testenv, /srv/cc-ci/upgrader.env, /srv/cc-ci/.sops/master-age.txt, ~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519, /etc/nginx/oc-* (self-signed cert + htpasswd for the opencode UI), claude/opencode/codex auth under ~loops.

Design

cc-ci repo (feat/nixos-module-export):

• nixosModules.cc-ci-server = nix/modules/default.nix: imports all service modules + the host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443, environment.systemPackages = ccciRuntimeTools, allowUnfree). No hardware, no networking, no tailscale, no root keys, no stateVersion — the host supplies those.
• New options under cc-ci.*: publicIPv4 (acme-dns listen + the ns-acme A record), sopsFile (absolute path to the decrypted-at-activation secrets.yaml, default the submodule path so #cc-ci keeps working), repoPath (/etc/cc-ci, used by nightly-sweep).
• nixosConfigurations.cc-ci (old host) keeps building unchanged via the same module.

cc-ci-orchestrator repo (feat/combined-cc-ci-host):

• flake input cc-ci (https, public) with nixpkgs/sops-nix follows so one nixpkgs + one sops-nix.
• nixosModules.cc-ci-orchestrator (the existing nix/modules/cc-ci.nix, kept exported as nixosModules.cc-ci too so notplants-nix keeps evaluating until it drops the input) — the loops, orchestrator session and the weekly/hourly timers.
• nix/modules/orchestrator-host.nix: the host contract the module assumes — loops user + sudo, nix-ld, claude/opencode/codex installers, opencode-web, the tailnet-only nginx oc. vhost (on the tailscale IP, port 8443, because traefik owns 80/443), tool packages, PATH.
• nixosConfigurations.cc-ci = nix/hosts/cc-ci/{configuration,hardware,networking}.nix importing both modules. /srv is a plain directory (no /mnt/data), 8 GB swapfile, root keys, tailscale --hostname=cc-ci, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
• loops' ssh config Host cc-ci → 127.0.0.1 so every ssh cc-ci … in skills/scripts keeps working.
• archive/: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
• README.md: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one nixos-rebuild, data restore, cutover, verification — written so a person or an LLM can redo it.

notplants-nix (chore/drop-cc-ci, after cutover): remove the cc-ci input, module import, the four cc-ci units' mount gating, loopsSshConfig, opencode-web + the oc. vhost (unless something notplants-side uses it), tailscale hostname → notplants-orchestrator.

Steps

1. nixos-infect the new box (NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner); capture hardware-configuration.nix + networking.nix.
2. cc-ci: module export + options; verify #cc-ci still evaluates; PR.
3. cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify #cc-ci evaluates; PR.
4. Stage secrets + clones on the new host; nixos-rebuild test → verify → switch. Immediately after: scale the new ccci-bridge_app to 0 and mask the two cc-ci timers so the new host does not double-process !testme or run a second weekly upgrade before cutover.
5. Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs, /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
6. Pre-cutover verification on the new IP (curl --resolve, port 53, dashboard, reports, drone, one direct cc-ci-run on custom-html-tiny).
7. Operator: Gandi A records ci, *.ci, ns-acme → 195.201.88.249. Then: old bridge + drone + timers off, new bridge up, one real !testme end-to-end, a !testme-driven report page.
8. [~] Move the orchestrator: stop cc-ci units here, final rsync of /srv/cc-ci-orch + agent state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
9. Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
10. Domain move to autonomic.zone — separate plan, after 1–9 are proven.

Log

• 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with notplants-orchestrator-ed25519.
• 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05, PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so nixos-infect's temp swapfile fails swapon: Invalid argument; fixed with NO_SWAP=true. Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the Hetzner console (no API token for that project on this host).
• 2026-09-07 20:40 UTC — cc-ci branch feat/nixos-module-export (9b99f81) pushed: the standalone #cc-ci drv is byte-identical before/after. Orchestrator branch feat/combined-cc-ci-host: #cc-ci evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL hardware/networking copied from the old CI server — to be replaced by the infect output.
• 2026-09-07 20:30 UTC — rescue-mode diagnosis + fixes (README §2a); #cc-ci test → switch → reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator #19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped). Waiting on operator: new tailscale auth key; Gandi DNS cutover.
• 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic auth on 443 (traefik labels on the opencode-ui socat relay → nginx on docker_gwbridge:8090); fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme) → 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge + drone scaled to 0, old-host timers masked, new bridge up; !testme on custom-html #7 = Drone build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped. nixos-rebuild switch = generation 4. The switch re-enabled cc-ci-orchestrator on the new box: a fresh Claude orchestrator session runs there (tmux cc-ci-orchestrator, remote-control label cc-ci-orchestrator) since 20:46 UTC; the old host's cc-ci units are runtime-masked and the notplants-nix branch chore/drop-cc-ci (pushed, evaluates) removes them for good — operator merges (that repo auto-deploys nightly).
• Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during the resolver mess).