cc-ci-orchestrator/cc-ci-plan/plan-cc-ci-hetzner-migration.md

Plan — migrate the cc-ci SERVER from b1 Incus to Hetzner (full cutover)

Status: PROPOSED. Move the cc-ci CI server (cc-nix-test) off the slow b1 host onto a fast Hetzner cpx32 (8 GB, dedicated vCPU, NVMe), repoint the Builder/Adversary loops + everything at it, then stop the old VM. This file: /srv/cc-ci/cc-ci-plan/plan-cc-ci-hetzner-migration.md. Owner: assistant (provisioning + cutover mechanics) + orchestrator (coordination); operator for the secret/DNS gates. Supersedes the narrower plan-cc-ci-hetzner-terraform.md (that is Phase 1's deliverable; this plan wraps it with the cutover + decommission).

---

0. Context (why, and what's where)

• Two VMs run on b1 (a 2015 Intel i5-6400T low-power CPU + a spinning HDD — measured: CPU pressure ~55%, root disk ROTA=1):
  • cc-ci server cc-nix-test (tailnet 100.90.116.4, 8 GB) — where the loops deploy recipes + run the harness (the heavy CI work). This is what we migrate.
  • orchestrator VM cc-ci-orchestrator (tailnet 100.116.55.106, 2 GB) — where the loops + orchestrator + assistant run (claude sessions). Stays for now.
• b1 is overloaded running both on a slow CPU + HDD — "everything is getting slow."
• The win (see the perf analysis): Hetzner cpx32 = modern dedicated vCPU + NVMe vs a 2015 low-power CPU + HDD → I/O-bound deploys (the ghost/discourse near-timeouts) likely 3–10× faster, CPU work ~2–3×. Moving the heavy server off b1 also relieves b1, so the orchestrator VM (still there) speeds up too.

1. Phase 1 — provision the Hetzner cc-ci, FULLY ready

The plan-cc-ci-hetzner-terraform.md deliverable, taken all the way to a converged, green server (not just "terraform applies"):

• terraform/ in the cc-ci repo (cpx32, ubuntu-24.04, pinned hcloud provider + nixos-infect). apply → nixos-infect → bare NixOS on Hetzner.
• Add the cc-ci-hetzner flake host (nixos-infect's DO/Hetzner hardware + the shared nix/modules/*).
• Full convergence (the D8 flow): clone cc-ci --recursive + place the bootstrap age key at /var/lib/sops-nix/key.txt (operator) + nixos-rebuild switch --flake .#cc-ci-hetzner → traefik / drone / bridge / dashboard / backupbot / swarm all up, 0 failed units.
• DNS/cert: point ci.commoninternet.net + *.ci A record at the Hetzner public IP (the server has one — can drop the b1 TLS-passthrough gateway). Keep the sops wildcard cert for v1 (or ACME — §decision).
• Readiness gate (before any cutover): ssh works; the dashboard + *.ci.commoninternet.net are reachable; a full !testme runs GREEN on the Hetzner server (drive one recipe end-to-end via the harness). Keep the b1 cc-ci running untouched in parallel during all of Phase 1.
• Operator inputs for Phase 1: HCLOUD_TOKEN (have), TS_AUTH_KEY (have), the bootstrap age key (needed for convergence), and the DNS change. Note: the token may be invalidated after the KEEPER server is applied — the server runs without it; only future terraform needs a (new) token.

2. Phase 2 — cut everything over to the Hetzner server

Once Phase 1 is green, switch all consumers from the b1 cc-nix-test to the Hetzner server:

• Loop access: update the Host cc-ci entry in the loops' ssh config (on the orchestrator VM, used by builder/adversary/orchestrator/assistant) — HostName from 100.90.116.4 → the Hetzner server's tailnet IP / MagicDNS. (ssh cc-ci is the single indirection the loops use, so this one change repoints all of them. The Hetzner box joins the SAME tailnet via TS_AUTH_KEY, so it's a direct peer like today.)
• CI flow: the !testme → bridge → Drone → harness path + the dashboard now run on the Hetzner server (they're part of the converged config there). The recipe mirrors stay on Gitea (unaffected).
• State carry-over (minimal — mostly stateless): recipes redeploy from the mirrors; warm canonicals re-seed on the first green cold runs; the harness lives in the cc-ci repo. Drone build history + dashboard state start fresh on the new server (acceptable; migrate only if wanted).
• Verify cutover: a full loop cycle works against Hetzner — Builder deploys + claims a gate, the Adversary cold-verifies green on the Hetzner server; phase-2 recipe work continues, now fast. Watch a ghost/discourse deploy to confirm the timeouts are gone.

3. Phase 3 — stop the old cc-ci VM (free b1)

• Once everything is confirmed serving green on Hetzner, stop cc-nix-test on b1 (Incus PUT .../state {"action":"stop"}). Keep it as a cold standby for a few days (don't delete) for rollback, then retire.
• b1 now runs only the small orchestrator VM → it gets b1's full (modest) resources → the loops' runtime is less starved too. "Everything faster from here on out."
• Rollback (until the old VM is deleted): if Hetzner has a problem, revert the Host cc-ci ssh entry to 100.90.116.4 and start the b1 VM again.

4. Sequencing & gates (don't break the running CI)

• Strictly parallel bring-up: Phase 1 stands Hetzner up alongside the live b1 cc-ci; no consumer is repointed until the Hetzner !testme is green (Phase 1 readiness gate).
• The cutover (Phase 2) is a single ssh-config repoint + DNS — fast and reversible.
• Phase 3 (stop b1) only after Phase 2 is verified.
• The loops keep working on b1 throughout Phase 1 (no disruption); the brief cutover window is the only moment they switch servers.

5. Open decisions (log in DECISIONS.md)

• DNS/cert: point *.ci at the Hetzner public IP + drop the gateway; sops cert (v1) vs ACME.
• Drone/dashboard history: fresh on Hetzner (default) vs migrate the volumes.
• Orchestrator VM: leave on b1 (freed) for now; a later, separate plan could also move the loops' runtime to Hetzner and fully retire b1 — out of scope here (the runtime needn't be fast).
• Token lifecycle: invalidate HCLOUD_TOKEN after the keeper apply, or keep a (rotated) one for ongoing terraform management of the server.

6. Definition of Done

• Hetzner cpx32 cc-ci fully converged (0 failed units) + a green !testme on it.
• Loops + dashboard + *.ci.commoninternet.net all served from Hetzner; a full Builder→Adversary cycle verified green there; deploy/convergence visibly faster (ghost/discourse no longer near-timeout).
• Old b1 cc-nix-test stopped (cold standby, not deleted).
• terraform/ committed to the cc-ci repo (via PR); no secrets/state in git; docs/install.md updated for the Hetzner host. Adversary-verifiable: from-scratch reproducibility holds on Hetzner.

7. Guardrails

• Parallel bring-up; never repoint consumers until Hetzner is green; keep b1 as cold standby.
• No secrets in git (token, TS key, age key, tfstate). Pin everything. x86 only (cpx32/cx32).
• Real Nix provisioning (the flake) + real abra; don't weaken anything to make the new server "pass."