# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator

**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
Last step (separate, after everything works on the current names): move both to `autonomic.zone`
subdomains.

## Facts (2026-09-07)

| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
|---|---|---|---|
| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |

Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
oneshots recreate them from sops.

Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.

## Design

**cc-ci repo** (`feat/nixos-module-export`):
- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
  host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
  `environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
  tailscale, no root keys, no stateVersion — the host supplies those.
- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
  `sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
  path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.

**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
  `nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
  orchestrator session and the weekly/hourly timers.
- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
  nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
  (on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
  both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
  `--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
  `nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.

**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
notplants-side uses it), tailscale hostname → `notplants-orchestrator`.

## Steps

1. [ ] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
   `hardware-configuration.nix` + `networking.nix`.
2. [ ] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
3. [ ] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
   `#cc-ci` evaluates; PR.
4. [ ] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
   Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
   new host does not double-process `!testme` or run a second weekly upgrade before cutover.
5. [ ] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
   /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
6. [ ] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
   drone, one direct `cc-ci-run` on custom-html-tiny).
7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
   drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
   state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.

## Log

- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
  `notplants-orchestrator-ed25519`.
- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
  PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
  nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
  Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
  Hetzner console (no API token for that project on this host).
- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
  `#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
  `#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
  hardware/networking copied from the old CI server — to be replaced by the infect output.