notplantsandClaude Fable 5.1
a895bd3e3e
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
107 lines
8.0 KiB
Markdown
107 lines
8.0 KiB
Markdown
# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator
|
||
|
||
**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
|
||
cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
|
||
cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
|
||
(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
|
||
`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
|
||
orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
|
||
Last step (separate, after everything works on the current names): move both to `autonomic.zone`
|
||
subdomains.
|
||
|
||
## Facts (2026-09-07)
|
||
|
||
| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
|
||
|---|---|---|---|
|
||
| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
|
||
| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
|
||
| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
|
||
| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
|
||
| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |
|
||
|
||
Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
|
||
`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
|
||
(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
|
||
`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
|
||
`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
|
||
volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
|
||
rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
|
||
oneshots recreate them from sops.
|
||
|
||
Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
|
||
key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
|
||
`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
|
||
`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
|
||
htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.
|
||
|
||
## Design
|
||
|
||
**cc-ci repo** (`feat/nixos-module-export`):
|
||
- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
|
||
host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
|
||
`environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
|
||
tailscale, no root keys, no stateVersion — the host supplies those.
|
||
- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
|
||
`sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
|
||
path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
|
||
- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.
|
||
|
||
**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
|
||
- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
|
||
- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
|
||
`nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
|
||
orchestrator session and the weekly/hourly timers.
|
||
- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
|
||
nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
|
||
(on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
|
||
- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
|
||
both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
|
||
`--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
|
||
- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
|
||
- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
|
||
- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
|
||
`nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.
|
||
|
||
**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
|
||
four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
|
||
notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
|
||
|
||
## Steps
|
||
|
||
1. [x] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
|
||
`hardware-configuration.nix` + `networking.nix`.
|
||
2. [x] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
|
||
3. [x] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
|
||
`#cc-ci` evaluates; PR.
|
||
4. [x] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
|
||
Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
|
||
new host does not double-process `!testme` or run a second weekly upgrade before cutover.
|
||
5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
|
||
/root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
|
||
6. [~] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
|
||
drone, one direct `cc-ci-run` on custom-html-tiny).
|
||
7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
|
||
drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
|
||
8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
|
||
state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
|
||
9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
|
||
10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
|
||
|
||
## Log
|
||
|
||
- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
|
||
`notplants-orchestrator-ed25519`.
|
||
- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
|
||
PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
|
||
nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
|
||
Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
|
||
Hetzner console (no API token for that project on this host).
|
||
- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
|
||
`#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
|
||
`#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
|
||
hardware/networking copied from the old CI server — to be replaced by the infect output.
|
||
- 2026-09-07 20:30 UTC — rescue-mode diagnosis + fixes (README §2a); `#cc-ci` test → switch →
|
||
reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator
|
||
#19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped).
|
||
Waiting on operator: new tailscale auth key; Gandi DNS cutover.
|