recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

nix: one Hetzner host for the CI server + the orchestrator (#cc-ci)

Browse Source 
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).

nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.

README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 19:58:33 +00:00
co-authored by Claude Fable 5.1
parent 5bebd13e49
commit 31af820079
32 changed files with 863 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cc-ci-plan/plan-cc-ci-combined-host.md
+102
View File
@@ -0,0 +1,102 @@
# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator
**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
Last step (separate, after everything works on the current names): move both to `autonomic.zone`
subdomains.
## Facts (2026-09-07)
| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
|---|---|---|---|
| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |
Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
oneshots recreate them from sops.
Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.
## Design
**cc-ci repo** (`feat/nixos-module-export`):
- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
`environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
tailscale, no root keys, no stateVersion — the host supplies those.
- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
`sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.
**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
`nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
orchestrator session and the weekly/hourly timers.
- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
(on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
`--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
`nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.
**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
## Steps
1. [ ] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
`hardware-configuration.nix` + `networking.nix`.
2. [ ] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
3. [ ] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
`#cc-ci` evaluates; PR.
4. [ ] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
new host does not double-process `!testme` or run a second weekly upgrade before cutover.
5. [ ] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
/root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
6. [ ] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
drone, one direct `cc-ci-run` on custom-html-tiny).
7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
## Log
- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
`notplants-orchestrator-ed25519`.
- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
Hetzner console (no API token for that project on this host).
- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
`#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
`#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
hardware/networking copied from the old CI server — to be replaced by the infect output.
cc-ci-plan/plan-cc-ci-hetzner-migration.md
-96
View File
@@ -1,96 +0,0 @@
# Plan — migrate the cc-ci SERVER from b1 Incus to Hetzner (full cutover)
**Status:** PROPOSED. Move the cc-ci **CI server** (`cc-nix-test`) off the slow b1 host onto a fast
Hetzner **cpx32** (8 GB, dedicated vCPU, NVMe), repoint the Builder/Adversary loops + everything at it,
then stop the old VM. **This file:** `/srv/cc-ci/cc-ci-plan/plan-cc-ci-hetzner-migration.md`.
**Owner:** assistant (provisioning + cutover mechanics) + orchestrator (coordination); operator for the
secret/DNS gates. **Supersedes** the narrower `plan-cc-ci-hetzner-terraform.md` (that is Phase 1's
deliverable; this plan wraps it with the cutover + decommission).
---
## 0. Context (why, and what's where)
- **Two VMs run on b1** (a 2015 **Intel i5-6400T low-power CPU + a spinning HDD** — measured: CPU
pressure ~55%, root disk `ROTA=1`):
- **cc-ci server** `cc-nix-test` (tailnet `100.90.116.4`, 8 GB) — where the loops deploy recipes +
run the harness (the heavy CI work). **This is what we migrate.**
- **orchestrator VM** `cc-ci-orchestrator` (tailnet `100.116.55.106`, 2 GB) — where the loops +
orchestrator + assistant *run* (claude sessions). Stays for now.
- b1 is overloaded running both on a slow CPU + HDD — "everything is getting slow."
- **The win (see the perf analysis):** Hetzner cpx32 = modern dedicated vCPU + **NVMe** vs a 2015
low-power CPU + **HDD** → I/O-bound deploys (the ghost/discourse near-timeouts) likely **3–10×**
faster, CPU work **~2–3×**. Moving the *heavy* server off b1 also relieves b1, so the orchestrator VM
(still there) speeds up too.
## 1. Phase 1 — provision the Hetzner cc-ci, FULLY ready
The `plan-cc-ci-hetzner-terraform.md` deliverable, taken all the way to a **converged, green** server
(not just "terraform applies"):
- `terraform/` in the cc-ci repo (cpx32, ubuntu-24.04, pinned hcloud provider + nixos-infect). `apply`
→ nixos-infect → bare NixOS on Hetzner.
- Add the `cc-ci-hetzner` flake host (nixos-infect's DO/Hetzner hardware + the shared `nix/modules/*`).
- **Full convergence (the D8 flow):** clone cc-ci `--recursive` + place the **bootstrap age key** at
`/var/lib/sops-nix/key.txt` (operator) + `nixos-rebuild switch --flake .#cc-ci-hetzner` → traefik /
drone / bridge / dashboard / backupbot / swarm all up, **0 failed units**.
- **DNS/cert:** point `ci.commoninternet.net` + `*.ci` **A record at the Hetzner public IP** (the
server has one — can drop the b1 TLS-passthrough gateway). Keep the sops wildcard cert for v1
(or ACME — §decision).
- **Readiness gate (before any cutover):** ssh works; the dashboard + `*.ci.commoninternet.net` are
reachable; a **full `!testme` runs GREEN on the Hetzner server** (drive one recipe end-to-end via
the harness). Keep the b1 cc-ci running untouched in parallel during all of Phase 1.
- **Operator inputs for Phase 1:** `HCLOUD_TOKEN` (have), `TS_AUTH_KEY` (have), the **bootstrap age
key** (needed for convergence), and the **DNS change**. Note: the token may be invalidated after the
KEEPER server is applied — the server runs without it; only future `terraform` needs a (new) token.
## 2. Phase 2 — cut everything over to the Hetzner server
Once Phase 1 is green, switch all consumers from the b1 `cc-nix-test` to the Hetzner server:
- **Loop access:** update the `Host cc-ci` entry in the loops' ssh config (on the orchestrator VM,
used by builder/adversary/orchestrator/assistant) — `HostName` from `100.90.116.4` →
the **Hetzner server's tailnet IP / MagicDNS**. (`ssh cc-ci` is the single indirection the loops
use, so this one change repoints all of them. The Hetzner box joins the SAME tailnet via
`TS_AUTH_KEY`, so it's a direct peer like today.)
- **CI flow:** the `!testme` → bridge → Drone → harness path + the dashboard now run on the Hetzner
server (they're part of the converged config there). The recipe mirrors stay on Gitea (unaffected).
- **State carry-over (minimal — mostly stateless):** recipes redeploy from the mirrors; **warm
canonicals re-seed** on the first green cold runs; the harness lives in the cc-ci repo. Drone build
history + dashboard state start **fresh** on the new server (acceptable; migrate only if wanted).
- **Verify cutover:** a full loop cycle works against Hetzner — Builder deploys + claims a gate, the
Adversary **cold-verifies green** on the Hetzner server; phase-2 recipe work continues, now fast.
Watch a ghost/discourse deploy to confirm the timeouts are gone.
## 3. Phase 3 — stop the old cc-ci VM (free b1)
- Once everything is confirmed serving green on Hetzner, **stop `cc-nix-test` on b1** (Incus
`PUT .../state {"action":"stop"}`). **Keep it as a cold standby for a few days** (don't delete) for
rollback, then retire.
- b1 now runs only the small orchestrator VM → it gets b1's full (modest) resources → the loops'
*runtime* is less starved too. "Everything faster from here on out."
- **Rollback (until the old VM is deleted):** if Hetzner has a problem, revert the `Host cc-ci` ssh
entry to `100.90.116.4` and start the b1 VM again.
## 4. Sequencing & gates (don't break the running CI)
- **Strictly parallel bring-up:** Phase 1 stands Hetzner up *alongside* the live b1 cc-ci; **no
consumer is repointed until the Hetzner `!testme` is green** (Phase 1 readiness gate).
- The cutover (Phase 2) is a **single ssh-config repoint** + DNS — fast and reversible.
- Phase 3 (stop b1) only after Phase 2 is verified.
- The loops keep working on b1 throughout Phase 1 (no disruption); the brief cutover window is the
only moment they switch servers.
## 5. Open decisions (log in DECISIONS.md)
- **DNS/cert:** point `*.ci` at the Hetzner public IP + drop the gateway; sops cert (v1) vs ACME.
- **Drone/dashboard history:** fresh on Hetzner (default) vs migrate the volumes.
- **Orchestrator VM:** leave on b1 (freed) for now; a *later, separate* plan could also move the loops'
runtime to Hetzner and fully retire b1 — out of scope here (the runtime needn't be fast).
- **Token lifecycle:** invalidate `HCLOUD_TOKEN` after the keeper apply, or keep a (rotated) one for
ongoing `terraform` management of the server.
## 6. Definition of Done
- Hetzner cpx32 cc-ci fully converged (0 failed units) + a **green `!testme`** on it.
- Loops + dashboard + `*.ci.commoninternet.net` all served from Hetzner; a full Builder→Adversary
cycle verified green there; deploy/convergence visibly faster (ghost/discourse no longer near-timeout).
- Old b1 `cc-nix-test` **stopped** (cold standby, not deleted).
- `terraform/` committed to the cc-ci repo (via PR); no secrets/state in git; `docs/install.md`
updated for the Hetzner host. Adversary-verifiable: from-scratch reproducibility holds on Hetzner.
## 7. Guardrails
- Parallel bring-up; never repoint consumers until Hetzner is green; keep b1 as cold standby.
- No secrets in git (token, TS key, age key, tfstate). Pin everything. x86 only (cpx32/cx32).
- Real Nix provisioning (the flake) + real abra; don't weaken anything to make the new server "pass."
cc-ci-plan/plan-cc-ci-hetzner-terraform.md
-133
View File
@@ -1,133 +0,0 @@
# Plan — cc-ci on Hetzner Cloud: `terraform/` + nixos-infect + Nix provisioning
**Status:** PROPOSED → handed to the assistant to implement. Add a **`terraform/`** folder to the
**cc-ci product repo** (`recipe-maintainers/cc-ci`) that provisions the cc-ci server on **Hetzner
Cloud** (8 GB server), converts it to NixOS via **nixos-infect**, then applies the existing cc-ci
flake config — making the CI server reproducible-from-scratch on real cloud hosting.
**This file:** `/srv/cc-ci/cc-ci-plan/plan-cc-ci-hetzner-terraform.md`.
**Token (operator, 2026-05-31):** an `HCLOUD_TOKEN` with **read/write to an isolated Hetzner project**
(just for this) is in `/srv/cc-ci/.testenv`. The operator **will invalidate it** once the terraform is
verified working — so the goal is **write + apply + verify the working terraform**, then report.
---
## 0. Why
cc-ci currently runs as the Incus VM `cc-nix-test` on b1 — a small, shared 4-core host (the contention
we kept hitting). A dedicated Hetzner **8 GB** server gives standard, reliable hosting with a **public
IP**, fully reproducible via Terraform + the existing cc-ci NixOS flake. "Spin up cc-ci from nothing"
becomes a `terraform apply`.
## 1. What already exists — build ON this, don't reinvent
- cc-ci is a **flake-based NixOS system**: `flake.nix` → `nixosConfigurations.cc-ci` (pinned nixpkgs
24.11, **`system = "x86_64-linux"`**) → `nix/hosts/cc-ci/{configuration.nix, hardware.nix}` +
`nix/modules/*` (proxy/traefik, drone, drone-runner, bridge, dashboard, backupbot, swarm, abra,
harness, warm-keycloak, secrets).
- **From-scratch install is already VERIFIED (D8, `docs/install.md`):** a blank NixOS host + the two
repos (cc-ci cloned `--recursive` so the `cc-ci-secrets` submodule at `secrets/` comes too) + the
**one bootstrap age key** at `/var/lib/sops-nix/key.txt` → a single `nixos-rebuild switch` converges
the whole server (0 failed units; serialized reconcile oneshots). The wildcard TLS cert + all secrets
are **sops-encrypted in `cc-ci-secrets`** (not out-of-band).
- So **"provision via Nix in the expected way" = that exact D8 flow:** clone `--recursive` + bootstrap
age key + `nixos-rebuild switch --flake .#<host>`.
- The current `nix/hosts/cc-ci/hardware.nix` is **Incus-VM-specific** — Hetzner needs its own
hardware/bootloader/networking, which **nixos-infect generates**.
## 2. `terraform/` layout (in `recipe-maintainers/cc-ci`)
```
terraform/
versions.tf # terraform + hetznercloud/hcloud provider, pinned
variables.tf # hcloud_token(sensitive), location, server_type, image, ssh_key, ts_auth_key(sensitive), hostname
main.tf # hcloud_ssh_key + hcloud_server + user_data
outputs.tf # server ipv4, id
user-data.sh # cloud-init stage-1: run nixos-infect (pinned)
README.md # apply instructions + operator inputs
.gitignore # *.tfstate*, *.auto.tfvars, .terraform/ (NEVER commit secrets/state)
```
- **Provider:** `hetznercloud/hcloud` (pinned in `versions.tf`). The token comes from
**`HCLOUD_TOKEN`** (env, read by the provider) or `TF_VAR_hcloud_token` — it's in `.testenv`; do
NOT hardcode/commit it.
- **Server:** `hcloud_server` — type **`cpx32`** (AMD **dedicated vCPU**, **8 GB RAM**, NVMe SSD) —
**DEFAULT** (operator 2026-05-31: `cpx31` is **retired**; `cpx32` is the current dedicated-vCPU 8 GB
type). Dedicated vCPU avoids noisy-neighbor variance for bursty CI. Must be **x86** (the flake is
`x86_64-linux`; do **NOT** use the `cax*` ARM types). `cx32` (Intel shared vCPU, 8 GB) is a cheaper
alt. Confirm exact specs from the hcloud API at apply time. `image = "ubuntu-24.04"` (nixos-infect-supported base), a `location`
(e.g. `nbg1`/`fsn1`/`hel1` EU or `ash`/`hil` US — pick one, make it a var), `ssh_keys=[hcloud_ssh_key.id]`,
`user_data=file("user-data.sh")`, `public_net { ipv4_enabled = true }`, a stable name + label.
- Keep the token + TS key **sensitive**; `terraform.tfstate` **gitignored** (can hold secrets) — mirrors
cc-ci's no-secrets-in-git rule.
## 3. Stage 1 — nixos-infect (base Ubuntu → NixOS)
`user-data.sh` on first boot:
```sh
#!/usr/bin/env bash
set -euo pipefail
export NIX_CHANNEL=nixos-24.11
export PROVIDER=hetzner # nixos-infect provider hint (Hetzner Cloud is supported)
curl -fsSL https://raw.githubusercontent.com/elitak/nixos-infect/<PINNED_SHA>/nixos-infect | bash -x
```
nixos-infect converts the server to NixOS in place, generates `/etc/nixos/{configuration.nix,
hardware-configuration.nix, networking.nix}` (Hetzner-correct bootloader + public-IP networking), and
reboots into NixOS. **Pin the nixos-infect revision** — don't `curl|bash` master blind. After this the
server is **bare NixOS on Hetzner**, ssh-able as root.
## 4. Stage 2 — provision via Nix (bare NixOS → converged cc-ci) — "the expected way"
1. **Capture Hetzner hardware into the flake.** Take the `hardware-configuration.nix` + `networking.nix`
nixos-infect generated and add them as a flake host. **Cleaner: a new host `nix/hosts/cc-ci-hetzner/`**
importing the shared `nix/modules/*` + the Hetzner hardware, with `nixosConfigurations.cc-ci-hetzner`
in `flake.nix` (keeps the Incus `cc-ci` host buildable during transition). Make Hetzner the canonical
`cc-ci` after cutover.
2. **Run the D8 install flow on the server:** clone `recipe-maintainers/cc-ci` `--recursive` (brings
`cc-ci-secrets`), provision the **bootstrap age key** at `/var/lib/sops-nix/key.txt`, then
`nixos-rebuild switch --flake .#cc-ci-hetzner`. The reconcile oneshots converge the swarm.
3. **Where stage 2 runs:** **v1 = documented step run after `terraform apply`** (Terraform provisions +
infects; the age-key placement + `nixos-rebuild` is the explicit step, like `docs/install.md`).
Automate later via a Terraform `remote-exec` provisioner once key-delivery is settled.
- **Note on secrets for verification:** full cc-ci convergence needs the bootstrap age key (decrypts
`cc-ci-secrets`). If that key isn't available to the implementer, verify as far as possible —
`terraform apply` → nixos-infect → bare NixOS → the flake **builds/evaluates** for the Hetzner host
(`nixos-rebuild build --flake .#cc-ci-hetzner`) — and flag the age-key step as operator-pending.
## 5. Operator inputs (class-A1 — provide at apply, NEVER commit)
- **`HCLOUD_TOKEN`** — already in `.testenv` (isolated project, read/write; operator will invalidate
after). The provider reads it from env.
- **SSH key** — register a public key as `hcloud_ssh_key`; hold the private half to ssh + run stage 2.
- **`TS_AUTH_KEY`** — tailnet join (cc-ci enables tailscale; the server joins the same tailnet so the
orchestrator/loops reach it as today, direct peer). Already in `.testenv`.
- **Bootstrap age key** → `/var/lib/sops-nix/key.txt` (decrypts `cc-ci-secrets` incl. the TLS cert).
The single out-of-band secret per `docs/install.md`.
## 6. DNS / gateway — a simplification the public IP enables (open decision)
Today `*.ci.commoninternet.net` reaches the Incus VM (no public IP) via an external nginx
TLS-passthrough gateway → MagicDNS. A Hetzner server has a **public IP**, so point
`ci.commoninternet.net` + the `*.ci` wildcard **A record straight at the server** and **drop the
gateway** — Traefik terminates TLS directly. The sops wildcard cert still works as-is; or switch
Traefik to **ACME** and retire the manual cert + renewal. **v1: keep the sops cert (no behavior
change); evaluate ACME-on-public-IP as a follow-up.** Record in DECISIONS.md.
## 7. Open decisions (log in DECISIONS.md)
- **Replace vs. parallel:** stand Hetzner up **in parallel**, verify a full `!testme` + the D-gates
green, then cut DNS over and **retire the Incus `cc-nix-test`**. Nothing stateful is lost — recipes
redeploy, warm canonicals re-seed on first green runs.
- **Flake host:** parallel `cc-ci-hetzner` host until cutover, then make Hetzner the canonical `cc-ci`.
- **Server type/location** (cx32 vs cpx31; region); **ACME vs sops cert** (§6); **stage-2 automation** (§4.3).
## 8. Definition of Done
- `terraform/` in the cc-ci repo; `terraform apply` (with `HCLOUD_TOKEN`) creates an **8 GB cx32**
Hetzner server and nixos-infect converts it to NixOS.
- The flake **builds for the Hetzner host** (`nixos-rebuild build --flake .#cc-ci-hetzner`); given the
bootstrap age key it **switches** to a fully converged cc-ci (the D8 flow) — 0 failed units.
- (Once secrets available) a real recipe `!testme` runs **green** on the Hetzner cc-ci; dashboard +
`*.ci.commoninternet.net` reachable via the chosen DNS path.
- `terraform/README.md` documents apply + operator inputs; **no secrets/state committed**.
- The terraform is proven **idempotent** (`terraform plan` clean after apply); test resources cleaned
up (`terraform destroy`) if this is a throwaway verification rather than the real cutover.
## 9. Guardrails
- **No secrets in git** (HCLOUD_TOKEN, TS key, age key, tfstate all out-of-band/gitignored) — cc-ci's rule.
- **Pin everything** (hcloud provider, nixos-infect rev; nixpkgs already pinned) — reproducible, no drift.
- **x86 only** — the flake is `x86_64-linux`; use `cx32`/`cpx31`, never `cax*` (ARM).
- **Don't break the running Incus cc-ci** until the Hetzner one is verified green (parallel + cutover).
- **Real Nix provisioning** (the flake), not hand-installed packages.
- **The token is invalidatable + isolated** — but still treat it as a live secret: never commit it,
never echo it into logs.
cc-ci-plan/plan-migrate-cc-ci-to-hetzner.md
-83
View File
@@ -1,83 +0,0 @@
# Plan — migrate the cc-ci SERVER from the Incus VM to Hetzner (provision → benchmark → cutover → retire)
**Status:** PROPOSED. Move the **cc-ci CI server** off the Incus VM `cc-nix-test` (b1: 2015 i5-6400T
+ **spinning HDD**, CPU-pressure ~55%, getting very slow) onto a **Hetzner `cpx32`** (4 vCPU / 8 GB /
160 GB **NVMe**, x86, ~€16.49/mo). Everything (Builder, Adversary, the !testme pipeline) then targets
the fast new server. **This file:** `/srv/cc-ci/cc-ci-plan/plan-migrate-cc-ci-to-hetzner.md`.
**Key enabler (verified 2026-05-31):** the bootstrap age key is **already on this VM** at
`/srv/cc-ci/.sops/master-age.txt` and the `cc-ci-secrets` submodule is populated — so the new server
can be **fully provisioned end-to-end with NO operator secret-blocker** (the D8 flow decrypts the TLS
cert + all secrets). The Pi is not needed.
**Architecture reminder:** the Builder/Adversary **loops run on this orchestrator VM** and reach the
CI server via `ssh cc-ci`; the **!testme pipeline (Gitea webhook → bridge → Drone → harness) runs ON
the cc-ci server**, and `*.ci.commoninternet.net` + the dashboard are served from it. "Switch
everything to the new server" = make the Hetzner box the cc-ci, then repoint `ssh cc-ci`, the
webhook/DNS, and the dashboard at it. The loops' code/clones don't move — only their target.
---
## Phase 1 — Provision the new Hetzner cc-ci, fully converged (assistant)
Per **`plan-cc-ci-hetzner-terraform.md`** (the provisioning detail): `terraform/` in the cc-ci repo →
`hcloud` `cpx32` from `ubuntu-24.04` → **pinned nixos-infect** → bare NixOS → add the **`cc-ci-hetzner`
flake host** (the nixos-infect-generated DO/Hetzner hardware + the shared `nix/modules/*`) → run the
**D8 flow**: clone `--recursive`, place `/srv/cc-ci/.sops/master-age.txt` at `/var/lib/sops-nix/key.txt`,
`nixos-rebuild switch --flake .#cc-ci-hetzner`. The server joins the tailnet (TS_AUTH_KEY).
- **Accept:** 0 failed units; traefik/drone/bridge/dashboard/backupbot up; the box is on the tailnet
and ssh-able; terraform is idempotent (`plan` clean). This is a **real** server we keep (not the
throwaway the terraform-plan first described) — do **not** `terraform destroy` once it converges.
- Done in **parallel** — the old Incus cc-ci keeps serving the loops until Phase 3.
## Phase 2 — Benchmark: old vs new, two recipes (a short report)
Pick **two representative recipes** — one light (e.g. `n8n` or `custom-html`) and one heavy/slow (e.g.
`ghost` or `discourse` — the HDD-bound timeout cases). Run the **same full harness** (cold,
install+upgrade+backup+restore+custom) on **both servers**:
- old: `ssh cc-ci-incus` (the current `cc-nix-test`), new: `ssh cc-ci-hetzner`.
- Capture **per-tier + total wall-clock** from the `RUN SUMMARY` for each recipe on each host.
Write a short comparison report → **`docs/perf/hetzner-vs-incus.md`** in the cc-ci repo (table: recipe
× tier × old-time × new-time × speedup). This empirically confirms the expected ~2–4× (more on the
I/O-bound phases). *(Run identical conditions — same recipe versions, cold cache both sides.)*
## Phase 3 — Cutover: point everything at the new server (orchestrated; pick a quiet moment)
1. **Quiesce briefly:** ensure no live `!testme`/deploy is mid-run on the old server.
2. **Repoint the loops' `ssh cc-ci`** → the Hetzner box's tailnet IP: update `Host cc-ci` in
`/home/loops/.ssh/config` (and root's) `HostName` → new IP. The loops keep working from this VM;
only their target changes. (Keep a `Host cc-ci-incus` alias for the old box during the overlap.)
3. **DNS / webhook / gateway:** point `ci.commoninternet.net` + the `*.ci` wildcard **A record at the
Hetzner public IP** (drop the TLS-passthrough gateway — Traefik on the droplet terminates directly;
the sops wildcard cert works as-is). Re-point the Gitea `issue_comment` webhook → the new server so
`!testme` triggers there. **DNS is operator-owned (`commoninternet.net`)** — the one operator step.
4. **Verify end-to-end on the new server:** a real PR `!testme` runs green through the new
bridge→Drone→harness; the dashboard + `*.ci.commoninternet.net` load; the loops' `ssh cc-ci` deploys
land on Hetzner. Re-run the relevant D-gates cold-verified by the Adversary.
5. Make `cc-ci-hetzner` the **canonical** `nixosConfigurations.cc-ci` in the flake (retire the Incus
`hardware.nix` once the old box is gone).
## Phase 4 — Retire the old Incus cc-nix-test
Once Hetzner is the verified live cc-ci: **stop** the Incus VM via the b1 Incus API (mTLS certs are on
this VM under `incus-terraform-nix-vm-creator/terraform-secrets/`) — `PUT .../instances/cc-nix-test/
state {"action":"stop"}`. Keep it as a **cold standby for a few days**, then delete (frees b1). Update
the memory/docs ([[cc-ci-setup]]) to point cc-ci at Hetzner.
## Who does what
- **Assistant:** Phase 1 (the terraform + full convergence) and the Phase-2 benchmark runs.
- **Orchestrator (me) + operator:** Phase 3 cutover (I do the ssh-repoint + the Incus stop via the
API; **operator does the DNS change** + the go/no-go) and Phase 4.
## Guardrails
- **Parallel bring-up** — never break the running Incus cc-ci until Hetzner is verified green; the
cutover is the only switch moment, at a quiet point.
- **No secrets in git** — `HCLOUD_TOKEN`, TS key, the age key (`.sops/`), tfstate all gitignored
(`.gitignore` hardened for `*age*.txt`/`.sops/`); never echo/commit them.
- **x86 `cpx32`**, pin the hcloud provider + nixos-infect rev (nixpkgs already pinned).
- **Reproducible-from-scratch holds** (the D8 guarantee) — the Hetzner cc-ci comes from `terraform
apply` + one `nixos-rebuild switch`, no hand steps beyond the operator DNS + age key.
## Definition of Done
- Hetzner `cpx32` cc-ci fully converged (0 failed units) via terraform + the D8 flake flow.
- `docs/perf/hetzner-vs-incus.md` shows the two-recipe old-vs-new comparison (real numbers).
- The loops, `!testme` pipeline, dashboard, and `*.ci.commoninternet.net` all run on Hetzner; a PR
`!testme` is green end-to-end there; D-gates re-verified.
- The Incus `cc-nix-test` is stopped (cold standby → deletion); the flake's canonical `cc-ci` host is
Hetzner; docs/memory updated.
cc-ci-plan/plan-orchestrator-hetzner-migration.md
-80
View File
@@ -1,80 +0,0 @@
# Plan/record — migrate the ORCHESTRATOR off the Incus VM onto a Hetzner cloud server
**Status:** COMPLETE (2026-05-31). The orchestrator (Builder/Adversary loops + watchdog + this
supervising session) now runs on a dedicated **Hetzner** cloud server, declared by the
`cc-ci-orchestrator-hetzner` flake host. Kept as a historical record.
**Why:** the previous orchestrator host was the Incus VM `cc-ci-orchestrator` on b1
(`100.116.55.106`, 2 GB / 2 vCPU, see [[plan-orchestrator-migration]] — the earlier Pi→Incus move).
A dedicated Hetzner box gives dedicated vCPU + NVMe and decouples the orchestrator from b1's hardware.
This is the orchestrator analogue of the cc-ci **server** move in [[plan-migrate-cc-ci-to-hetzner]]
(that one moves the *CI server*; this one moves the *orchestrator that drives the loops*).
> **Note on naming:** this migration was carried out directly via `terraform/` + the
> `cc-ci-orchestrator-hetzner` flake host. It is **not** the same as `plan-migrate-cc-ci-to-hetzner.md`
> (the cc-ci CI server → Hetzner `cpx32`) nor `plan-orchestrator-migration.md` (Pi → Incus VM). All
> three are distinct moves; only this file records the orchestrator → Hetzner step.
---
## The new host (facts)
| | |
|---|---|
| Provider / type | **Hetzner Cloud `cpx22`** — AMD **2 vCPU / 4 GB**, dedicated vCPU, NVMe |
| Location | `nbg1` (cpx11/cpx21 are retired there — hence `cpx22`) |
| Hetzner server ID | **134487234** |
| Public IPv4 | **168.119.126.100** (IPv6 disabled) |
| Tailnet | **`cc-ci-orchestrator-1`** @ **100.84.190.30** (`taila4a0bf.ts.net`); joins via `/etc/ts-auth-key` |
| OS | `debian-12` image → **nixos-infect** → NixOS, converged by the flake |
| Flake host | **`nixosConfigurations.cc-ci-orchestrator-hetzner`** (`flake.nix` → `nix/hosts/cc-ci-orchestrator-hetzner/{configuration,hardware}.nix`) |
| Workspace | `/srv/cc-ci-orch` (this repo); `/srv/cc-ci` is a **symlink** to it. Loop clones: `/srv/cc-ci/cc-ci`, `/srv/cc-ci/cc-ci-adv` |
The login keys (root `authorizedKeys`) and swap (4 GB disk swap — 4 GB RAM is tight for 3+ claude
sessions) are declared in `configuration.nix`.
## How it was provisioned (reproducible)
The whole box is reproducible from `terraform/` + one `nixos-rebuild`:
1. **`terraform apply`** (`terraform/main.tf`): `hcloud_server` `cpx22` from `debian-12` in `nbg1`,
`user_data = user-data.sh` runs **nixos-infect** on first boot (Debian→NixOS, reboot).
2. **Stage 2** (`terraform/README.md`): SSH in, capture the nixos-infect hardware config
(→ `nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix`), then converge:
```bash
# on the server, from the repo root (/srv/cc-ci-orch)
nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner
```
3. Stage credentials (not in git, placed once): `/etc/ts-auth-key` (tailnet join), the loops'
`~/.ssh/cc-ci-root-ed25519` + `.testenv`, and the sops master age key. `claude auth login`
(device code) is the one interactive step so the loops can run `--remote-control`.
4. Stage the workspace: clone this repo to `/srv/cc-ci-orch` (symlink `/srv/cc-ci`), the Builder /
Adversary clones, `cc-ci-secrets`, `references/`; copy `.cc-ci-logs/.phase-idx` (resume point).
**Commit trail:** `0103f36` (terraform + flake host, initial `cpx11`) → `17951b8` (fix → `cpx22`,
add lock) → `c44b967` (real cpx22 hardware config from nixos-infect, server 134487234). Plus the
close-out commit below (root keys, drop tailscale `--ssh`, enable the loops service, this doc).
## Reboot-resilience (the point of running on a managed host)
`configuration.nix` declares **`systemd.services.cc-ci-loops`** — a oneshot that runs
`launch.sh start` with `RESUME_PHASE=1` after `network-online`/`tailscaled`, bringing the loops +
watchdog back on boot. It was authored **disabled** ("defined but NOT enabled until workspace is
staged") with `wantedBy` commented out. **Close-out (2026-05-31):** the workspace is staged and the
loops are running, so `wantedBy = [ "multi-user.target" ]` was uncommented and `nixos-rebuild switch`
re-run → `systemctl is-enabled cc-ci-loops.service` = **enabled**. A reboot is now a non-event:
systemd resumes the saved phase. (`reboot-log.sh`, the ExecStartPre, appends to
[[REBOOTS.md]] boot_id-gated.)
> **Caveat seen at first boot on this host:** the loops were initially started *by hand* during
> staging (not by the service), so the first boot did NOT log to `REBOOTS.md` and the service showed
> `linked`/not-enabled. Enabling `wantedBy` (above) is what wires the automatic path.
## Status of the migration
- ✅ Hetzner `cpx22` provisioned + converged from the flake (terraform + nixos-infect + one rebuild).
- ✅ On the tailnet (`cc-ci-orchestrator-1`) and ssh-able on the public IP.
- ✅ Loops + Adversary + watchdog running; phase sequence auto-advancing (watchdog on per-phase `## DONE`).
- ✅ `cc-ci-loops.service` **enabled** → reboot-resilient.
- ◻︎ Old Incus orchestrator VM (`100.116.55.106`) — keep as cold standby a few days, then delete.
- ◻︎ Rotate the tailnet name once the old `cc-ci-orchestrator` peer is gone (this box is `…-1`).
cc-ci-plan/plan-orchestrator-migration.md
-151
View File
@@ -1,151 +0,0 @@
# Plan — migrate the orchestrator off the Pi onto a dedicated NixOS Incus VM
**Goal:** move everything that drives the cc-ci loops (the Builder/Adversary loops, the watchdog,
the SOCKS proxy, the orchestrator session itself) off the Raspberry Pi and onto a new, dedicated,
**reboot-resilient NixOS VM** on b1 — declared in a new git repo **`cc-ci-orchestrator`**. Finish by
relocating this orchestrator session there too.
**Why:** the Pi has rebooted twice today, each time silently killing the tmux loops + watchdog
(they don't survive reboot, nothing auto-restarts them). A NixOS VM lets us declare the whole rig
(claude CLI, proxy, loop supervisor) as systemd services that come back on boot — turning a reboot
into a non-event. It also consolidates the orchestrator next to the infra it manages.
**Status:** COMPLETE (2026-05-31). All agents run on the VM; Pi fully decommissioned. Kept as a historical record.
**Phase A ✅ COMPLETE (2026-05-30):** VM `cc-ci-orchestrator` (**2 GB / 2 vCPU / 30 GB**,
`incus-base-vm`, NixOS 24.11) created via the Incus API + booted; **on the tailnet at
`100.116.55.106`**; **ssh works** (`ssh cc-ci-orchestrator` through the :1055 proxy — `cc-ci-root`
pubkey added via `incus exec`). Reproducible Terraform record at
`incus-terraform-nix-vm-creator/projects/cc-ci-orchestrator/` (note: this instance was API-created, so
TF drift — see PROVENANCE.txt).
- **TS-key finding:** the VM-creator's `.test.env` reusable key is **REVOKED** ("API key does not
exist"). The **`/srv/cc-ci/.testenv` `TS_AUTH_KEY` is valid** — used it to join, and persisted it into
the VM's `/etc/ts-auth-key`. So the plan's "operator provides a fresh TS key" item is **resolved** (no
new key needed); housekeeping: revoke/rotate the dead key in `.test.env`.
- **Sizing watch:** 2 GB ≈ 1.7 GiB usable; fine idle (284 MiB) but tight for 3 concurrent claude
sessions (Pi OOM lesson). Phase B will declare a **swapfile**; bump to 4 GB pre-cutover if needed.
**Next — Phase B:** the `cc-ci-orchestrator` NixOS-config git repo (SOCKS proxy + loop-supervisor boot
service + claude CLI + sops secrets). Then C (stage workspace), claude auth (operator), D/E (cutover).
---
## 0. Current footprint (what has to move)
On the Pi (`raspberrypi`, aarch64), workspace `/srv/cc-ci` (itself the
`cc-ci-orchestrator` git repo — formerly `cc-ci-autonomous-orchestrator`):
| Item | What | Move strategy |
|---|---|---|
| `cc-ci-plan/` | loop code: `launch.sh`, `plan*.md`, `prompts/`, `kickoff.md` | in git (this repo) → clone on VM |
| `cc-ci/`, `cc-ci-adv/` | Builder + Adversary working clones (~13M each) | **re-clone from git.autonomic.zone** on the VM (cleaner than copying) |
| `.cc-ci-logs/` | watchdog/loop logs + `.phase-idx` | copy `.phase-idx` (the resume point); logs start fresh |
| `cc-ci-secrets/` | sops-encrypted secrets repo | in git → clone |
| `references/` | recipe-maintainer corpus (read-only parity source) | clone/rsync from `/srv/recipe-maintainer` |
| **`.testenv`** | TS auth key, Gitea bot creds | **out-of-band copy** (gitignored, never in git) |
| **`~/.ssh/cc-ci-root-ed25519`** | root SSH key to cc-ci | **out-of-band copy** |
| **`.sops/master-age.txt`** | master recovery age key | **out-of-band copy** |
| **Incus mTLS certs** (`/srv/incus-terraform-nix-vm-creator/terraform-secrets/`) | `terraform.{crt,key}`, `vm_ssh_key` | **out-of-band copy** — so the VM can itself manage VMs |
| `cc-ci-tailscaled.service` | userspace SOCKS proxy :1055 | **re-declare as NixOS** (see §3) |
| **claude CLI + auth** | `~/.local/bin/claude` v2.1.154 + `~/.claude.json` | install on VM + **operator `claude auth login`** (§4) |
| this orchestrator session | the supervising claude conversation | **operator-assisted cutover** (§6) |
Two hard human-in-the-loop steps, called out explicitly: **claude auth on the new VM** (device-code
login, can't be scripted) and the **final session cutover** (the operator connects to the new
orchestrator session). Everything else I can do.
## 1. Target VM spec
- **Host/API:** b1 Incus, `https://100.117.251.31:8443`, project `terraform-ci`, mTLS certs (have).
- **Name:** `cc-ci-orchestrator` (tailnet hostname too).
- **Resources:** **2 GB RAM, 2 vCPU, 30 GB disk** (dir backend → resize needs a reboot; size at
create time so no later grow). b1 has ample headroom (only cc-nix-test @8GB running).
- **Image:** the existing imported NixOS base VM image (`incus-base-vm`) — already ships tailscale,
openssh, git/jq/curl, flakes, cloud-init.
- **Tailnet:** joins via a fresh `TS_AUTH_KEY` (operator provides, or reuse the keyed approach in
`terraform-secrets/.test.env`). MagicDNS name `cc-ci-orchestrator.taila4a0bf.ts.net`.
- **Bootstrap:** cloud-init writes the `cc-ci-orchestrator` flake config + `nixos-rebuild switch`.
## 2. The new `cc-ci-orchestrator` git repo (NixOS config)
A new **private** repo on `git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator` (bot is org
admin). It is the NixOS config for this VM — the orchestrator's equivalent of what `cc-ci` is for the
test server. Contents:
- `flake.nix` + `hosts/cc-ci-orchestrator/configuration.nix` — the VM's NixOS config.
- **Packages:** `claude-code` (CLI), `git`, `tmux`, `python3`, `jq`, `openssh`, `nodejs` (claude
runtime), `coreutils`, `nettools` (`nc` for the proxy ProxyCommand).
- **`services.cc-ci-tailscaled`** — the userspace tailscaled SOCKS proxy on :1055, as a NixOS
systemd service (port to NixOS from the Pi's `cc-ci-tailscaled.service`). This is the path to b1 +
cc-ci.
- **`services.cc-ci-orchestrator`** — a systemd service that runs `launch.sh start` with
`RESUME_PHASE=1` **on boot** (after the proxy + network are up), as the workspace user. **This is
the reboot-resilience fix** — the loops + watchdog come back automatically after any reboot.
- **Secrets via sops-nix** (like cc-ci): the out-of-band secrets (`.testenv`, ssh key, incus certs)
are sops-encrypted into the repo, decrypted at activation to their runtime paths. The **master age
key** is the one irreducible out-of-band bootstrap secret placed on the VM once.
- `~/.ssh/config` for `cc-ci` (root, ProxyCommand via :1055) declared.
- **Excluded from git:** claude's own auth (`~/.claude.json`) — that's per-user login state, set up
once interactively (§4), not committed.
## 3. Execution phases
### Phase A — provision the VM (reversible; safe to do while Pi loops keep running)
1. Create `cc-ci-orchestrator` VM via the Incus API (2 GB / 2 vCPU / 30 GB, NixOS base image, TS auth
key in cloud-init). Wait for tailnet join + ssh.
2. Verify: `ssh` in, `tailscale status`, `nixos-rebuild` available, can reach b1 API + cc-ci through
its own proxy once configured.
### Phase B — author + apply the `cc-ci-orchestrator` repo
3. Create the private git repo; author the flake/config (§2); commit/push.
4. Place the master age key on the VM; sops-encrypt the out-of-band secrets into the repo.
5. `nixos-rebuild switch` on the VM → proxy service up, packages present, services defined (loop
supervisor **not yet started** — or started in a dry mode).
### Phase C — stage the workspace (no cutover yet)
6. On the VM: clone `cc-ci-orchestrator` (the loop code), clone the Builder/Adversary
working repos fresh from git.autonomic.zone, clone `cc-ci-secrets`, rsync `references/`.
7. Copy `.phase-idx` (resume point = phase 2) so the VM watchdog resumes the right phase.
8. **Operator step:** `claude auth login` on the VM (device code) so the loops can run
`--remote-control --dangerously-skip-permissions`. Verify with a throwaway interactive claude.
### Phase D — cutover (the only disruptive moment; pick a clean point)
9. **Quiesce the Pi:** stop the Pi loops + watchdog (`launch.sh stop`); confirm both loops are at a
safe point (no half-written commit; `git status` clean in both clones, last work pushed).
10. **Start on the VM:** enable + start the `cc-ci-orchestrator` systemd service → `launch.sh start`
(RESUME_PHASE=1) brings up Builder + Adversary + watchdog on the VM, resuming phase 2 from the
repo state. Verify all three sessions + a handoff + public health.
11. **Decommission the Pi loops:** disable the Pi's `cc-ci-tailscaled` + leave the workspace in place
(read-only fallback) but not running loops. (Keep the Pi as a cold standby for a few days before
deleting anything.)
### Phase E — move the orchestrator session (operator-assisted)
12. On the VM, start the orchestrator session: `claude --remote-control 'autonomous-orchestrator'
--dangerously-skip-permissions` in a tmux session, seeded with AGENTS.md + this plan so it picks
up the supervising role. The **operator connects** to it (claude.ai/code) — this is the
"move myself" step; a session can't transplant itself across machines, so it's a fresh
orchestrator session on the VM with full context from the repo.
13. This Pi-side orchestrator session hands off (writes a short state note) and goes idle/ends.
## 4. Risks & mitigations
- **claude auth (human step):** unavoidable device-code login on the VM. Mitigation: do it in Phase
C, well before cutover; verify before quiescing the Pi.
- **Loops mid-work at cutover:** pick a quiet point (between gate claims / after a push); the loops
re-orient from git on restart anyway, so worst case is a re-run of an in-flight iteration.
- **Secrets sprawl:** out-of-band secrets are copied once, then sops-managed in the new repo; never
committed in plaintext (same discipline as cc-ci). The master age key is the sole bootstrap secret.
- **Self-move gap:** between Pi-session-ends and VM-session-connected, there's no live orchestrator.
The watchdog (now a boot service) keeps the loops alive independently, so this gap is safe.
- **Rollback:** until the Pi workspace is deleted, reverting = stop VM service, `launch.sh start` on
the Pi again. Keep the Pi intact until the VM has run clean through at least one reboot + one gate
handshake.
- **Reboot-resilience proof:** before trusting the VM, reboot it once and confirm the loops +
watchdog + proxy all come back via systemd (the whole point of the move).
## 5. Operator-assisted steps (the only things I can't fully do)
1. Provide a fresh `TS_AUTH_KEY` for the VM (or confirm reuse of the one in `terraform-secrets`).
2. `claude auth login` on the VM (device code).
3. Connect to the new orchestrator session on the VM at cutover (Phase E).
Everything else (VM create, repo author, NixOS config, secret migration, workspace staging, the
loop cutover) I can drive.
cc-ci-plan/plan-repo-consolidation.md
-58
View File
@@ -1,58 +0,0 @@
# Plan stub — repo consolidation + recipe-maintainer submodule
**Status:** COMPLETE (2026-05-31).
Both cleanups executed. See commit history for details.
---
## Task 1 — merge the two orchestrator repos into one `cc-ci-orchestrator`
Today there are **two** repos:
- `recipe-maintainers/cc-ci-autonomous-orchestrator` — the loop code/plans/prompts (`cc-ci-plan/`,
`launch.sh`, `prompts/`, `references/`, `AGENTS.md`). This is the workspace at `/srv/cc-ci`
(`/home/loops/cc-ci` on the VM).
- `recipe-maintainers/cc-ci-orchestrator` — the VM's NixOS config (`configuration.nix`, `README.md`),
created during the VM migration.
**Goal:** combine into a single repo named **`cc-ci-orchestrator`** (loop code/plans + the VM's NixOS
config in one place). Likely: move the NixOS config into the autonomous-orchestrator repo (e.g.
`nixos/` or `vm/`), then **rename** that repo to `cc-ci-orchestrator` on Gitea.
**Touch-ups when doing it:** the workspace clone URL in `launch-orchestrator.sh` / kickoff docs /
`cc-ci-loops.service`, the VM's `git remote`, and `plan.md`/`kickoff.md` references to the repo name.
Keep `recipe-maintainers/cc-ci` (the CI **product** repo the loops build) separate — this is only
about the two **orchestrator** repos.
## Task 2 — make `references/recipe-maintainer` a git submodule
Today `references/recipe-maintainer` is a **symlink → `/srv/recipe-maintainer/`** (an absolute path).
On the Pi that target existed; **on the VM it does NOT, so the symlink dangles and the parity corpus
is empty** — Phase-2 parity-porting (P2) reads from `references/recipe-maintainer/recipe-info/<recipe>/
tests/*.py`, so recipes not already ported can't be ported until this is fixed.
**Goal:** replace the symlink with a proper **git submodule** from
**`https://git.autonomic.zone/notplants/recipe-maintainer`**:
```
git rm references/recipe-maintainer # remove the symlink
git submodule add https://git.autonomic.zone/notplants/recipe-maintainer references/recipe-maintainer
git commit
# on each clone / the VM: git submodule update --init --recursive
```
This both fixes the current VM gap (corpus present + reproducible, no absolute-path dependency) and
makes the workspace self-contained on any host.
**Interim (before this plan runs), if the loops need the corpus on the VM:** clone recipe-maintainer
to `/srv/recipe-maintainer` on the VM (needs root — `/srv` is root-owned) so the existing symlink
resolves; OR do Task 2 directly.
## Operator action required — Pi clone remote update
The Gitea rename (`cc-ci-autonomous-orchestrator` → `cc-ci-orchestrator`) breaks the `origin` remote
on any existing clone. **On the Pi** (and any other machine with a clone), run:
```
cd /srv/cc-ci # or wherever the workspace is cloned
git remote set-url origin https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git
# (include bot credentials in the URL if needed, as before)
```
The VM clone's remote was updated automatically as part of this task.