nix: one Hetzner host for the CI server + the orchestrator (#cc-ci)
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).
nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.
README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
Claude Fable 5.1
@@ -0,0 +1,22 @@
|
||||
# archive/ — how cc-ci and its orchestrator were built and moved, before the combined host
|
||||
|
||||
Historical record only. Nothing in here is deployed or evaluated. It was moved out of the live
|
||||
tree on 2026-09-07 when the CI server and the orchestrator were consolidated onto one Hetzner
|
||||
host (`nixosConfigurations.cc-ci` in `../flake.nix`; deploy guide in `../README.md`; the plan
|
||||
that did it is `../cc-ci-plan/plan-cc-ci-combined-host.md`).
|
||||
|
||||
| path | what it was |
|
||||
|---|---|
|
||||
| `nix/configuration-incus-vm.nix` | Channel-based NixOS config of the first orchestrator VM on b1 (Incus, 2 GB). Ran the loops as root; hard-coded the dead Incus cc-ci IP. Replaced by the Hetzner host 2026-05-31. |
|
||||
| `nix/README.md` | The README for that Incus VM config. |
|
||||
| `nix/cc-ci-orchestrator-hetzner/` | The orchestrator's own Hetzner `cpx22` host (`168.119.126.100`, tailnet `cc-ci-orchestrator-1`), 2026-05-31 → 2026-09. From 2026-08-20 the live copy of this config was `notplants-nix`'s `notplants-orchestrator` host (the box became a shared agent host for several projects); this one had drifted and still carried lichen/project-orchestrator units. Superseded by `../nix/hosts/cc-ci` + `../nix/modules/orchestrator-host.nix`. |
|
||||
| `nix/atproto-likes.nix` | A notplants (not cc-ci) service that lived on the shared box; kept by notplants-nix. |
|
||||
| `terraform/` | OpenTofu for the `cpx22` orchestrator server (Debian 12 → nixos-infect at `nixos-24.11`). The combined host was provisioned by hand instead; the README documents that path. Note its `user-data.sh` would fail on the Debian 13 image (nixos-infect's temp swapfile on a tmpfs `/tmp`) — see the README's `NO_SWAP=true` note. |
|
||||
| `plans/plan-orchestrator-migration.md` | Pi → Incus VM move of the orchestrator (2026-05). |
|
||||
| `plans/plan-orchestrator-hetzner-migration.md` | Incus VM → Hetzner `cpx22` move of the orchestrator (2026-05-31). Has the reboot-resilience design (`cc-ci-loops.service`). |
|
||||
| `plans/plan-migrate-cc-ci-to-hetzner.md`, `plans/plan-cc-ci-hetzner-migration.md`, `plans/plan-cc-ci-hetzner-terraform.md` | The CI server's own move from the `cc-nix-test` Incus VM to Hetzner `cpx32` (`91.98.47.73`, 2026-05-31), and the terraform that provisioned it (lives in the cc-ci repo). |
|
||||
| `plans/plan-repo-consolidation.md` | The earlier repo layout consolidation. |
|
||||
|
||||
The cc-ci server's own history (machine-docs, decisions, the clean-room rebuild that proved
|
||||
"two repos + one age key + one `nixos-rebuild switch`") is in the cc-ci repo under
|
||||
`machine-docs/` and `docs/`.
|
||||
@@ -0,0 +1,35 @@
|
||||
# cc-ci-orchestrator
|
||||
|
||||
> ⚠️ **HISTORICAL.** This README describes the retired **Incus VM** (`100.116.55.106`). The
|
||||
> orchestrator now runs on **Hetzner** — the live host config is
|
||||
> `nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix`. See
|
||||
> `cc-ci-plan/plan-orchestrator-hetzner-migration.md` for the current setup. Kept for history.
|
||||
|
||||
NixOS config for the **`cc-ci-orchestrator`** Incus VM (b1, project `terraform-ci`, tailnet
|
||||
`100.116.55.106`) — the reboot-resilient host for the cc-ci Builder/Adversary loops + watchdog +
|
||||
orchestrator session, moved off the unstable 905 MiB Pi.
|
||||
|
||||
See `cc-ci-plan/plan-orchestrator-migration.md` for the full migration.
|
||||
|
||||
## Files
|
||||
- `configuration.nix` — the VM's NixOS config (channel-based, `nixos-24.11`). Deployed to
|
||||
`/etc/nixos/configuration.nix` on the VM. Provides: nix-ld (so the standalone Claude Code Bun binary
|
||||
runs), tmux/git/python/jq + tools, a 4 GB swapfile, direct ssh to cc-ci (the VM is a tailnet peer —
|
||||
no SOCKS proxy needed, unlike the Pi), an idempotent `claude-install` oneshot, and the
|
||||
`cc-ci-loops` supervisor service (defined, **enabled in Phase D** once the workspace is staged).
|
||||
|
||||
## Deploy (until this is wired to a flake/auto-pull)
|
||||
```
|
||||
# copy configuration.nix to the VM, then:
|
||||
ssh cc-ci-orchestrator 'nixos-rebuild switch' # or run detached: see below
|
||||
```
|
||||
Over the (currently flaky) Pi→VM link, run the rebuild **detached** on the VM so an ssh/proxy drop
|
||||
doesn't abort it, e.g. `systemd-run --unit=orch-rebuild --collect nixos-rebuild switch` then poll
|
||||
`journalctl -u orch-rebuild`.
|
||||
|
||||
## Status
|
||||
- Phase A: VM created (2 GB / 2 vCPU / 30 GB), on tailnet, ssh-able. ✅
|
||||
- Phase B: this config (DRAFT) — nix-ld/claude validation pending on the VM.
|
||||
- Operator step pending (Phase C): `claude auth login` on the VM (device-code; can't be scripted).
|
||||
- Secrets to stage (Phase C, out-of-band): `/srv/cc-ci/.testenv`, `~/.ssh/cc-ci-root-ed25519`,
|
||||
Incus mTLS certs, the sops master age key.
|
||||
@@ -0,0 +1,135 @@
|
||||
# atproto-likes — the "most-liked accounts" web UI, run as a docker-compose stack
|
||||
#
|
||||
# ⚠️ COPY. Canonical source:
|
||||
# /srv/project-orchestrator/projects/notplants-atproto/nix/atproto-likes.nix
|
||||
# Pure evaluation cannot import a path outside the flake tree. Re-copy after editing.
|
||||
#
|
||||
# Host-specific context that is deliberately NOT in the public project repo:
|
||||
# - the domain is fronted by the gateway (143.244.213.108); it forwards everything here
|
||||
# - this vhost lists 0.0.0.0 only so it never shares an nginx address group with
|
||||
# oc.commoninternet.net (opencode, on the tailscale address, no auth of its own)
|
||||
# - /srv/project-orchestrator/projects/notplants-atproto/tools/check-exposure.sh asserts
|
||||
# what this host serves publicly; run it after any nginx change
|
||||
# behind the host's nginx.
|
||||
#
|
||||
# Import this from the host configuration:
|
||||
# imports = [ /srv/project-orchestrator/projects/notplants-atproto/nix/atproto-likes.nix ];
|
||||
#
|
||||
# What it sets up:
|
||||
# * the Docker daemon
|
||||
# * a systemd unit that builds and runs docker-compose.yml from the project checkout
|
||||
# * an nginx vhost for atproto.commoninternet.net with a Let's Encrypt cert,
|
||||
# proxying to the container on 127.0.0.1:8731
|
||||
# * ports 80/443 opened in the firewall
|
||||
#
|
||||
# DNS / TLS: point the domain's A record at this host (or at a reverse proxy that
|
||||
# forwards to it). ACME HTTP-01 needs the challenge to reach this nginx, so if a
|
||||
# proxy fronts the domain it must forward /.well-known/acme-challenge/ through.
|
||||
# Until a certificate is issued, NixOS installs a self-signed placeholder so nginx
|
||||
# still starts.
|
||||
#
|
||||
# A proxy in front should preserve the Host header (nginx routes by server_name),
|
||||
# pass Upgrade/Connection through (the loading page is a websocket) and allow a
|
||||
# long read timeout (a cold scan can run for minutes).
|
||||
{ config, pkgs, lib, ... }:
|
||||
|
||||
let
|
||||
projectDir = "/srv/project-orchestrator/projects/notplants-atproto";
|
||||
domain = "atproto.commoninternet.net";
|
||||
hostPort = 8731; # must match the ports: mapping in docker-compose.yml
|
||||
compose = "${pkgs.docker-compose}/bin/docker-compose";
|
||||
in
|
||||
{
|
||||
virtualisation.docker = {
|
||||
enable = true;
|
||||
# Reclaim dangling images from repeated `--build` runs.
|
||||
autoPrune = { enable = true; dates = "weekly"; };
|
||||
};
|
||||
|
||||
# Build + run the compose stack. Type=oneshot with RemainAfterExit: compose
|
||||
# detaches, and the containers' own restart policy keeps them alive.
|
||||
systemd.services.atproto-likes = {
|
||||
description = "atproto-likes — most-liked-accounts web UI (docker compose)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "docker.service" "docker.socket" "network-online.target" ];
|
||||
requires = [ "docker.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
path = [ pkgs.docker pkgs.docker-compose ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
WorkingDirectory = projectDir;
|
||||
# A cold `--build` pulls python:3.12-slim and installs pip deps.
|
||||
TimeoutStartSec = "1800";
|
||||
# The container runs unprivileged as uid 10001 (see Dockerfile), but the
|
||||
# page cache is a host bind mount, so its ownership is the host's business.
|
||||
# Without this the app 500s on any actor whose cache files were written by
|
||||
# an earlier root-running container.
|
||||
ExecStartPre = [
|
||||
"${pkgs.coreutils}/bin/mkdir -p ${projectDir}/cache"
|
||||
"${pkgs.coreutils}/bin/chown -R 10001:10001 ${projectDir}/cache"
|
||||
];
|
||||
ExecStart = "${compose} up -d --build --remove-orphans";
|
||||
ExecStop = "${compose} down";
|
||||
Restart = "on-failure";
|
||||
RestartSec = "30s";
|
||||
};
|
||||
};
|
||||
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
virtualHosts.${domain} = {
|
||||
# addSSL, NOT forceSSL. If a reverse proxy fronts this domain and its
|
||||
# back-end hop is plain HTTP, a forced 301 to https://<domain>/ would
|
||||
# resolve straight back to that proxy — a redirect loop. Serving both
|
||||
# schemes lets whatever is in front decide where TLS terminates.
|
||||
addSSL = true;
|
||||
enableACME = true;
|
||||
# Listens on 0.0.0.0 only, and the reject-everything default server below
|
||||
# owns those addresses. nginx groups server blocks by the connection's local
|
||||
# address, so keeping this vhost on the public address alone means it never
|
||||
# shares an address group with anything else the host may serve on another
|
||||
# interface — no other service can be reached by sending this one an
|
||||
# unexpected Host header.
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:${toString hostPort}";
|
||||
# The loading page streams scan progress over a websocket.
|
||||
proxyWebsockets = true;
|
||||
# A cold scan can run for minutes with the socket open; the default 60s
|
||||
# proxy read timeout would cut the loading page off mid-flower.
|
||||
extraConfig = ''
|
||||
proxy_read_timeout 1800s;
|
||||
proxy_send_timeout 1800s;
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
# Strict default server for the public addresses. Without an explicit
|
||||
# default_server, nginx promotes the first server block in the group — so any
|
||||
# vhost added later silently becomes what answers unmatched, absent or spoofed
|
||||
# Host headers from the internet. This closes that door once: only names
|
||||
# deliberately served get a response.
|
||||
#
|
||||
# port 80 -> 444 (close the connection, no response at all)
|
||||
# port 443 -> ssl_reject_handshake, so an unknown SNI never even gets TLS
|
||||
virtualHosts."public-default-reject" = {
|
||||
default = true;
|
||||
serverName = null;
|
||||
rejectSSL = true;
|
||||
listen = [
|
||||
{ addr = "0.0.0.0"; port = 80; ssl = false; }
|
||||
{ addr = "0.0.0.0"; port = 443; ssl = true; }
|
||||
];
|
||||
extraConfig = "return 444;";
|
||||
};
|
||||
};
|
||||
|
||||
security.acme = {
|
||||
acceptTerms = true;
|
||||
defaults.email = "mfowler.email@protonmail.com"; # ACME contact
|
||||
};
|
||||
|
||||
# Public HTTP/HTTPS.
|
||||
networking.firewall.allowedTCPPorts = [ 80 443 ];
|
||||
}
|
||||
@@ -0,0 +1,305 @@
|
||||
# cc-ci-orchestrator-hetzner — NixOS config for the Hetzner loops runtime host.
|
||||
#
|
||||
# Purpose: run the cc-ci Builder/Adversary/Watchdog loops + orchestrator/assistant sessions
|
||||
# on a Hetzner cpx11 (2 vCPU / 2 GB dedicated AMD / 40 GB NVMe), replacing the slow b1 Incus VM.
|
||||
#
|
||||
# Provision with terraform/ then converge with: nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner
|
||||
# See terraform/README.md for the full Stage 2 procedure.
|
||||
{ config, pkgs, lib, ... }:
|
||||
{
|
||||
# hardware.nix is the nixos-infect generated hardware-configuration.nix (see README Stage 2a).
|
||||
|
||||
# atproto-likes — the notplants-atproto "most-liked accounts" web UI, served at
|
||||
# atproto.commoninternet.net. Brings in Docker + a compose stack + an nginx vhost
|
||||
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
|
||||
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
|
||||
# because pure evaluation cannot import a path outside the flake.
|
||||
# cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
|
||||
# the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
|
||||
# a general agent box that also serves unrelated projects — can be reasoned about separately
|
||||
# from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
|
||||
imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
|
||||
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
settings.PermitRootLogin = "yes";
|
||||
};
|
||||
|
||||
# Root SSH access — all keys from the current orchestrator VM's /root/.ssh/authorized_keys.
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526"
|
||||
];
|
||||
networking.useDHCP = true;
|
||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||
networking.firewall = {
|
||||
enable = true;
|
||||
trustedInterfaces = [ "tailscale0" ];
|
||||
# Port 80 open only on the tailscale interface (trusted) — nginx binds there for oc.commoninternet.net.
|
||||
allowedTCPPorts = [ 22 ];
|
||||
};
|
||||
nix.settings.experimental-features = [ "nix-command" "flakes" ];
|
||||
system.stateVersion = "24.11";
|
||||
|
||||
# Tailscale — auth key at /etc/ts-auth-key (placed manually in Stage 2, not in git).
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
authKeyFile = "/etc/ts-auth-key";
|
||||
extraUpFlags = [ "--hostname=cc-ci-orchestrator" ];
|
||||
};
|
||||
|
||||
# 4 GB disk swap — claude session memory safety net (2 GB RAM is tight for 3+ sessions).
|
||||
swapDevices = [ { device = "/swapfile"; size = 4096; } ];
|
||||
|
||||
# nix-ld — lets the standalone Claude Code CLI (foreign dynamic ELF / Bun) run on NixOS.
|
||||
programs.nix-ld.enable = true;
|
||||
programs.nix-ld.libraries = with pkgs; [
|
||||
stdenv.cc.cc.lib
|
||||
zlib
|
||||
openssl
|
||||
curl
|
||||
glibc
|
||||
];
|
||||
|
||||
environment.systemPackages = with pkgs; [
|
||||
git tmux python3 jq curl cacert
|
||||
gnused gawk coreutils gnugrep findutils util-linux
|
||||
nettools openssh
|
||||
age sops # key management (same toolchain as cc-ci server)
|
||||
];
|
||||
|
||||
# loops user — claude sessions run as non-root (--dangerously-skip-permissions blocked for root).
|
||||
users.users.loops = {
|
||||
isNormalUser = true;
|
||||
home = "/home/loops";
|
||||
shell = pkgs.bash;
|
||||
extraGroups = [ "wheel" ];
|
||||
};
|
||||
security.sudo.wheelNeedsPassword = false;
|
||||
security.sudo.extraRules = [{
|
||||
users = [ "loops" ];
|
||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
||||
}];
|
||||
|
||||
# Ensure /home/loops/.local/bin (claude + opencode) is on the loops user PATH.
|
||||
# opencode binary is installed there manually (not yet in nixpkgs); re-install if missing:
|
||||
# curl -sL https://github.com/anomalyco/opencode/releases/download/v1.15.13/opencode-linux-x64.tar.gz \
|
||||
# | tar -xz -C /home/loops/.local/bin opencode && chmod +x /home/loops/.local/bin/opencode
|
||||
environment.variables.PATH = lib.mkForce
|
||||
"/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
||||
|
||||
# SSH config for the loops user — points to the cc-ci Hetzner server via tailnet.
|
||||
# HostName is the Hetzner cc-ci server's tailnet IP (cutover settled 2026-05-31).
|
||||
system.activationScripts.loopsSshConfig = ''
|
||||
mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
|
||||
# Only write if not already present (preserves manual customisation).
|
||||
if [ ! -f /home/loops/.ssh/config ]; then
|
||||
cat > /home/loops/.ssh/config <<'SSHCFG'
|
||||
Host cc-ci
|
||||
HostName 100.95.31.88
|
||||
User root
|
||||
IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
|
||||
IdentitiesOnly yes
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 30
|
||||
SSHCFG
|
||||
chmod 600 /home/loops/.ssh/config
|
||||
chown loops:users /home/loops/.ssh/config
|
||||
fi
|
||||
'';
|
||||
|
||||
# claude-install — fetch the standalone Claude Code CLI for the loops user if missing.
|
||||
systemd.services.claude-install = {
|
||||
description = "Install Claude Code CLI for loops user (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
|
||||
script = ''
|
||||
if [ ! -x "$HOME/.local/bin/claude" ]; then
|
||||
echo "installing Claude Code CLI for loops user..."
|
||||
curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
# opencode web server — one shared instance; agent sessions attach to it for web visibility.
|
||||
# Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
|
||||
# Public provider configuration is tracked in the cc-ci repo; credentials
|
||||
# remain in /srv/cc-ci/.testenv.
|
||||
systemd.services.opencode-web = {
|
||||
description = "opencode web server for cc-ci agents";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "simple";
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
||||
EnvironmentFile = [
|
||||
"-/srv/cc-ci/cc-ci/.env.public"
|
||||
"/srv/cc-ci/.testenv"
|
||||
];
|
||||
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
||||
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
||||
Restart = "on-failure";
|
||||
RestartSec = "5s";
|
||||
};
|
||||
environment = {
|
||||
HOME = "/home/loops";
|
||||
PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/home/loops/.nix-profile/bin:/nix/profile/bin:/home/loops/.local/state/nix/profile/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
|
||||
};
|
||||
path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
|
||||
};
|
||||
|
||||
# nginx — reverse-proxy oc.commoninternet.net → opencode web server.
|
||||
# Bound to the tailscale IP so it is only reachable on the tailnet.
|
||||
# DNS: add A record oc.commoninternet.net → 100.84.190.30 (operator step if hostname access is wanted).
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
virtualHosts."oc.commoninternet.net" = {
|
||||
# Listen on the tailscale interface only — not the public IP. Both 80 and 443.
|
||||
# 443 uses a SELF-SIGNED cert (below): this name resolves to a CGNAT tailscale
|
||||
# IP, so Let's Encrypt HTTP-01 can't validate it and there is no DNS-01 provider
|
||||
# configured on this host. The trusted *.ci.commoninternet.net wildcard lives on
|
||||
# the coop-cloud swarm (91.98.47.73), not here, and doesn't cover this label.
|
||||
# Self-signed is fine because the vhost is tailnet-only (trusted network); the
|
||||
# browser shows a one-time trust prompt. (Chosen by operator, 2026-08-03.)
|
||||
#
|
||||
# The cert/key are created out of band — same convention as oc-htpasswd, NOT in
|
||||
# git and NOT in the nix store (a store path would be world-readable):
|
||||
#
|
||||
# /etc/nginx/oc-selfsigned.crt root:nginx 0644
|
||||
# /etc/nginx/oc-selfsigned.key root:nginx 0640
|
||||
#
|
||||
# Regenerate (10y, SANs = hostname + tailscale IP + ts.net name) with:
|
||||
# openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
|
||||
# -keyout /etc/nginx/oc-selfsigned.key -out /etc/nginx/oc-selfsigned.crt \
|
||||
# -subj /CN=oc.commoninternet.net \
|
||||
# -addext "subjectAltName=DNS:oc.commoninternet.net,DNS:cc-ci-orchestrator-1.taila4a0bf.ts.net,IP:100.84.190.30"
|
||||
# sudo chown root:nginx /etc/nginx/oc-selfsigned.{crt,key}
|
||||
# sudo chmod 644 /etc/nginx/oc-selfsigned.crt && sudo chmod 640 /etc/nginx/oc-selfsigned.key
|
||||
#
|
||||
# NOTE: like oc-htpasswd, if these files go missing nginx fails to START — which
|
||||
# would take the atproto vhost down with it. Recreate them before rebuilding on
|
||||
# a fresh host.
|
||||
listen = [
|
||||
{ addr = "100.84.190.30"; port = 80; ssl = false; }
|
||||
{ addr = "100.84.190.30"; port = 443; ssl = true; }
|
||||
];
|
||||
# addSSL (serve BOTH http+https) is required, not cosmetic: the NixOS nginx
|
||||
# module only renders the `ssl_certificate` directives when a vhost is flagged
|
||||
# as an SSL vhost. An explicit `listen ... ssl` + sslCertificate WITHOUT this
|
||||
# flag produces an SSL listener with no cert → `nginx -t` fails and the whole
|
||||
# service (atproto included) won't start.
|
||||
addSSL = true;
|
||||
sslCertificate = "/etc/nginx/oc-selfsigned.crt";
|
||||
sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
|
||||
|
||||
# HTTP basic auth in front of opencode. The opencode web UI has no
|
||||
# authentication of its own and can drive agent sessions, so since this host
|
||||
# started serving 80/443 publicly (atproto-likes, 2026-08-01) it is worth a
|
||||
# second layer even though this vhost is tailnet-only.
|
||||
#
|
||||
# basicAuthFile, NOT basicAuth: the `basicAuth` attrset writes the password
|
||||
# into the nix store, which is world-readable. This file is created out of
|
||||
# band — same convention as /etc/ts-auth-key — and is not in git:
|
||||
#
|
||||
# /etc/nginx/oc-htpasswd root:nginx 0640, `oc:<bcrypt hash>`
|
||||
# /secrets/files/oc-basic-auth.txt the plaintext, loops-only 0600
|
||||
#
|
||||
# Rotate with:
|
||||
# P=$(python3 -c "import secrets,string;a=string.ascii_letters+string.digits;print(''.join(secrets.choice(a) for _ in range(32)))")
|
||||
# printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" | sudo tee /etc/nginx/oc-htpasswd
|
||||
# sudo chown root:nginx /etc/nginx/oc-htpasswd && sudo chmod 640 /etc/nginx/oc-htpasswd
|
||||
# sudo systemctl reload nginx
|
||||
#
|
||||
# NOTE: if this file goes missing, nginx fails to START — which would take
|
||||
# the atproto vhost down with it. Recreate it before rebuilding on a fresh
|
||||
# host.
|
||||
basicAuthFile = "/etc/nginx/oc-htpasswd";
|
||||
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:4096";
|
||||
proxyWebsockets = true;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
|
||||
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
|
||||
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
|
||||
# Control session + watchdog + pipeline via `engine/agents.py up`. Added 2026-07-08 after a reboot
|
||||
# (Hetzner rollback) left this orchestrator down while cc-ci-loops auto-recovered.
|
||||
# NOTE: still points at the /home path — will be re-pointed to /srv when that migration happens.
|
||||
systemd.services.lichen-orchestrator = {
|
||||
description = "p-lichen-orchestrator (lichen.page testing) — orchestrator + watchdog + pipeline";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/lichen-orchestrator";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/lichen-orchestrator"
|
||||
# boot marker (best-effort; boot_id-gated logging can be added later)
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — lichen-orchestrator up" \
|
||||
>> "$proj/.ao-state/boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 engine/agents.py up || echo "p-lichen agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
# project-orchestrator (fleet manager) — always-on so the operator can reach it over Remote
|
||||
# Control at any time (operator 2026-08-01). Same shape as lichen-orchestrator above; the PO's
|
||||
# own agents.toml declares NO `wake`, so the watchdog only heals a dead session — it never sends
|
||||
# periodic prompts. Starting it is `agents.py up`; that also starts its watchdog.
|
||||
systemd.services.project-orchestrator = {
|
||||
description = "project-orchestrator (fleet manager) — PO agent + watchdog, remote-control always up";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/project-orchestrator";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/project-orchestrator"
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — project-orchestrator up" \
|
||||
>> "$proj/.ao-state/boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 engine/agents.py up || echo "PO agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
|
||||
|
||||
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{ modulesPath, ... }:
|
||||
{
|
||||
imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
|
||||
boot.loader = {
|
||||
efi.efiSysMountPoint = "/boot/efi";
|
||||
grub = {
|
||||
efiSupport = true;
|
||||
efiInstallAsRemovable = true;
|
||||
device = "nodev";
|
||||
};
|
||||
};
|
||||
fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/39A5-C7B9"; fsType = "vfat"; };
|
||||
boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
|
||||
boot.initrd.kernelModules = [ "nvme" ];
|
||||
fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
|
||||
|
||||
# 150G Hetzner volume (scsi-0HC_Volume_106342723) — holds the Rust build trees, which repeatedly
|
||||
# filled the 75G root. A full disk here does NOT error: it silently truncates whatever is being
|
||||
# written (it destroyed a plan file mid-write before anyone noticed).
|
||||
#
|
||||
# by-uuid, not /dev/sdb: device names are not stable across reboots and attaching another volume
|
||||
# would silently mount the wrong disk here.
|
||||
#
|
||||
# nofail is REQUIRED, not decoration: without it, a detached/failed volume makes the mount unit a
|
||||
# boot dependency, systemd drops to emergency mode, and the box comes up WITHOUT SSH. That failure
|
||||
# has already happened once on this host from a bad nixos config, and it needed a recovery to the
|
||||
# previous generation. The build cache is not worth risking access to the machine.
|
||||
fileSystems."/mnt/data" = {
|
||||
device = "/dev/disk/by-uuid/16cd6650-1399-4cb9-a696-54b70d83203a";
|
||||
fsType = "ext4";
|
||||
options = [ "defaults" "nofail" "x-systemd.device-timeout=10s" ];
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
# ⚠️ HISTORICAL / DEAD CONFIG — DO NOT DEPLOY. (superseded 2026-05-31)
|
||||
# This is the OLD Incus VM (`cc-ci-orchestrator`, tailnet 100.116.55.106) config. The orchestrator
|
||||
# now runs on Hetzner — the LIVE config is nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix.
|
||||
# Kept only for historical comparison; it runs loops as root and hard-codes the dead cc-ci IP.
|
||||
# Delete once the old Incus VM is torn down.
|
||||
#
|
||||
# cc-ci-orchestrator VM — NixOS config (channel-based: nixos-24.11; deployed to /etc/nixos/configuration.nix)
|
||||
#
|
||||
# Purpose: a reboot-resilient host for the cc-ci Builder/Adversary loops + watchdog + the orchestrator
|
||||
# session, moved off the unstable 905 MiB Pi. See plan-orchestrator-migration.md.
|
||||
#
|
||||
# STATUS: DRAFT (Phase B). The nix-ld + claude-install bits need on-VM validation (the standalone
|
||||
# Claude Code is a Bun ELF binary; NixOS needs nix-ld to run a foreign dynamic binary). The
|
||||
# cc-ci-loops supervisor service is defined but NOT enabled until the workspace is staged (Phase C/D).
|
||||
{ config, pkgs, lib, modulesPath, ... }:
|
||||
{
|
||||
imports = [
|
||||
"${modulesPath}/virtualisation/incus-virtual-machine.nix"
|
||||
];
|
||||
|
||||
# --- base (mirrors the incus-base-vm) ---
|
||||
virtualisation.incus.agent.enable = true; # for `incus exec`
|
||||
services.cloud-init = { enable = true; network.enable = true; };
|
||||
services.openssh = { enable = true; settings.PermitRootLogin = "yes"; };
|
||||
networking.useDHCP = true;
|
||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||
networking.firewall = { enable = true; trustedInterfaces = [ "tailscale0" ]; allowedTCPPorts = [ 22 ]; };
|
||||
nix.settings.experimental-features = [ "nix-command" "flakes" ];
|
||||
system.stateVersion = "24.11";
|
||||
|
||||
# --- tailscale (auto-auth from /etc/ts-auth-key, hostname from /etc/ts-hostname; written by cloud-init) ---
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
authKeyFile = "/etc/ts-auth-key";
|
||||
extraUpFlags = let h = lib.strings.removeSuffix "\n" (builtins.readFile /etc/ts-hostname);
|
||||
in [ "--hostname=${h}" "--ssh" ]; # --ssh: allow tailscale-ssh as a fallback path
|
||||
};
|
||||
|
||||
# --- swap: the Pi OOM lesson. 2 GB RAM is tight for 3 concurrent claude sessions; 4 GB disk swap
|
||||
# as a real overflow tier (zram is in-RAM and doesn't add capacity). ---
|
||||
swapDevices = [ { device = "/swapfile"; size = 4096; } ];
|
||||
|
||||
# --- nix-ld: lets the standalone Claude Code (foreign dynamic ELF / Bun) run on NixOS ---
|
||||
programs.nix-ld.enable = true;
|
||||
programs.nix-ld.libraries = with pkgs; [
|
||||
stdenv.cc.cc.lib # libstdc++ / libgcc_s
|
||||
zlib
|
||||
openssl
|
||||
curl
|
||||
glibc
|
||||
];
|
||||
|
||||
# --- packages the loops + launch.sh + orchestrator need ---
|
||||
environment.systemPackages = with pkgs; [
|
||||
git tmux python3 jq curl cacert
|
||||
gnused gawk coreutils gnugrep findutils util-linux
|
||||
nettools openssh # nc, ssh
|
||||
docker-client # `docker` CLI is not needed (deploys run on cc-ci), but handy for probes
|
||||
];
|
||||
|
||||
# --- loops user: non-root account for running claude (--dangerously-skip-permissions blocked for root) ---
|
||||
users.users.loops = {
|
||||
isNormalUser = true;
|
||||
home = "/home/loops";
|
||||
shell = pkgs.bash;
|
||||
extraGroups = [ "wheel" ]; # sudo access
|
||||
};
|
||||
security.sudo.wheelNeedsPassword = false; # passwordless sudo for wheel
|
||||
# Allow loops user to use tmux/claude without a password prompt
|
||||
security.sudo.extraRules = [{
|
||||
users = [ "loops" ];
|
||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
||||
}];
|
||||
|
||||
# --- root PATH: ensure ~/.local/bin (where the standalone claude binary lives) is on root's PATH ---
|
||||
environment.variables.PATH = lib.mkForce "/root/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
||||
|
||||
# --- root ssh config: reach cc-ci DIRECTLY over the VM's own tailscale (this VM is a tailnet peer,
|
||||
# so NO SOCKS proxy is needed — unlike the Pi). Key staged at /root/.ssh/cc-ci-root-ed25519. ---
|
||||
system.activationScripts.ccciSshConfig = ''
|
||||
mkdir -p /root/.ssh && chmod 700 /root/.ssh
|
||||
cat > /root/.ssh/config <<'SSHCFG'
|
||||
Host cc-ci cc-nix-test 100.90.116.4
|
||||
HostName 100.90.116.4
|
||||
User root
|
||||
IdentityFile /root/.ssh/cc-ci-root-ed25519
|
||||
IdentitiesOnly yes
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 30
|
||||
SSHCFG
|
||||
chmod 600 /root/.ssh/config
|
||||
'';
|
||||
|
||||
# --- claude-install: idempotent oneshot — fetch the standalone Claude Code CLI if missing.
|
||||
# Runs via nix-ld. (Auth is a one-time operator step: `claude auth login` — see migration plan.) ---
|
||||
systemd.services.claude-install = {
|
||||
description = "Install the standalone Claude Code CLI if missing (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "nix-ld.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
|
||||
script = ''
|
||||
if [ ! -x /root/.local/bin/claude ]; then
|
||||
echo "installing standalone Claude Code CLI..."
|
||||
curl -fsSL https://claude.ai/install.sh | bash || echo "claude install failed (retry next activation)"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
# --- cc-ci-loops supervisor (DEFINED, NOT YET ENABLED). Enabled in Phase D after the workspace
|
||||
# (/srv/cc-ci: launch.sh, plan, prompts, the loop clones, secrets) is staged. This is the
|
||||
# reboot-resilience fix: it runs launch.sh start on every boot. Mirrors the Pi's cc-ci-loops.service. ---
|
||||
systemd.services.cc-ci-loops = {
|
||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start, RESUME_PHASE)";
|
||||
wantedBy = [ "multi-user.target" ]; # enabled: workspace staged (Phase C/D 2026-05-30)
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; RemainAfterExit = true; User = "root";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
};
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/root"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||
echo "workspace not staged yet (/srv/cc-ci/cc-ci-plan/launch.sh missing) — skipping"
|
||||
'';
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
# Plan — migrate the cc-ci SERVER from b1 Incus to Hetzner (full cutover)
|
||||
|
||||
**Status:** PROPOSED. Move the cc-ci **CI server** (`cc-nix-test`) off the slow b1 host onto a fast
|
||||
Hetzner **cpx32** (8 GB, dedicated vCPU, NVMe), repoint the Builder/Adversary loops + everything at it,
|
||||
then stop the old VM. **This file:** `/srv/cc-ci/cc-ci-plan/plan-cc-ci-hetzner-migration.md`.
|
||||
**Owner:** assistant (provisioning + cutover mechanics) + orchestrator (coordination); operator for the
|
||||
secret/DNS gates. **Supersedes** the narrower `plan-cc-ci-hetzner-terraform.md` (that is Phase 1's
|
||||
deliverable; this plan wraps it with the cutover + decommission).
|
||||
|
||||
---
|
||||
|
||||
## 0. Context (why, and what's where)
|
||||
- **Two VMs run on b1** (a 2015 **Intel i5-6400T low-power CPU + a spinning HDD** — measured: CPU
|
||||
pressure ~55%, root disk `ROTA=1`):
|
||||
- **cc-ci server** `cc-nix-test` (tailnet `100.90.116.4`, 8 GB) — where the loops deploy recipes +
|
||||
run the harness (the heavy CI work). **This is what we migrate.**
|
||||
- **orchestrator VM** `cc-ci-orchestrator` (tailnet `100.116.55.106`, 2 GB) — where the loops +
|
||||
orchestrator + assistant *run* (claude sessions). Stays for now.
|
||||
- b1 is overloaded running both on a slow CPU + HDD — "everything is getting slow."
|
||||
- **The win (see the perf analysis):** Hetzner cpx32 = modern dedicated vCPU + **NVMe** vs a 2015
|
||||
low-power CPU + **HDD** → I/O-bound deploys (the ghost/discourse near-timeouts) likely **3–10×**
|
||||
faster, CPU work **~2–3×**. Moving the *heavy* server off b1 also relieves b1, so the orchestrator VM
|
||||
(still there) speeds up too.
|
||||
|
||||
## 1. Phase 1 — provision the Hetzner cc-ci, FULLY ready
|
||||
The `plan-cc-ci-hetzner-terraform.md` deliverable, taken all the way to a **converged, green** server
|
||||
(not just "terraform applies"):
|
||||
- `terraform/` in the cc-ci repo (cpx32, ubuntu-24.04, pinned hcloud provider + nixos-infect). `apply`
|
||||
→ nixos-infect → bare NixOS on Hetzner.
|
||||
- Add the `cc-ci-hetzner` flake host (nixos-infect's DO/Hetzner hardware + the shared `nix/modules/*`).
|
||||
- **Full convergence (the D8 flow):** clone cc-ci `--recursive` + place the **bootstrap age key** at
|
||||
`/var/lib/sops-nix/key.txt` (operator) + `nixos-rebuild switch --flake .#cc-ci-hetzner` → traefik /
|
||||
drone / bridge / dashboard / backupbot / swarm all up, **0 failed units**.
|
||||
- **DNS/cert:** point `ci.commoninternet.net` + `*.ci` **A record at the Hetzner public IP** (the
|
||||
server has one — can drop the b1 TLS-passthrough gateway). Keep the sops wildcard cert for v1
|
||||
(or ACME — §decision).
|
||||
- **Readiness gate (before any cutover):** ssh works; the dashboard + `*.ci.commoninternet.net` are
|
||||
reachable; a **full `!testme` runs GREEN on the Hetzner server** (drive one recipe end-to-end via
|
||||
the harness). Keep the b1 cc-ci running untouched in parallel during all of Phase 1.
|
||||
- **Operator inputs for Phase 1:** `HCLOUD_TOKEN` (have), `TS_AUTH_KEY` (have), the **bootstrap age
|
||||
key** (needed for convergence), and the **DNS change**. Note: the token may be invalidated after the
|
||||
KEEPER server is applied — the server runs without it; only future `terraform` needs a (new) token.
|
||||
|
||||
## 2. Phase 2 — cut everything over to the Hetzner server
|
||||
Once Phase 1 is green, switch all consumers from the b1 `cc-nix-test` to the Hetzner server:
|
||||
- **Loop access:** update the `Host cc-ci` entry in the loops' ssh config (on the orchestrator VM,
|
||||
used by builder/adversary/orchestrator/assistant) — `HostName` from `100.90.116.4` →
|
||||
the **Hetzner server's tailnet IP / MagicDNS**. (`ssh cc-ci` is the single indirection the loops
|
||||
use, so this one change repoints all of them. The Hetzner box joins the SAME tailnet via
|
||||
`TS_AUTH_KEY`, so it's a direct peer like today.)
|
||||
- **CI flow:** the `!testme` → bridge → Drone → harness path + the dashboard now run on the Hetzner
|
||||
server (they're part of the converged config there). The recipe mirrors stay on Gitea (unaffected).
|
||||
- **State carry-over (minimal — mostly stateless):** recipes redeploy from the mirrors; **warm
|
||||
canonicals re-seed** on the first green cold runs; the harness lives in the cc-ci repo. Drone build
|
||||
history + dashboard state start **fresh** on the new server (acceptable; migrate only if wanted).
|
||||
- **Verify cutover:** a full loop cycle works against Hetzner — Builder deploys + claims a gate, the
|
||||
Adversary **cold-verifies green** on the Hetzner server; phase-2 recipe work continues, now fast.
|
||||
Watch a ghost/discourse deploy to confirm the timeouts are gone.
|
||||
|
||||
## 3. Phase 3 — stop the old cc-ci VM (free b1)
|
||||
- Once everything is confirmed serving green on Hetzner, **stop `cc-nix-test` on b1** (Incus
|
||||
`PUT .../state {"action":"stop"}`). **Keep it as a cold standby for a few days** (don't delete) for
|
||||
rollback, then retire.
|
||||
- b1 now runs only the small orchestrator VM → it gets b1's full (modest) resources → the loops'
|
||||
*runtime* is less starved too. "Everything faster from here on out."
|
||||
- **Rollback (until the old VM is deleted):** if Hetzner has a problem, revert the `Host cc-ci` ssh
|
||||
entry to `100.90.116.4` and start the b1 VM again.
|
||||
|
||||
## 4. Sequencing & gates (don't break the running CI)
|
||||
- **Strictly parallel bring-up:** Phase 1 stands Hetzner up *alongside* the live b1 cc-ci; **no
|
||||
consumer is repointed until the Hetzner `!testme` is green** (Phase 1 readiness gate).
|
||||
- The cutover (Phase 2) is a **single ssh-config repoint** + DNS — fast and reversible.
|
||||
- Phase 3 (stop b1) only after Phase 2 is verified.
|
||||
- The loops keep working on b1 throughout Phase 1 (no disruption); the brief cutover window is the
|
||||
only moment they switch servers.
|
||||
|
||||
## 5. Open decisions (log in DECISIONS.md)
|
||||
- **DNS/cert:** point `*.ci` at the Hetzner public IP + drop the gateway; sops cert (v1) vs ACME.
|
||||
- **Drone/dashboard history:** fresh on Hetzner (default) vs migrate the volumes.
|
||||
- **Orchestrator VM:** leave on b1 (freed) for now; a *later, separate* plan could also move the loops'
|
||||
runtime to Hetzner and fully retire b1 — out of scope here (the runtime needn't be fast).
|
||||
- **Token lifecycle:** invalidate `HCLOUD_TOKEN` after the keeper apply, or keep a (rotated) one for
|
||||
ongoing `terraform` management of the server.
|
||||
|
||||
## 6. Definition of Done
|
||||
- Hetzner cpx32 cc-ci fully converged (0 failed units) + a **green `!testme`** on it.
|
||||
- Loops + dashboard + `*.ci.commoninternet.net` all served from Hetzner; a full Builder→Adversary
|
||||
cycle verified green there; deploy/convergence visibly faster (ghost/discourse no longer near-timeout).
|
||||
- Old b1 `cc-nix-test` **stopped** (cold standby, not deleted).
|
||||
- `terraform/` committed to the cc-ci repo (via PR); no secrets/state in git; `docs/install.md`
|
||||
updated for the Hetzner host. Adversary-verifiable: from-scratch reproducibility holds on Hetzner.
|
||||
|
||||
## 7. Guardrails
|
||||
- Parallel bring-up; never repoint consumers until Hetzner is green; keep b1 as cold standby.
|
||||
- No secrets in git (token, TS key, age key, tfstate). Pin everything. x86 only (cpx32/cx32).
|
||||
- Real Nix provisioning (the flake) + real abra; don't weaken anything to make the new server "pass."
|
||||
@@ -0,0 +1,133 @@
|
||||
# Plan — cc-ci on Hetzner Cloud: `terraform/` + nixos-infect + Nix provisioning
|
||||
|
||||
**Status:** PROPOSED → handed to the assistant to implement. Add a **`terraform/`** folder to the
|
||||
**cc-ci product repo** (`recipe-maintainers/cc-ci`) that provisions the cc-ci server on **Hetzner
|
||||
Cloud** (8 GB server), converts it to NixOS via **nixos-infect**, then applies the existing cc-ci
|
||||
flake config — making the CI server reproducible-from-scratch on real cloud hosting.
|
||||
**This file:** `/srv/cc-ci/cc-ci-plan/plan-cc-ci-hetzner-terraform.md`.
|
||||
|
||||
**Token (operator, 2026-05-31):** an `HCLOUD_TOKEN` with **read/write to an isolated Hetzner project**
|
||||
(just for this) is in `/srv/cc-ci/.testenv`. The operator **will invalidate it** once the terraform is
|
||||
verified working — so the goal is **write + apply + verify the working terraform**, then report.
|
||||
|
||||
---
|
||||
|
||||
## 0. Why
|
||||
cc-ci currently runs as the Incus VM `cc-nix-test` on b1 — a small, shared 4-core host (the contention
|
||||
we kept hitting). A dedicated Hetzner **8 GB** server gives standard, reliable hosting with a **public
|
||||
IP**, fully reproducible via Terraform + the existing cc-ci NixOS flake. "Spin up cc-ci from nothing"
|
||||
becomes a `terraform apply`.
|
||||
|
||||
## 1. What already exists — build ON this, don't reinvent
|
||||
- cc-ci is a **flake-based NixOS system**: `flake.nix` → `nixosConfigurations.cc-ci` (pinned nixpkgs
|
||||
24.11, **`system = "x86_64-linux"`**) → `nix/hosts/cc-ci/{configuration.nix, hardware.nix}` +
|
||||
`nix/modules/*` (proxy/traefik, drone, drone-runner, bridge, dashboard, backupbot, swarm, abra,
|
||||
harness, warm-keycloak, secrets).
|
||||
- **From-scratch install is already VERIFIED (D8, `docs/install.md`):** a blank NixOS host + the two
|
||||
repos (cc-ci cloned `--recursive` so the `cc-ci-secrets` submodule at `secrets/` comes too) + the
|
||||
**one bootstrap age key** at `/var/lib/sops-nix/key.txt` → a single `nixos-rebuild switch` converges
|
||||
the whole server (0 failed units; serialized reconcile oneshots). The wildcard TLS cert + all secrets
|
||||
are **sops-encrypted in `cc-ci-secrets`** (not out-of-band).
|
||||
- So **"provision via Nix in the expected way" = that exact D8 flow:** clone `--recursive` + bootstrap
|
||||
age key + `nixos-rebuild switch --flake .#<host>`.
|
||||
- The current `nix/hosts/cc-ci/hardware.nix` is **Incus-VM-specific** — Hetzner needs its own
|
||||
hardware/bootloader/networking, which **nixos-infect generates**.
|
||||
|
||||
## 2. `terraform/` layout (in `recipe-maintainers/cc-ci`)
|
||||
```
|
||||
terraform/
|
||||
versions.tf # terraform + hetznercloud/hcloud provider, pinned
|
||||
variables.tf # hcloud_token(sensitive), location, server_type, image, ssh_key, ts_auth_key(sensitive), hostname
|
||||
main.tf # hcloud_ssh_key + hcloud_server + user_data
|
||||
outputs.tf # server ipv4, id
|
||||
user-data.sh # cloud-init stage-1: run nixos-infect (pinned)
|
||||
README.md # apply instructions + operator inputs
|
||||
.gitignore # *.tfstate*, *.auto.tfvars, .terraform/ (NEVER commit secrets/state)
|
||||
```
|
||||
- **Provider:** `hetznercloud/hcloud` (pinned in `versions.tf`). The token comes from
|
||||
**`HCLOUD_TOKEN`** (env, read by the provider) or `TF_VAR_hcloud_token` — it's in `.testenv`; do
|
||||
NOT hardcode/commit it.
|
||||
- **Server:** `hcloud_server` — type **`cpx32`** (AMD **dedicated vCPU**, **8 GB RAM**, NVMe SSD) —
|
||||
**DEFAULT** (operator 2026-05-31: `cpx31` is **retired**; `cpx32` is the current dedicated-vCPU 8 GB
|
||||
type). Dedicated vCPU avoids noisy-neighbor variance for bursty CI. Must be **x86** (the flake is
|
||||
`x86_64-linux`; do **NOT** use the `cax*` ARM types). `cx32` (Intel shared vCPU, 8 GB) is a cheaper
|
||||
alt. Confirm exact specs from the hcloud API at apply time. `image = "ubuntu-24.04"` (nixos-infect-supported base), a `location`
|
||||
(e.g. `nbg1`/`fsn1`/`hel1` EU or `ash`/`hil` US — pick one, make it a var), `ssh_keys=[hcloud_ssh_key.id]`,
|
||||
`user_data=file("user-data.sh")`, `public_net { ipv4_enabled = true }`, a stable name + label.
|
||||
- Keep the token + TS key **sensitive**; `terraform.tfstate` **gitignored** (can hold secrets) — mirrors
|
||||
cc-ci's no-secrets-in-git rule.
|
||||
|
||||
## 3. Stage 1 — nixos-infect (base Ubuntu → NixOS)
|
||||
`user-data.sh` on first boot:
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export NIX_CHANNEL=nixos-24.11
|
||||
export PROVIDER=hetzner # nixos-infect provider hint (Hetzner Cloud is supported)
|
||||
curl -fsSL https://raw.githubusercontent.com/elitak/nixos-infect/<PINNED_SHA>/nixos-infect | bash -x
|
||||
```
|
||||
nixos-infect converts the server to NixOS in place, generates `/etc/nixos/{configuration.nix,
|
||||
hardware-configuration.nix, networking.nix}` (Hetzner-correct bootloader + public-IP networking), and
|
||||
reboots into NixOS. **Pin the nixos-infect revision** — don't `curl|bash` master blind. After this the
|
||||
server is **bare NixOS on Hetzner**, ssh-able as root.
|
||||
|
||||
## 4. Stage 2 — provision via Nix (bare NixOS → converged cc-ci) — "the expected way"
|
||||
1. **Capture Hetzner hardware into the flake.** Take the `hardware-configuration.nix` + `networking.nix`
|
||||
nixos-infect generated and add them as a flake host. **Cleaner: a new host `nix/hosts/cc-ci-hetzner/`**
|
||||
importing the shared `nix/modules/*` + the Hetzner hardware, with `nixosConfigurations.cc-ci-hetzner`
|
||||
in `flake.nix` (keeps the Incus `cc-ci` host buildable during transition). Make Hetzner the canonical
|
||||
`cc-ci` after cutover.
|
||||
2. **Run the D8 install flow on the server:** clone `recipe-maintainers/cc-ci` `--recursive` (brings
|
||||
`cc-ci-secrets`), provision the **bootstrap age key** at `/var/lib/sops-nix/key.txt`, then
|
||||
`nixos-rebuild switch --flake .#cc-ci-hetzner`. The reconcile oneshots converge the swarm.
|
||||
3. **Where stage 2 runs:** **v1 = documented step run after `terraform apply`** (Terraform provisions +
|
||||
infects; the age-key placement + `nixos-rebuild` is the explicit step, like `docs/install.md`).
|
||||
Automate later via a Terraform `remote-exec` provisioner once key-delivery is settled.
|
||||
- **Note on secrets for verification:** full cc-ci convergence needs the bootstrap age key (decrypts
|
||||
`cc-ci-secrets`). If that key isn't available to the implementer, verify as far as possible —
|
||||
`terraform apply` → nixos-infect → bare NixOS → the flake **builds/evaluates** for the Hetzner host
|
||||
(`nixos-rebuild build --flake .#cc-ci-hetzner`) — and flag the age-key step as operator-pending.
|
||||
|
||||
## 5. Operator inputs (class-A1 — provide at apply, NEVER commit)
|
||||
- **`HCLOUD_TOKEN`** — already in `.testenv` (isolated project, read/write; operator will invalidate
|
||||
after). The provider reads it from env.
|
||||
- **SSH key** — register a public key as `hcloud_ssh_key`; hold the private half to ssh + run stage 2.
|
||||
- **`TS_AUTH_KEY`** — tailnet join (cc-ci enables tailscale; the server joins the same tailnet so the
|
||||
orchestrator/loops reach it as today, direct peer). Already in `.testenv`.
|
||||
- **Bootstrap age key** → `/var/lib/sops-nix/key.txt` (decrypts `cc-ci-secrets` incl. the TLS cert).
|
||||
The single out-of-band secret per `docs/install.md`.
|
||||
|
||||
## 6. DNS / gateway — a simplification the public IP enables (open decision)
|
||||
Today `*.ci.commoninternet.net` reaches the Incus VM (no public IP) via an external nginx
|
||||
TLS-passthrough gateway → MagicDNS. A Hetzner server has a **public IP**, so point
|
||||
`ci.commoninternet.net` + the `*.ci` wildcard **A record straight at the server** and **drop the
|
||||
gateway** — Traefik terminates TLS directly. The sops wildcard cert still works as-is; or switch
|
||||
Traefik to **ACME** and retire the manual cert + renewal. **v1: keep the sops cert (no behavior
|
||||
change); evaluate ACME-on-public-IP as a follow-up.** Record in DECISIONS.md.
|
||||
|
||||
## 7. Open decisions (log in DECISIONS.md)
|
||||
- **Replace vs. parallel:** stand Hetzner up **in parallel**, verify a full `!testme` + the D-gates
|
||||
green, then cut DNS over and **retire the Incus `cc-nix-test`**. Nothing stateful is lost — recipes
|
||||
redeploy, warm canonicals re-seed on first green runs.
|
||||
- **Flake host:** parallel `cc-ci-hetzner` host until cutover, then make Hetzner the canonical `cc-ci`.
|
||||
- **Server type/location** (cx32 vs cpx31; region); **ACME vs sops cert** (§6); **stage-2 automation** (§4.3).
|
||||
|
||||
## 8. Definition of Done
|
||||
- `terraform/` in the cc-ci repo; `terraform apply` (with `HCLOUD_TOKEN`) creates an **8 GB cx32**
|
||||
Hetzner server and nixos-infect converts it to NixOS.
|
||||
- The flake **builds for the Hetzner host** (`nixos-rebuild build --flake .#cc-ci-hetzner`); given the
|
||||
bootstrap age key it **switches** to a fully converged cc-ci (the D8 flow) — 0 failed units.
|
||||
- (Once secrets available) a real recipe `!testme` runs **green** on the Hetzner cc-ci; dashboard +
|
||||
`*.ci.commoninternet.net` reachable via the chosen DNS path.
|
||||
- `terraform/README.md` documents apply + operator inputs; **no secrets/state committed**.
|
||||
- The terraform is proven **idempotent** (`terraform plan` clean after apply); test resources cleaned
|
||||
up (`terraform destroy`) if this is a throwaway verification rather than the real cutover.
|
||||
|
||||
## 9. Guardrails
|
||||
- **No secrets in git** (HCLOUD_TOKEN, TS key, age key, tfstate all out-of-band/gitignored) — cc-ci's rule.
|
||||
- **Pin everything** (hcloud provider, nixos-infect rev; nixpkgs already pinned) — reproducible, no drift.
|
||||
- **x86 only** — the flake is `x86_64-linux`; use `cx32`/`cpx31`, never `cax*` (ARM).
|
||||
- **Don't break the running Incus cc-ci** until the Hetzner one is verified green (parallel + cutover).
|
||||
- **Real Nix provisioning** (the flake), not hand-installed packages.
|
||||
- **The token is invalidatable + isolated** — but still treat it as a live secret: never commit it,
|
||||
never echo it into logs.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Plan — migrate the cc-ci SERVER from the Incus VM to Hetzner (provision → benchmark → cutover → retire)
|
||||
|
||||
**Status:** PROPOSED. Move the **cc-ci CI server** off the Incus VM `cc-nix-test` (b1: 2015 i5-6400T
|
||||
+ **spinning HDD**, CPU-pressure ~55%, getting very slow) onto a **Hetzner `cpx32`** (4 vCPU / 8 GB /
|
||||
160 GB **NVMe**, x86, ~€16.49/mo). Everything (Builder, Adversary, the !testme pipeline) then targets
|
||||
the fast new server. **This file:** `/srv/cc-ci/cc-ci-plan/plan-migrate-cc-ci-to-hetzner.md`.
|
||||
|
||||
**Key enabler (verified 2026-05-31):** the bootstrap age key is **already on this VM** at
|
||||
`/srv/cc-ci/.sops/master-age.txt` and the `cc-ci-secrets` submodule is populated — so the new server
|
||||
can be **fully provisioned end-to-end with NO operator secret-blocker** (the D8 flow decrypts the TLS
|
||||
cert + all secrets). The Pi is not needed.
|
||||
|
||||
**Architecture reminder:** the Builder/Adversary **loops run on this orchestrator VM** and reach the
|
||||
CI server via `ssh cc-ci`; the **!testme pipeline (Gitea webhook → bridge → Drone → harness) runs ON
|
||||
the cc-ci server**, and `*.ci.commoninternet.net` + the dashboard are served from it. "Switch
|
||||
everything to the new server" = make the Hetzner box the cc-ci, then repoint `ssh cc-ci`, the
|
||||
webhook/DNS, and the dashboard at it. The loops' code/clones don't move — only their target.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Provision the new Hetzner cc-ci, fully converged (assistant)
|
||||
Per **`plan-cc-ci-hetzner-terraform.md`** (the provisioning detail): `terraform/` in the cc-ci repo →
|
||||
`hcloud` `cpx32` from `ubuntu-24.04` → **pinned nixos-infect** → bare NixOS → add the **`cc-ci-hetzner`
|
||||
flake host** (the nixos-infect-generated DO/Hetzner hardware + the shared `nix/modules/*`) → run the
|
||||
**D8 flow**: clone `--recursive`, place `/srv/cc-ci/.sops/master-age.txt` at `/var/lib/sops-nix/key.txt`,
|
||||
`nixos-rebuild switch --flake .#cc-ci-hetzner`. The server joins the tailnet (TS_AUTH_KEY).
|
||||
- **Accept:** 0 failed units; traefik/drone/bridge/dashboard/backupbot up; the box is on the tailnet
|
||||
and ssh-able; terraform is idempotent (`plan` clean). This is a **real** server we keep (not the
|
||||
throwaway the terraform-plan first described) — do **not** `terraform destroy` once it converges.
|
||||
- Done in **parallel** — the old Incus cc-ci keeps serving the loops until Phase 3.
|
||||
|
||||
## Phase 2 — Benchmark: old vs new, two recipes (a short report)
|
||||
Pick **two representative recipes** — one light (e.g. `n8n` or `custom-html`) and one heavy/slow (e.g.
|
||||
`ghost` or `discourse` — the HDD-bound timeout cases). Run the **same full harness** (cold,
|
||||
install+upgrade+backup+restore+custom) on **both servers**:
|
||||
- old: `ssh cc-ci-incus` (the current `cc-nix-test`), new: `ssh cc-ci-hetzner`.
|
||||
- Capture **per-tier + total wall-clock** from the `RUN SUMMARY` for each recipe on each host.
|
||||
Write a short comparison report → **`docs/perf/hetzner-vs-incus.md`** in the cc-ci repo (table: recipe
|
||||
× tier × old-time × new-time × speedup). This empirically confirms the expected ~2–4× (more on the
|
||||
I/O-bound phases). *(Run identical conditions — same recipe versions, cold cache both sides.)*
|
||||
|
||||
## Phase 3 — Cutover: point everything at the new server (orchestrated; pick a quiet moment)
|
||||
1. **Quiesce briefly:** ensure no live `!testme`/deploy is mid-run on the old server.
|
||||
2. **Repoint the loops' `ssh cc-ci`** → the Hetzner box's tailnet IP: update `Host cc-ci` in
|
||||
`/home/loops/.ssh/config` (and root's) `HostName` → new IP. The loops keep working from this VM;
|
||||
only their target changes. (Keep a `Host cc-ci-incus` alias for the old box during the overlap.)
|
||||
3. **DNS / webhook / gateway:** point `ci.commoninternet.net` + the `*.ci` wildcard **A record at the
|
||||
Hetzner public IP** (drop the TLS-passthrough gateway — Traefik on the droplet terminates directly;
|
||||
the sops wildcard cert works as-is). Re-point the Gitea `issue_comment` webhook → the new server so
|
||||
`!testme` triggers there. **DNS is operator-owned (`commoninternet.net`)** — the one operator step.
|
||||
4. **Verify end-to-end on the new server:** a real PR `!testme` runs green through the new
|
||||
bridge→Drone→harness; the dashboard + `*.ci.commoninternet.net` load; the loops' `ssh cc-ci` deploys
|
||||
land on Hetzner. Re-run the relevant D-gates cold-verified by the Adversary.
|
||||
5. Make `cc-ci-hetzner` the **canonical** `nixosConfigurations.cc-ci` in the flake (retire the Incus
|
||||
`hardware.nix` once the old box is gone).
|
||||
|
||||
## Phase 4 — Retire the old Incus cc-nix-test
|
||||
Once Hetzner is the verified live cc-ci: **stop** the Incus VM via the b1 Incus API (mTLS certs are on
|
||||
this VM under `incus-terraform-nix-vm-creator/terraform-secrets/`) — `PUT .../instances/cc-nix-test/
|
||||
state {"action":"stop"}`. Keep it as a **cold standby for a few days**, then delete (frees b1). Update
|
||||
the memory/docs ([[cc-ci-setup]]) to point cc-ci at Hetzner.
|
||||
|
||||
## Who does what
|
||||
- **Assistant:** Phase 1 (the terraform + full convergence) and the Phase-2 benchmark runs.
|
||||
- **Orchestrator (me) + operator:** Phase 3 cutover (I do the ssh-repoint + the Incus stop via the
|
||||
API; **operator does the DNS change** + the go/no-go) and Phase 4.
|
||||
|
||||
## Guardrails
|
||||
- **Parallel bring-up** — never break the running Incus cc-ci until Hetzner is verified green; the
|
||||
cutover is the only switch moment, at a quiet point.
|
||||
- **No secrets in git** — `HCLOUD_TOKEN`, TS key, the age key (`.sops/`), tfstate all gitignored
|
||||
(`.gitignore` hardened for `*age*.txt`/`.sops/`); never echo/commit them.
|
||||
- **x86 `cpx32`**, pin the hcloud provider + nixos-infect rev (nixpkgs already pinned).
|
||||
- **Reproducible-from-scratch holds** (the D8 guarantee) — the Hetzner cc-ci comes from `terraform
|
||||
apply` + one `nixos-rebuild switch`, no hand steps beyond the operator DNS + age key.
|
||||
|
||||
## Definition of Done
|
||||
- Hetzner `cpx32` cc-ci fully converged (0 failed units) via terraform + the D8 flake flow.
|
||||
- `docs/perf/hetzner-vs-incus.md` shows the two-recipe old-vs-new comparison (real numbers).
|
||||
- The loops, `!testme` pipeline, dashboard, and `*.ci.commoninternet.net` all run on Hetzner; a PR
|
||||
`!testme` is green end-to-end there; D-gates re-verified.
|
||||
- The Incus `cc-nix-test` is stopped (cold standby → deletion); the flake's canonical `cc-ci` host is
|
||||
Hetzner; docs/memory updated.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Plan/record — migrate the ORCHESTRATOR off the Incus VM onto a Hetzner cloud server
|
||||
|
||||
**Status:** COMPLETE (2026-05-31). The orchestrator (Builder/Adversary loops + watchdog + this
|
||||
supervising session) now runs on a dedicated **Hetzner** cloud server, declared by the
|
||||
`cc-ci-orchestrator-hetzner` flake host. Kept as a historical record.
|
||||
|
||||
**Why:** the previous orchestrator host was the Incus VM `cc-ci-orchestrator` on b1
|
||||
(`100.116.55.106`, 2 GB / 2 vCPU, see [[plan-orchestrator-migration]] — the earlier Pi→Incus move).
|
||||
A dedicated Hetzner box gives dedicated vCPU + NVMe and decouples the orchestrator from b1's hardware.
|
||||
This is the orchestrator analogue of the cc-ci **server** move in [[plan-migrate-cc-ci-to-hetzner]]
|
||||
(that one moves the *CI server*; this one moves the *orchestrator that drives the loops*).
|
||||
|
||||
> **Note on naming:** this migration was carried out directly via `terraform/` + the
|
||||
> `cc-ci-orchestrator-hetzner` flake host. It is **not** the same as `plan-migrate-cc-ci-to-hetzner.md`
|
||||
> (the cc-ci CI server → Hetzner `cpx32`) nor `plan-orchestrator-migration.md` (Pi → Incus VM). All
|
||||
> three are distinct moves; only this file records the orchestrator → Hetzner step.
|
||||
|
||||
---
|
||||
|
||||
## The new host (facts)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Provider / type | **Hetzner Cloud `cpx22`** — AMD **2 vCPU / 4 GB**, dedicated vCPU, NVMe |
|
||||
| Location | `nbg1` (cpx11/cpx21 are retired there — hence `cpx22`) |
|
||||
| Hetzner server ID | **134487234** |
|
||||
| Public IPv4 | **168.119.126.100** (IPv6 disabled) |
|
||||
| Tailnet | **`cc-ci-orchestrator-1`** @ **100.84.190.30** (`taila4a0bf.ts.net`); joins via `/etc/ts-auth-key` |
|
||||
| OS | `debian-12` image → **nixos-infect** → NixOS, converged by the flake |
|
||||
| Flake host | **`nixosConfigurations.cc-ci-orchestrator-hetzner`** (`flake.nix` → `nix/hosts/cc-ci-orchestrator-hetzner/{configuration,hardware}.nix`) |
|
||||
| Workspace | `/srv/cc-ci-orch` (this repo); `/srv/cc-ci` is a **symlink** to it. Loop clones: `/srv/cc-ci/cc-ci`, `/srv/cc-ci/cc-ci-adv` |
|
||||
|
||||
The login keys (root `authorizedKeys`) and swap (4 GB disk swap — 4 GB RAM is tight for 3+ claude
|
||||
sessions) are declared in `configuration.nix`.
|
||||
|
||||
## How it was provisioned (reproducible)
|
||||
|
||||
The whole box is reproducible from `terraform/` + one `nixos-rebuild`:
|
||||
|
||||
1. **`terraform apply`** (`terraform/main.tf`): `hcloud_server` `cpx22` from `debian-12` in `nbg1`,
|
||||
`user_data = user-data.sh` runs **nixos-infect** on first boot (Debian→NixOS, reboot).
|
||||
2. **Stage 2** (`terraform/README.md`): SSH in, capture the nixos-infect hardware config
|
||||
(→ `nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix`), then converge:
|
||||
```bash
|
||||
# on the server, from the repo root (/srv/cc-ci-orch)
|
||||
nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner
|
||||
```
|
||||
3. Stage credentials (not in git, placed once): `/etc/ts-auth-key` (tailnet join), the loops'
|
||||
`~/.ssh/cc-ci-root-ed25519` + `.testenv`, and the sops master age key. `claude auth login`
|
||||
(device code) is the one interactive step so the loops can run `--remote-control`.
|
||||
4. Stage the workspace: clone this repo to `/srv/cc-ci-orch` (symlink `/srv/cc-ci`), the Builder /
|
||||
Adversary clones, `cc-ci-secrets`, `references/`; copy `.cc-ci-logs/.phase-idx` (resume point).
|
||||
|
||||
**Commit trail:** `0103f36` (terraform + flake host, initial `cpx11`) → `17951b8` (fix → `cpx22`,
|
||||
add lock) → `c44b967` (real cpx22 hardware config from nixos-infect, server 134487234). Plus the
|
||||
close-out commit below (root keys, drop tailscale `--ssh`, enable the loops service, this doc).
|
||||
|
||||
## Reboot-resilience (the point of running on a managed host)
|
||||
|
||||
`configuration.nix` declares **`systemd.services.cc-ci-loops`** — a oneshot that runs
|
||||
`launch.sh start` with `RESUME_PHASE=1` after `network-online`/`tailscaled`, bringing the loops +
|
||||
watchdog back on boot. It was authored **disabled** ("defined but NOT enabled until workspace is
|
||||
staged") with `wantedBy` commented out. **Close-out (2026-05-31):** the workspace is staged and the
|
||||
loops are running, so `wantedBy = [ "multi-user.target" ]` was uncommented and `nixos-rebuild switch`
|
||||
re-run → `systemctl is-enabled cc-ci-loops.service` = **enabled**. A reboot is now a non-event:
|
||||
systemd resumes the saved phase. (`reboot-log.sh`, the ExecStartPre, appends to
|
||||
[[REBOOTS.md]] boot_id-gated.)
|
||||
|
||||
> **Caveat seen at first boot on this host:** the loops were initially started *by hand* during
|
||||
> staging (not by the service), so the first boot did NOT log to `REBOOTS.md` and the service showed
|
||||
> `linked`/not-enabled. Enabling `wantedBy` (above) is what wires the automatic path.
|
||||
|
||||
## Status of the migration
|
||||
|
||||
- ✅ Hetzner `cpx22` provisioned + converged from the flake (terraform + nixos-infect + one rebuild).
|
||||
- ✅ On the tailnet (`cc-ci-orchestrator-1`) and ssh-able on the public IP.
|
||||
- ✅ Loops + Adversary + watchdog running; phase sequence auto-advancing (watchdog on per-phase `## DONE`).
|
||||
- ✅ `cc-ci-loops.service` **enabled** → reboot-resilient.
|
||||
- ◻︎ Old Incus orchestrator VM (`100.116.55.106`) — keep as cold standby a few days, then delete.
|
||||
- ◻︎ Rotate the tailnet name once the old `cc-ci-orchestrator` peer is gone (this box is `…-1`).
|
||||
@@ -0,0 +1,151 @@
|
||||
# Plan — migrate the orchestrator off the Pi onto a dedicated NixOS Incus VM
|
||||
|
||||
**Goal:** move everything that drives the cc-ci loops (the Builder/Adversary loops, the watchdog,
|
||||
the SOCKS proxy, the orchestrator session itself) off the Raspberry Pi and onto a new, dedicated,
|
||||
**reboot-resilient NixOS VM** on b1 — declared in a new git repo **`cc-ci-orchestrator`**. Finish by
|
||||
relocating this orchestrator session there too.
|
||||
|
||||
**Why:** the Pi has rebooted twice today, each time silently killing the tmux loops + watchdog
|
||||
(they don't survive reboot, nothing auto-restarts them). A NixOS VM lets us declare the whole rig
|
||||
(claude CLI, proxy, loop supervisor) as systemd services that come back on boot — turning a reboot
|
||||
into a non-event. It also consolidates the orchestrator next to the infra it manages.
|
||||
|
||||
**Status:** COMPLETE (2026-05-31). All agents run on the VM; Pi fully decommissioned. Kept as a historical record.
|
||||
|
||||
**Phase A ✅ COMPLETE (2026-05-30):** VM `cc-ci-orchestrator` (**2 GB / 2 vCPU / 30 GB**,
|
||||
`incus-base-vm`, NixOS 24.11) created via the Incus API + booted; **on the tailnet at
|
||||
`100.116.55.106`**; **ssh works** (`ssh cc-ci-orchestrator` through the :1055 proxy — `cc-ci-root`
|
||||
pubkey added via `incus exec`). Reproducible Terraform record at
|
||||
`incus-terraform-nix-vm-creator/projects/cc-ci-orchestrator/` (note: this instance was API-created, so
|
||||
TF drift — see PROVENANCE.txt).
|
||||
- **TS-key finding:** the VM-creator's `.test.env` reusable key is **REVOKED** ("API key does not
|
||||
exist"). The **`/srv/cc-ci/.testenv` `TS_AUTH_KEY` is valid** — used it to join, and persisted it into
|
||||
the VM's `/etc/ts-auth-key`. So the plan's "operator provides a fresh TS key" item is **resolved** (no
|
||||
new key needed); housekeeping: revoke/rotate the dead key in `.test.env`.
|
||||
- **Sizing watch:** 2 GB ≈ 1.7 GiB usable; fine idle (284 MiB) but tight for 3 concurrent claude
|
||||
sessions (Pi OOM lesson). Phase B will declare a **swapfile**; bump to 4 GB pre-cutover if needed.
|
||||
|
||||
**Next — Phase B:** the `cc-ci-orchestrator` NixOS-config git repo (SOCKS proxy + loop-supervisor boot
|
||||
service + claude CLI + sops secrets). Then C (stage workspace), claude auth (operator), D/E (cutover).
|
||||
|
||||
---
|
||||
|
||||
## 0. Current footprint (what has to move)
|
||||
|
||||
On the Pi (`raspberrypi`, aarch64), workspace `/srv/cc-ci` (itself the
|
||||
`cc-ci-orchestrator` git repo — formerly `cc-ci-autonomous-orchestrator`):
|
||||
|
||||
| Item | What | Move strategy |
|
||||
|---|---|---|
|
||||
| `cc-ci-plan/` | loop code: `launch.sh`, `plan*.md`, `prompts/`, `kickoff.md` | in git (this repo) → clone on VM |
|
||||
| `cc-ci/`, `cc-ci-adv/` | Builder + Adversary working clones (~13M each) | **re-clone from git.autonomic.zone** on the VM (cleaner than copying) |
|
||||
| `.cc-ci-logs/` | watchdog/loop logs + `.phase-idx` | copy `.phase-idx` (the resume point); logs start fresh |
|
||||
| `cc-ci-secrets/` | sops-encrypted secrets repo | in git → clone |
|
||||
| `references/` | recipe-maintainer corpus (read-only parity source) | clone/rsync from `/srv/recipe-maintainer` |
|
||||
| **`.testenv`** | TS auth key, Gitea bot creds | **out-of-band copy** (gitignored, never in git) |
|
||||
| **`~/.ssh/cc-ci-root-ed25519`** | root SSH key to cc-ci | **out-of-band copy** |
|
||||
| **`.sops/master-age.txt`** | master recovery age key | **out-of-band copy** |
|
||||
| **Incus mTLS certs** (`/srv/incus-terraform-nix-vm-creator/terraform-secrets/`) | `terraform.{crt,key}`, `vm_ssh_key` | **out-of-band copy** — so the VM can itself manage VMs |
|
||||
| `cc-ci-tailscaled.service` | userspace SOCKS proxy :1055 | **re-declare as NixOS** (see §3) |
|
||||
| **claude CLI + auth** | `~/.local/bin/claude` v2.1.154 + `~/.claude.json` | install on VM + **operator `claude auth login`** (§4) |
|
||||
| this orchestrator session | the supervising claude conversation | **operator-assisted cutover** (§6) |
|
||||
|
||||
Two hard human-in-the-loop steps, called out explicitly: **claude auth on the new VM** (device-code
|
||||
login, can't be scripted) and the **final session cutover** (the operator connects to the new
|
||||
orchestrator session). Everything else I can do.
|
||||
|
||||
## 1. Target VM spec
|
||||
|
||||
- **Host/API:** b1 Incus, `https://100.117.251.31:8443`, project `terraform-ci`, mTLS certs (have).
|
||||
- **Name:** `cc-ci-orchestrator` (tailnet hostname too).
|
||||
- **Resources:** **2 GB RAM, 2 vCPU, 30 GB disk** (dir backend → resize needs a reboot; size at
|
||||
create time so no later grow). b1 has ample headroom (only cc-nix-test @8GB running).
|
||||
- **Image:** the existing imported NixOS base VM image (`incus-base-vm`) — already ships tailscale,
|
||||
openssh, git/jq/curl, flakes, cloud-init.
|
||||
- **Tailnet:** joins via a fresh `TS_AUTH_KEY` (operator provides, or reuse the keyed approach in
|
||||
`terraform-secrets/.test.env`). MagicDNS name `cc-ci-orchestrator.taila4a0bf.ts.net`.
|
||||
- **Bootstrap:** cloud-init writes the `cc-ci-orchestrator` flake config + `nixos-rebuild switch`.
|
||||
|
||||
## 2. The new `cc-ci-orchestrator` git repo (NixOS config)
|
||||
|
||||
A new **private** repo on `git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator` (bot is org
|
||||
admin). It is the NixOS config for this VM — the orchestrator's equivalent of what `cc-ci` is for the
|
||||
test server. Contents:
|
||||
|
||||
- `flake.nix` + `hosts/cc-ci-orchestrator/configuration.nix` — the VM's NixOS config.
|
||||
- **Packages:** `claude-code` (CLI), `git`, `tmux`, `python3`, `jq`, `openssh`, `nodejs` (claude
|
||||
runtime), `coreutils`, `nettools` (`nc` for the proxy ProxyCommand).
|
||||
- **`services.cc-ci-tailscaled`** — the userspace tailscaled SOCKS proxy on :1055, as a NixOS
|
||||
systemd service (port to NixOS from the Pi's `cc-ci-tailscaled.service`). This is the path to b1 +
|
||||
cc-ci.
|
||||
- **`services.cc-ci-orchestrator`** — a systemd service that runs `launch.sh start` with
|
||||
`RESUME_PHASE=1` **on boot** (after the proxy + network are up), as the workspace user. **This is
|
||||
the reboot-resilience fix** — the loops + watchdog come back automatically after any reboot.
|
||||
- **Secrets via sops-nix** (like cc-ci): the out-of-band secrets (`.testenv`, ssh key, incus certs)
|
||||
are sops-encrypted into the repo, decrypted at activation to their runtime paths. The **master age
|
||||
key** is the one irreducible out-of-band bootstrap secret placed on the VM once.
|
||||
- `~/.ssh/config` for `cc-ci` (root, ProxyCommand via :1055) declared.
|
||||
- **Excluded from git:** claude's own auth (`~/.claude.json`) — that's per-user login state, set up
|
||||
once interactively (§4), not committed.
|
||||
|
||||
## 3. Execution phases
|
||||
|
||||
### Phase A — provision the VM (reversible; safe to do while Pi loops keep running)
|
||||
1. Create `cc-ci-orchestrator` VM via the Incus API (2 GB / 2 vCPU / 30 GB, NixOS base image, TS auth
|
||||
key in cloud-init). Wait for tailnet join + ssh.
|
||||
2. Verify: `ssh` in, `tailscale status`, `nixos-rebuild` available, can reach b1 API + cc-ci through
|
||||
its own proxy once configured.
|
||||
|
||||
### Phase B — author + apply the `cc-ci-orchestrator` repo
|
||||
3. Create the private git repo; author the flake/config (§2); commit/push.
|
||||
4. Place the master age key on the VM; sops-encrypt the out-of-band secrets into the repo.
|
||||
5. `nixos-rebuild switch` on the VM → proxy service up, packages present, services defined (loop
|
||||
supervisor **not yet started** — or started in a dry mode).
|
||||
|
||||
### Phase C — stage the workspace (no cutover yet)
|
||||
6. On the VM: clone `cc-ci-orchestrator` (the loop code), clone the Builder/Adversary
|
||||
working repos fresh from git.autonomic.zone, clone `cc-ci-secrets`, rsync `references/`.
|
||||
7. Copy `.phase-idx` (resume point = phase 2) so the VM watchdog resumes the right phase.
|
||||
8. **Operator step:** `claude auth login` on the VM (device code) so the loops can run
|
||||
`--remote-control --dangerously-skip-permissions`. Verify with a throwaway interactive claude.
|
||||
|
||||
### Phase D — cutover (the only disruptive moment; pick a clean point)
|
||||
9. **Quiesce the Pi:** stop the Pi loops + watchdog (`launch.sh stop`); confirm both loops are at a
|
||||
safe point (no half-written commit; `git status` clean in both clones, last work pushed).
|
||||
10. **Start on the VM:** enable + start the `cc-ci-orchestrator` systemd service → `launch.sh start`
|
||||
(RESUME_PHASE=1) brings up Builder + Adversary + watchdog on the VM, resuming phase 2 from the
|
||||
repo state. Verify all three sessions + a handoff + public health.
|
||||
11. **Decommission the Pi loops:** disable the Pi's `cc-ci-tailscaled` + leave the workspace in place
|
||||
(read-only fallback) but not running loops. (Keep the Pi as a cold standby for a few days before
|
||||
deleting anything.)
|
||||
|
||||
### Phase E — move the orchestrator session (operator-assisted)
|
||||
12. On the VM, start the orchestrator session: `claude --remote-control 'autonomous-orchestrator'
|
||||
--dangerously-skip-permissions` in a tmux session, seeded with AGENTS.md + this plan so it picks
|
||||
up the supervising role. The **operator connects** to it (claude.ai/code) — this is the
|
||||
"move myself" step; a session can't transplant itself across machines, so it's a fresh
|
||||
orchestrator session on the VM with full context from the repo.
|
||||
13. This Pi-side orchestrator session hands off (writes a short state note) and goes idle/ends.
|
||||
|
||||
## 4. Risks & mitigations
|
||||
- **claude auth (human step):** unavoidable device-code login on the VM. Mitigation: do it in Phase
|
||||
C, well before cutover; verify before quiescing the Pi.
|
||||
- **Loops mid-work at cutover:** pick a quiet point (between gate claims / after a push); the loops
|
||||
re-orient from git on restart anyway, so worst case is a re-run of an in-flight iteration.
|
||||
- **Secrets sprawl:** out-of-band secrets are copied once, then sops-managed in the new repo; never
|
||||
committed in plaintext (same discipline as cc-ci). The master age key is the sole bootstrap secret.
|
||||
- **Self-move gap:** between Pi-session-ends and VM-session-connected, there's no live orchestrator.
|
||||
The watchdog (now a boot service) keeps the loops alive independently, so this gap is safe.
|
||||
- **Rollback:** until the Pi workspace is deleted, reverting = stop VM service, `launch.sh start` on
|
||||
the Pi again. Keep the Pi intact until the VM has run clean through at least one reboot + one gate
|
||||
handshake.
|
||||
- **Reboot-resilience proof:** before trusting the VM, reboot it once and confirm the loops +
|
||||
watchdog + proxy all come back via systemd (the whole point of the move).
|
||||
|
||||
## 5. Operator-assisted steps (the only things I can't fully do)
|
||||
1. Provide a fresh `TS_AUTH_KEY` for the VM (or confirm reuse of the one in `terraform-secrets`).
|
||||
2. `claude auth login` on the VM (device code).
|
||||
3. Connect to the new orchestrator session on the VM at cutover (Phase E).
|
||||
|
||||
Everything else (VM create, repo author, NixOS config, secret migration, workspace staging, the
|
||||
loop cutover) I can drive.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Plan stub — repo consolidation + recipe-maintainer submodule
|
||||
|
||||
**Status:** COMPLETE (2026-05-31).
|
||||
Both cleanups executed. See commit history for details.
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — merge the two orchestrator repos into one `cc-ci-orchestrator`
|
||||
|
||||
Today there are **two** repos:
|
||||
- `recipe-maintainers/cc-ci-autonomous-orchestrator` — the loop code/plans/prompts (`cc-ci-plan/`,
|
||||
`launch.sh`, `prompts/`, `references/`, `AGENTS.md`). This is the workspace at `/srv/cc-ci`
|
||||
(`/home/loops/cc-ci` on the VM).
|
||||
- `recipe-maintainers/cc-ci-orchestrator` — the VM's NixOS config (`configuration.nix`, `README.md`),
|
||||
created during the VM migration.
|
||||
|
||||
**Goal:** combine into a single repo named **`cc-ci-orchestrator`** (loop code/plans + the VM's NixOS
|
||||
config in one place). Likely: move the NixOS config into the autonomous-orchestrator repo (e.g.
|
||||
`nixos/` or `vm/`), then **rename** that repo to `cc-ci-orchestrator` on Gitea.
|
||||
**Touch-ups when doing it:** the workspace clone URL in `launch-orchestrator.sh` / kickoff docs /
|
||||
`cc-ci-loops.service`, the VM's `git remote`, and `plan.md`/`kickoff.md` references to the repo name.
|
||||
Keep `recipe-maintainers/cc-ci` (the CI **product** repo the loops build) separate — this is only
|
||||
about the two **orchestrator** repos.
|
||||
|
||||
## Task 2 — make `references/recipe-maintainer` a git submodule
|
||||
|
||||
Today `references/recipe-maintainer` is a **symlink → `/srv/recipe-maintainer/`** (an absolute path).
|
||||
On the Pi that target existed; **on the VM it does NOT, so the symlink dangles and the parity corpus
|
||||
is empty** — Phase-2 parity-porting (P2) reads from `references/recipe-maintainer/recipe-info/<recipe>/
|
||||
tests/*.py`, so recipes not already ported can't be ported until this is fixed.
|
||||
|
||||
**Goal:** replace the symlink with a proper **git submodule** from
|
||||
**`https://git.autonomic.zone/notplants/recipe-maintainer`**:
|
||||
```
|
||||
git rm references/recipe-maintainer # remove the symlink
|
||||
git submodule add https://git.autonomic.zone/notplants/recipe-maintainer references/recipe-maintainer
|
||||
git commit
|
||||
# on each clone / the VM: git submodule update --init --recursive
|
||||
```
|
||||
This both fixes the current VM gap (corpus present + reproducible, no absolute-path dependency) and
|
||||
makes the workspace self-contained on any host.
|
||||
|
||||
**Interim (before this plan runs), if the loops need the corpus on the VM:** clone recipe-maintainer
|
||||
to `/srv/recipe-maintainer` on the VM (needs root — `/srv` is root-owned) so the existing symlink
|
||||
resolves; OR do Task 2 directly.
|
||||
|
||||
## Operator action required — Pi clone remote update
|
||||
|
||||
The Gitea rename (`cc-ci-autonomous-orchestrator` → `cc-ci-orchestrator`) breaks the `origin` remote
|
||||
on any existing clone. **On the Pi** (and any other machine with a clone), run:
|
||||
|
||||
```
|
||||
cd /srv/cc-ci # or wherever the workspace is cloned
|
||||
git remote set-url origin https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git
|
||||
# (include bot credentials in the URL if needed, as before)
|
||||
```
|
||||
|
||||
The VM clone's remote was updated automatically as part of this task.
|
||||
Generated
+23
@@ -0,0 +1,23 @@
|
||||
# This file is maintained automatically by "tofu init".
|
||||
# Manual edits may be lost in future updates.
|
||||
|
||||
provider "registry.opentofu.org/hetznercloud/hcloud" {
|
||||
version = "1.64.0"
|
||||
constraints = "1.64.0"
|
||||
hashes = [
|
||||
"h1:FUkTfFrWlmv0JhsbjQvTk3zY7A2Q0LuoSs0PKEzaLpk=",
|
||||
"zh:5bf7f8f429b1a8f485988d199f46295676a6cdf7d84ad11f1f4613faecfa89d5",
|
||||
"zh:63b3d182474dd5afd0d5ab3f5f66228b752504436bcb2f4721bd6f1233d0f2ae",
|
||||
"zh:6867da2d89d297b6760d80dde373e74df511bea72f7daccf6a944a9de4b4d4ed",
|
||||
"zh:766fdcea1b03038a92414eafaa430b9ac0c57b36ce4c1573e6e291431659d528",
|
||||
"zh:7f3186dfcae4028eac4f2c9c2c382b49c1fad0b63d0471b50748ee6817fbd8d2",
|
||||
"zh:bb8a33b6ff9a4d3bce87628c49b08a4780e2c034762f40112058d96f5a4e52bd",
|
||||
"zh:cc93751c7c90a37f180cf3e5439ed34f3154e60de5920a13d153d93954938239",
|
||||
"zh:d6e2abf05a0eb8fe0544eb099960a4962db61532e7757016ccacbf0b83bcd1ae",
|
||||
"zh:da9e3adedd8d33623aac4929fa8b1210f98d2931d5737c201da0dda992dd25ab",
|
||||
"zh:dffc931aec4d7b0733690e115b1aabdf5c157b7d347a09a9d149ee6b7e9d8ce3",
|
||||
"zh:e565dea4f28182099a271f794e3b781f069ea54976f5f05dbb79a1c2b6627459",
|
||||
"zh:e79411287af28ccf6187bd418b7ea2ee217e642026392ddc8027bf3e3287fb80",
|
||||
"zh:f5102d7141a04c193dffbb5cbc3f7e3588c41b87e11877d2e20d57ea5ef64123",
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
# terraform — Hetzner cc-ci-orchestrator server
|
||||
|
||||
Provisions a Hetzner **cpx11** (2 vCPU / 2 GB dedicated AMD / 40 GB NVMe) for the cc-ci loops
|
||||
runtime (Builder + Adversary + Watchdog + Orchestrator sessions), replacing the slow b1 Incus VM.
|
||||
Uses nixos-infect to convert Debian → NixOS, then converges via the cc-ci-orchestrator flake.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — provision the server
|
||||
|
||||
```bash
|
||||
# from /srv/cc-ci/terraform/
|
||||
source /srv/cc-ci/.testenv # loads HCLOUD_TOKEN
|
||||
export TF_VAR_ssh_public_key="$(cat /home/loops/.ssh/cc-ci-root-ed25519.pub)"
|
||||
|
||||
tofu init
|
||||
tofu plan
|
||||
tofu apply
|
||||
```
|
||||
|
||||
Note the `server_ipv4` output. nixos-infect runs on first boot — wait ~5 min, then:
|
||||
|
||||
```bash
|
||||
# confirm NixOS is up (may need to retry while infect reboots)
|
||||
ssh root@<server_ipv4> 'nixos-version'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — converge to cc-ci-orchestrator-hetzner
|
||||
|
||||
### 2a. Capture hardware config
|
||||
|
||||
```bash
|
||||
ssh root@<server_ipv4> 'cat /etc/nixos/hardware-configuration.nix'
|
||||
```
|
||||
|
||||
Copy the output to `nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix` in this repo, commit, push.
|
||||
|
||||
### 2b. Stage workspace on the new server
|
||||
|
||||
```bash
|
||||
ssh root@<server_ipv4>
|
||||
|
||||
# Install Tailscale auth key (from .testenv TS_AUTH_KEY)
|
||||
echo "<TS_AUTH_KEY>" > /etc/ts-auth-key && chmod 600 /etc/ts-auth-key
|
||||
|
||||
# Clone this repo as the loops user workspace
|
||||
git clone --recursive \
|
||||
https://autonomic-bot:<token>@git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git \
|
||||
/srv/cc-ci-orch
|
||||
ln -sfn /srv/cc-ci-orch /srv/cc-ci # loops expect /srv/cc-ci
|
||||
|
||||
# Place master age key (copied from current VM .sops/master-age.txt)
|
||||
mkdir -p /srv/cc-ci/.sops
|
||||
scp loops@<old-vm>:/srv/cc-ci/.sops/master-age.txt /srv/cc-ci/.sops/master-age.txt
|
||||
chmod 600 /srv/cc-ci/.sops/master-age.txt
|
||||
```
|
||||
|
||||
### 2c. Run nixos-rebuild
|
||||
|
||||
```bash
|
||||
# on the new server
|
||||
cd /srv/cc-ci
|
||||
nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner
|
||||
```
|
||||
|
||||
### 2d. Stage credentials (not in git — placed once)
|
||||
|
||||
```bash
|
||||
# SSH key for reaching cc-ci
|
||||
mkdir -p /home/loops/.ssh && chmod 700 /home/loops/.ssh
|
||||
# scp cc-ci-root-ed25519 from current VM or copy content
|
||||
chmod 600 /home/loops/.ssh/cc-ci-root-ed25519
|
||||
|
||||
# .testenv (GITEA creds, etc.)
|
||||
cp /path/to/.testenv /srv/cc-ci/.testenv && chmod 600 /srv/cc-ci/.testenv
|
||||
```
|
||||
|
||||
### 2e. Auth claude and start loops
|
||||
|
||||
```bash
|
||||
# as loops user on new server
|
||||
sudo -u loops /home/loops/.local/bin/claude auth login # device code — operator step
|
||||
|
||||
# start the loops
|
||||
cd /srv/cc-ci && sudo -u loops ./cc-ci-plan/launch.sh start
|
||||
```
|
||||
|
||||
### 2f. Verify
|
||||
|
||||
```bash
|
||||
tmux ls # should show cc-ci-builder, cc-ci-adv, cc-ci-watchdog
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cutover
|
||||
|
||||
Once the new server is running and the loops are verified:
|
||||
|
||||
1. Update the `Host cc-ci` entry in the current VM's `/home/loops/.ssh/config` if needed
|
||||
2. Stop the old Incus VM (or just leave it idle — it costs nothing in disk)
|
||||
|
||||
---
|
||||
|
||||
## Variables
|
||||
|
||||
| Variable | Default | Notes |
|
||||
|---|---|---|
|
||||
| `location` | `nbg1` | Nuremberg |
|
||||
| `server_type` | `cpx11` | 2 vCPU / 2 GB dedicated AMD. Upgrade to `cpx21` (4 GB) if OOM. |
|
||||
| `image` | `debian-12` | nixos-infect base |
|
||||
| `server_name` | `cc-ci-orchestrator` | |
|
||||
| `ssh_public_key` | required | Pass via `TF_VAR_ssh_public_key` |
|
||||
|
||||
---
|
||||
|
||||
## State
|
||||
|
||||
`terraform.tfstate` and `terraform.tfstate.backup` are gitignored. Keep the state file locally or
|
||||
in a remote backend — losing it means `tofu destroy` can't find the server (use `tofu import` to
|
||||
recover, or delete directly via the Hetzner console).
|
||||
@@ -0,0 +1,32 @@
|
||||
resource "hcloud_ssh_key" "cc_ci_orch" {
|
||||
name = "cc-ci-orchestrator-deploy"
|
||||
public_key = var.ssh_public_key
|
||||
|
||||
labels = {
|
||||
project = "cc-ci-orchestrator"
|
||||
managed = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
resource "hcloud_server" "cc_ci_orch" {
|
||||
name = var.server_name
|
||||
server_type = var.server_type
|
||||
image = var.image
|
||||
location = var.location
|
||||
ssh_keys = [hcloud_ssh_key.cc_ci_orch.id]
|
||||
|
||||
# Stage 1: cloud-init runs nixos-infect on first boot, converting Debian to NixOS, then reboots.
|
||||
# Wait ~5 min after apply, then SSH in and run Stage 2 per README.md.
|
||||
user_data = file("${path.module}/user-data.sh")
|
||||
|
||||
public_net {
|
||||
ipv4_enabled = true
|
||||
ipv6_enabled = false
|
||||
}
|
||||
|
||||
labels = {
|
||||
project = "cc-ci-orchestrator"
|
||||
managed = "terraform"
|
||||
stage = "infect"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
output "server_ipv4" {
|
||||
description = "Public IPv4 address of the cc-ci-orchestrator Hetzner server"
|
||||
value = hcloud_server.cc_ci_orch.ipv4_address
|
||||
}
|
||||
|
||||
output "server_id" {
|
||||
description = "Hetzner internal server ID"
|
||||
value = hcloud_server.cc_ci_orch.id
|
||||
}
|
||||
|
||||
output "ssh_connect" {
|
||||
description = "SSH command to connect as root (after nixos-infect)"
|
||||
value = "ssh root@${hcloud_server.cc_ci_orch.ipv4_address}"
|
||||
}
|
||||
|
||||
output "nixos_infect_log" {
|
||||
description = "Check infect progress"
|
||||
value = "ssh root@${hcloud_server.cc_ci_orch.ipv4_address} 'cat /var/log/nixos-infect.log'"
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stage 1 — convert Debian 12 → NixOS via nixos-infect (pinned revision).
|
||||
#
|
||||
# nixos-infect generates /etc/nixos/{configuration.nix,hardware-configuration.nix,networking.nix}
|
||||
# with Hetzner-correct bootloader (GRUB) and networking, then reboots into NixOS.
|
||||
#
|
||||
# After the reboot SSH as root is available. Run Stage 2 per terraform/README.md.
|
||||
# Logs: /var/log/nixos-infect.log
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Same pinned revision as the cc-ci server terraform (2026-03-22).
|
||||
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
|
||||
|
||||
export NIX_CHANNEL="nixos-24.11"
|
||||
export PROVIDER="hetzner"
|
||||
export NIXOS_IMPORT=""
|
||||
|
||||
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" \
|
||||
| bash -x 2>&1 | tee /var/log/nixos-infect.log
|
||||
@@ -0,0 +1,39 @@
|
||||
variable "location" {
|
||||
description = "Hetzner datacenter (nbg1=Nuremberg, fsn1=Falkenstein, hel1=Helsinki)"
|
||||
type = string
|
||||
default = "nbg1"
|
||||
}
|
||||
|
||||
variable "server_type" {
|
||||
description = <<-EOT
|
||||
Hetzner server type. Must be x86 — the flake is x86_64-linux; NEVER use cax* (ARM).
|
||||
cpx22 = AMD 2 vCPU / 4 GB (default; dedicated vCPU, NVMe — the orchestrator loops runtime).
|
||||
cpx32 = AMD 4 vCPU / 8 GB (upgrade if more CPU headroom needed).
|
||||
cx23 = Intel 2 vCPU / 4 GB (shared vCPU, cheaper alternative).
|
||||
Note: cpx11/cpx21 are retired in nbg1.
|
||||
EOT
|
||||
type = string
|
||||
default = "cpx22"
|
||||
|
||||
validation {
|
||||
condition = !startswith(var.server_type, "cax")
|
||||
error_message = "ARM server types (cax*) are not supported — the flake is x86_64-linux only."
|
||||
}
|
||||
}
|
||||
|
||||
variable "image" {
|
||||
description = "Base OS image. nixos-infect supports debian-12 and ubuntu-24.04. debian-12 preferred."
|
||||
type = string
|
||||
default = "debian-12"
|
||||
}
|
||||
|
||||
variable "ssh_public_key" {
|
||||
description = "SSH public key content (the full line). Registered with Hetzner for root access post-infect. Pass via TF_VAR_ssh_public_key."
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "server_name" {
|
||||
description = "Hetzner server name and initial NixOS hostname"
|
||||
type = string
|
||||
default = "cc-ci-orchestrator"
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
terraform {
|
||||
required_version = ">= 1.0"
|
||||
required_providers {
|
||||
hcloud = {
|
||||
source = "hetznercloud/hcloud"
|
||||
version = "1.64.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# The hcloud provider reads HCLOUD_TOKEN from the environment automatically.
|
||||
# Never put the token value in any .tf file or .tfvars — keep it in the shell
|
||||
# environment (export HCLOUD_TOKEN=...) or pass via TF_VAR_hcloud_token.
|
||||
provider "hcloud" {}
|
||||
Reference in New Issue
Block a user