nix: one Hetzner host for the CI server + the orchestrator (#cc-ci)

The orchestrator's flake now builds the machine it shares with the cc-ci CI server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server (new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix — the host contract those units always assumed (loops user, claude/opencode CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns 80/443, nix-ld, tool set, `ssh cc-ci` → loopback). nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old server's layout so the flake evaluates; they get replaced by the nixos-infect output of 195.201.88.249. README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake → staging → data restore → cutover). archive/ holds the retired Incus/Hetzner orchestrator host configs, the old terraform and the migration plans; references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working plan for the move. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
2026-09-07 19:58:33 +00:00
parent 5bebd13e49
commit 31af820079
32 changed files with 863 additions and 90 deletions
@@ -0,0 +1,35 @@
+# cc-ci-orchestrator
+
+> ⚠️ **HISTORICAL.** This README describes the retired **Incus VM** (`100.116.55.106`). The
+> orchestrator now runs on **Hetzner** — the live host config is
+> `nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix`. See
+> `cc-ci-plan/plan-orchestrator-hetzner-migration.md` for the current setup. Kept for history.
+
+NixOS config for the **`cc-ci-orchestrator`** Incus VM (b1, project `terraform-ci`, tailnet
+`100.116.55.106`) — the reboot-resilient host for the cc-ci Builder/Adversary loops + watchdog +
+orchestrator session, moved off the unstable 905 MiB Pi.
+
+See `cc-ci-plan/plan-orchestrator-migration.md` for the full migration.
+
+## Files
+- `configuration.nix` — the VM's NixOS config (channel-based, `nixos-24.11`). Deployed to
+  `/etc/nixos/configuration.nix` on the VM. Provides: nix-ld (so the standalone Claude Code Bun binary
+  runs), tmux/git/python/jq + tools, a 4 GB swapfile, direct ssh to cc-ci (the VM is a tailnet peer —
+  no SOCKS proxy needed, unlike the Pi), an idempotent `claude-install` oneshot, and the
+  `cc-ci-loops` supervisor service (defined, **enabled in Phase D** once the workspace is staged).
+
+## Deploy (until this is wired to a flake/auto-pull)
+```
+# copy configuration.nix to the VM, then:
+ssh cc-ci-orchestrator 'nixos-rebuild switch'    # or run detached: see below
+```
+Over the (currently flaky) Pi→VM link, run the rebuild **detached** on the VM so an ssh/proxy drop
+doesn't abort it, e.g. `systemd-run --unit=orch-rebuild --collect nixos-rebuild switch` then poll
+`journalctl -u orch-rebuild`.
+
+## Status
+- Phase A: VM created (2 GB / 2 vCPU / 30 GB), on tailnet, ssh-able. ✅
+- Phase B: this config (DRAFT) — nix-ld/claude validation pending on the VM.
+- Operator step pending (Phase C): `claude auth login` on the VM (device-code; can't be scripted).
+- Secrets to stage (Phase C, out-of-band): `/srv/cc-ci/.testenv`, `~/.ssh/cc-ci-root-ed25519`,
+  Incus mTLS certs, the sops master age key.
@@ -0,0 +1,135 @@
+# atproto-likes — the "most-liked accounts" web UI, run as a docker-compose stack
+#
+# ⚠️  COPY. Canonical source:
+#     /srv/project-orchestrator/projects/notplants-atproto/nix/atproto-likes.nix
+#     Pure evaluation cannot import a path outside the flake tree. Re-copy after editing.
+#
+# Host-specific context that is deliberately NOT in the public project repo:
+#   - the domain is fronted by the gateway (143.244.213.108); it forwards everything here
+#   - this vhost lists 0.0.0.0 only so it never shares an nginx address group with
+#     oc.commoninternet.net (opencode, on the tailscale address, no auth of its own)
+#   - /srv/project-orchestrator/projects/notplants-atproto/tools/check-exposure.sh asserts
+#     what this host serves publicly; run it after any nginx change
+# behind the host's nginx.
+#
+# Import this from the host configuration:
+#   imports = [ /srv/project-orchestrator/projects/notplants-atproto/nix/atproto-likes.nix ];
+#
+# What it sets up:
+#   * the Docker daemon
+#   * a systemd unit that builds and runs docker-compose.yml from the project checkout
+#   * an nginx vhost for atproto.commoninternet.net with a Let's Encrypt cert,
+#     proxying to the container on 127.0.0.1:8731
+#   * ports 80/443 opened in the firewall
+#
+# DNS / TLS: point the domain's A record at this host (or at a reverse proxy that
+# forwards to it). ACME HTTP-01 needs the challenge to reach this nginx, so if a
+# proxy fronts the domain it must forward /.well-known/acme-challenge/ through.
+# Until a certificate is issued, NixOS installs a self-signed placeholder so nginx
+# still starts.
+#
+# A proxy in front should preserve the Host header (nginx routes by server_name),
+# pass Upgrade/Connection through (the loading page is a websocket) and allow a
+# long read timeout (a cold scan can run for minutes).
+{ config, pkgs, lib, ... }:
+
+let
+  projectDir = "/srv/project-orchestrator/projects/notplants-atproto";
+  domain = "atproto.commoninternet.net";
+  hostPort = 8731;                        # must match the ports: mapping in docker-compose.yml
+  compose = "${pkgs.docker-compose}/bin/docker-compose";
+in
+{
+  virtualisation.docker = {
+    enable = true;
+    # Reclaim dangling images from repeated `--build` runs.
+    autoPrune = { enable = true; dates = "weekly"; };
+  };
+
+  # Build + run the compose stack. Type=oneshot with RemainAfterExit: compose
+  # detaches, and the containers' own restart policy keeps them alive.
+  systemd.services.atproto-likes = {
+    description = "atproto-likes — most-liked-accounts web UI (docker compose)";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "docker.service" "docker.socket" "network-online.target" ];
+    requires = [ "docker.service" ];
+    wants = [ "network-online.target" ];
+    path = [ pkgs.docker pkgs.docker-compose ];
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+      WorkingDirectory = projectDir;
+      # A cold `--build` pulls python:3.12-slim and installs pip deps.
+      TimeoutStartSec = "1800";
+      # The container runs unprivileged as uid 10001 (see Dockerfile), but the
+      # page cache is a host bind mount, so its ownership is the host's business.
+      # Without this the app 500s on any actor whose cache files were written by
+      # an earlier root-running container.
+      ExecStartPre = [
+        "${pkgs.coreutils}/bin/mkdir -p ${projectDir}/cache"
+        "${pkgs.coreutils}/bin/chown -R 10001:10001 ${projectDir}/cache"
+      ];
+      ExecStart = "${compose} up -d --build --remove-orphans";
+      ExecStop = "${compose} down";
+      Restart = "on-failure";
+      RestartSec = "30s";
+    };
+  };
+
+  services.nginx = {
+    enable = true;
+    recommendedProxySettings = true;
+    virtualHosts.${domain} = {
+      # addSSL, NOT forceSSL. If a reverse proxy fronts this domain and its
+      # back-end hop is plain HTTP, a forced 301 to https://<domain>/ would
+      # resolve straight back to that proxy — a redirect loop. Serving both
+      # schemes lets whatever is in front decide where TLS terminates.
+      addSSL = true;
+      enableACME = true;
+      # Listens on 0.0.0.0 only, and the reject-everything default server below
+      # owns those addresses. nginx groups server blocks by the connection's local
+      # address, so keeping this vhost on the public address alone means it never
+      # shares an address group with anything else the host may serve on another
+      # interface — no other service can be reached by sending this one an
+      # unexpected Host header.
+      locations."/" = {
+        proxyPass = "http://127.0.0.1:${toString hostPort}";
+        # The loading page streams scan progress over a websocket.
+        proxyWebsockets = true;
+        # A cold scan can run for minutes with the socket open; the default 60s
+        # proxy read timeout would cut the loading page off mid-flower.
+        extraConfig = ''
+          proxy_read_timeout 1800s;
+          proxy_send_timeout 1800s;
+        '';
+      };
+    };
+
+    # Strict default server for the public addresses. Without an explicit
+    # default_server, nginx promotes the first server block in the group — so any
+    # vhost added later silently becomes what answers unmatched, absent or spoofed
+    # Host headers from the internet. This closes that door once: only names
+    # deliberately served get a response.
+    #
+    #   port 80  -> 444 (close the connection, no response at all)
+    #   port 443 -> ssl_reject_handshake, so an unknown SNI never even gets TLS
+    virtualHosts."public-default-reject" = {
+      default = true;
+      serverName = null;
+      rejectSSL = true;
+      listen = [
+        { addr = "0.0.0.0"; port = 80;  ssl = false; }
+        { addr = "0.0.0.0"; port = 443; ssl = true;  }
+      ];
+      extraConfig = "return 444;";
+    };
+  };
+
+  security.acme = {
+    acceptTerms = true;
+    defaults.email = "mfowler.email@protonmail.com";   # ACME contact
+  };
+
+  # Public HTTP/HTTPS.
+  networking.firewall.allowedTCPPorts = [ 80 443 ];
+}
@@ -0,0 +1,305 @@
+# cc-ci-orchestrator-hetzner — NixOS config for the Hetzner loops runtime host.
+#
+# Purpose: run the cc-ci Builder/Adversary/Watchdog loops + orchestrator/assistant sessions
+# on a Hetzner cpx11 (2 vCPU / 2 GB dedicated AMD / 40 GB NVMe), replacing the slow b1 Incus VM.
+#
+# Provision with terraform/ then converge with: nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner
+# See terraform/README.md for the full Stage 2 procedure.
+{ config, pkgs, lib, ... }:
+{
+  # hardware.nix is the nixos-infect generated hardware-configuration.nix (see README Stage 2a).
+
+  # atproto-likes — the notplants-atproto "most-liked accounts" web UI, served at
+  # atproto.commoninternet.net. Brings in Docker + a compose stack + an nginx vhost
+  # with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
+  # the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
+  # because pure evaluation cannot import a path outside the flake.
+  # cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
+  # the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
+  # a general agent box that also serves unrelated projects — can be reasoned about separately
+  # from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
+  imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
+
+  services.openssh = {
+    enable = true;
+    settings.PermitRootLogin = "yes";
+  };
+
+  # Root SSH access — all keys from the current orchestrator VM's /root/.ssh/authorized_keys.
+  users.users.root.openssh.authorizedKeys.keys = [
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526"
+  ];
+  networking.useDHCP = true;
+  networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
+  networking.firewall = {
+    enable = true;
+    trustedInterfaces = [ "tailscale0" ];
+    # Port 80 open only on the tailscale interface (trusted) — nginx binds there for oc.commoninternet.net.
+    allowedTCPPorts = [ 22 ];
+  };
+  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  system.stateVersion = "24.11";
+
+  # Tailscale — auth key at /etc/ts-auth-key (placed manually in Stage 2, not in git).
+  services.tailscale = {
+    enable = true;
+    authKeyFile = "/etc/ts-auth-key";
+    extraUpFlags = [ "--hostname=cc-ci-orchestrator" ];
+  };
+
+  # 4 GB disk swap — claude session memory safety net (2 GB RAM is tight for 3+ sessions).
+  swapDevices = [ { device = "/swapfile"; size = 4096; } ];
+
+  # nix-ld — lets the standalone Claude Code CLI (foreign dynamic ELF / Bun) run on NixOS.
+  programs.nix-ld.enable = true;
+  programs.nix-ld.libraries = with pkgs; [
+    stdenv.cc.cc.lib
+    zlib
+    openssl
+    curl
+    glibc
+  ];
+
+  environment.systemPackages = with pkgs; [
+    git tmux python3 jq curl cacert
+    gnused gawk coreutils gnugrep findutils util-linux
+    nettools openssh
+    age sops                # key management (same toolchain as cc-ci server)
+  ];
+
+  # loops user — claude sessions run as non-root (--dangerously-skip-permissions blocked for root).
+  users.users.loops = {
+    isNormalUser = true;
+    home = "/home/loops";
+    shell = pkgs.bash;
+    extraGroups = [ "wheel" ];
+  };
+  security.sudo.wheelNeedsPassword = false;
+  security.sudo.extraRules = [{
+    users = [ "loops" ];
+    commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
+  }];
+
+  # Ensure /home/loops/.local/bin (claude + opencode) is on the loops user PATH.
+  # opencode binary is installed there manually (not yet in nixpkgs); re-install if missing:
+  #   curl -sL https://github.com/anomalyco/opencode/releases/download/v1.15.13/opencode-linux-x64.tar.gz \
+  #     | tar -xz -C /home/loops/.local/bin opencode && chmod +x /home/loops/.local/bin/opencode
+  environment.variables.PATH = lib.mkForce
+    "/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
+
+  # SSH config for the loops user — points to the cc-ci Hetzner server via tailnet.
+  # HostName is the Hetzner cc-ci server's tailnet IP (cutover settled 2026-05-31).
+  system.activationScripts.loopsSshConfig = ''
+    mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
+    # Only write if not already present (preserves manual customisation).
+    if [ ! -f /home/loops/.ssh/config ]; then
+      cat > /home/loops/.ssh/config <<'SSHCFG'
+Host cc-ci
+    HostName 100.95.31.88
+    User root
+    IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
+    IdentitiesOnly yes
+    StrictHostKeyChecking accept-new
+    ServerAliveInterval 30
+SSHCFG
+      chmod 600 /home/loops/.ssh/config
+      chown loops:users /home/loops/.ssh/config
+    fi
+  '';
+
+  # claude-install — fetch the standalone Claude Code CLI for the loops user if missing.
+  systemd.services.claude-install = {
+    description = "Install Claude Code CLI for loops user (idempotent)";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
+    script = ''
+      if [ ! -x "$HOME/.local/bin/claude" ]; then
+        echo "installing Claude Code CLI for loops user..."
+        curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
+      fi
+    '';
+  };
+
+  # opencode web server — one shared instance; agent sessions attach to it for web visibility.
+  # Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
+  # Public provider configuration is tracked in the cc-ci repo; credentials
+  # remain in /srv/cc-ci/.testenv.
+  systemd.services.opencode-web = {
+    description = "opencode web server for cc-ci agents";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "tailscaled.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "simple";
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
+      EnvironmentFile = [
+        "-/srv/cc-ci/cc-ci/.env.public"
+        "/srv/cc-ci/.testenv"
+      ];
+      ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
+      ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
+      Restart = "on-failure";
+      RestartSec = "5s";
+    };
+    environment = {
+      HOME = "/home/loops";
+      PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/home/loops/.nix-profile/bin:/nix/profile/bin:/home/loops/.local/state/nix/profile/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
+    };
+    path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
+  };
+
+  # nginx — reverse-proxy oc.commoninternet.net → opencode web server.
+  # Bound to the tailscale IP so it is only reachable on the tailnet.
+  # DNS: add A record  oc.commoninternet.net → 100.84.190.30  (operator step if hostname access is wanted).
+  services.nginx = {
+    enable = true;
+    recommendedProxySettings = true;
+    virtualHosts."oc.commoninternet.net" = {
+      # Listen on the tailscale interface only — not the public IP. Both 80 and 443.
+      # 443 uses a SELF-SIGNED cert (below): this name resolves to a CGNAT tailscale
+      # IP, so Let's Encrypt HTTP-01 can't validate it and there is no DNS-01 provider
+      # configured on this host. The trusted *.ci.commoninternet.net wildcard lives on
+      # the coop-cloud swarm (91.98.47.73), not here, and doesn't cover this label.
+      # Self-signed is fine because the vhost is tailnet-only (trusted network); the
+      # browser shows a one-time trust prompt. (Chosen by operator, 2026-08-03.)
+      #
+      # The cert/key are created out of band — same convention as oc-htpasswd, NOT in
+      # git and NOT in the nix store (a store path would be world-readable):
+      #
+      #   /etc/nginx/oc-selfsigned.crt    root:nginx 0644
+      #   /etc/nginx/oc-selfsigned.key    root:nginx 0640
+      #
+      # Regenerate (10y, SANs = hostname + tailscale IP + ts.net name) with:
+      #   openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
+      #     -keyout /etc/nginx/oc-selfsigned.key -out /etc/nginx/oc-selfsigned.crt \
+      #     -subj /CN=oc.commoninternet.net \
+      #     -addext "subjectAltName=DNS:oc.commoninternet.net,DNS:cc-ci-orchestrator-1.taila4a0bf.ts.net,IP:100.84.190.30"
+      #   sudo chown root:nginx /etc/nginx/oc-selfsigned.{crt,key}
+      #   sudo chmod 644 /etc/nginx/oc-selfsigned.crt && sudo chmod 640 /etc/nginx/oc-selfsigned.key
+      #
+      # NOTE: like oc-htpasswd, if these files go missing nginx fails to START — which
+      # would take the atproto vhost down with it. Recreate them before rebuilding on
+      # a fresh host.
+      listen = [
+        { addr = "100.84.190.30"; port = 80;  ssl = false; }
+        { addr = "100.84.190.30"; port = 443; ssl = true;  }
+      ];
+      # addSSL (serve BOTH http+https) is required, not cosmetic: the NixOS nginx
+      # module only renders the `ssl_certificate` directives when a vhost is flagged
+      # as an SSL vhost. An explicit `listen ... ssl` + sslCertificate WITHOUT this
+      # flag produces an SSL listener with no cert → `nginx -t` fails and the whole
+      # service (atproto included) won't start.
+      addSSL = true;
+      sslCertificate    = "/etc/nginx/oc-selfsigned.crt";
+      sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
+
+      # HTTP basic auth in front of opencode. The opencode web UI has no
+      # authentication of its own and can drive agent sessions, so since this host
+      # started serving 80/443 publicly (atproto-likes, 2026-08-01) it is worth a
+      # second layer even though this vhost is tailnet-only.
+      #
+      # basicAuthFile, NOT basicAuth: the `basicAuth` attrset writes the password
+      # into the nix store, which is world-readable. This file is created out of
+      # band — same convention as /etc/ts-auth-key — and is not in git:
+      #
+      #   /etc/nginx/oc-htpasswd          root:nginx 0640, `oc:<bcrypt hash>`
+      #   /secrets/files/oc-basic-auth.txt the plaintext, loops-only 0600
+      #
+      # Rotate with:
+      #   P=$(python3 -c "import secrets,string;a=string.ascii_letters+string.digits;print(''.join(secrets.choice(a) for _ in range(32)))")
+      #   printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" | sudo tee /etc/nginx/oc-htpasswd
+      #   sudo chown root:nginx /etc/nginx/oc-htpasswd && sudo chmod 640 /etc/nginx/oc-htpasswd
+      #   sudo systemctl reload nginx
+      #
+      # NOTE: if this file goes missing, nginx fails to START — which would take
+      # the atproto vhost down with it. Recreate it before rebuilding on a fresh
+      # host.
+      basicAuthFile = "/etc/nginx/oc-htpasswd";
+
+      locations."/" = {
+        proxyPass = "http://127.0.0.1:4096";
+        proxyWebsockets = true;
+      };
+    };
+  };
+
+
+
+  # p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
+  # (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
+  # Control session + watchdog + pipeline via `engine/agents.py up`. Added 2026-07-08 after a reboot
+  # (Hetzner rollback) left this orchestrator down while cc-ci-loops auto-recovered.
+  # NOTE: still points at the /home path — will be re-pointed to /srv when that migration happens.
+  systemd.services.lichen-orchestrator = {
+    description = "p-lichen-orchestrator (lichen.page testing) — orchestrator + watchdog + pipeline";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
+      # default (control-group) systemd kills every leftover process in the cgroup when the unit
+      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
+      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
+      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/lichen-orchestrator";
+    };
+    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      proj="/srv/lichen-orchestrator"
+      # boot marker (best-effort; boot_id-gated logging can be added later)
+      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — lichen-orchestrator up" \
+        >> "$proj/.ao-state/boot.log" 2>/dev/null || true
+      cd "$proj" && python3 engine/agents.py up || echo "p-lichen agents.py up failed"
+    '';
+  };
+
+  # project-orchestrator (fleet manager) — always-on so the operator can reach it over Remote
+  # Control at any time (operator 2026-08-01). Same shape as lichen-orchestrator above; the PO's
+  # own agents.toml declares NO `wake`, so the watchdog only heals a dead session — it never sends
+  # periodic prompts. Starting it is `agents.py up`; that also starts its watchdog.
+  systemd.services.project-orchestrator = {
+    description = "project-orchestrator (fleet manager) — PO agent + watchdog, remote-control always up";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
+      # default (control-group) systemd kills every leftover process in the cgroup when the unit
+      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
+      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
+      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/project-orchestrator";
+    };
+    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      proj="/srv/project-orchestrator"
+      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — project-orchestrator up" \
+        >> "$proj/.ao-state/boot.log" 2>/dev/null || true
+      cd "$proj" && python3 engine/agents.py up || echo "PO agents.py up failed"
+    '';
+  };
+
+
+
+
+}
@@ -0,0 +1,33 @@
+{ modulesPath, ... }:
+{
+  imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
+  boot.loader = {
+    efi.efiSysMountPoint = "/boot/efi";
+    grub = {
+      efiSupport = true;
+      efiInstallAsRemovable = true;
+      device = "nodev";
+    };
+  };
+  fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/39A5-C7B9"; fsType = "vfat"; };
+  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
+  boot.initrd.kernelModules = [ "nvme" ];
+  fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
+
+  # 150G Hetzner volume (scsi-0HC_Volume_106342723) — holds the Rust build trees, which repeatedly
+  # filled the 75G root. A full disk here does NOT error: it silently truncates whatever is being
+  # written (it destroyed a plan file mid-write before anyone noticed).
+  #
+  # by-uuid, not /dev/sdb: device names are not stable across reboots and attaching another volume
+  # would silently mount the wrong disk here.
+  #
+  # nofail is REQUIRED, not decoration: without it, a detached/failed volume makes the mount unit a
+  # boot dependency, systemd drops to emergency mode, and the box comes up WITHOUT SSH. That failure
+  # has already happened once on this host from a bad nixos config, and it needed a recovery to the
+  # previous generation. The build cache is not worth risking access to the machine.
+  fileSystems."/mnt/data" = {
+    device = "/dev/disk/by-uuid/16cd6650-1399-4cb9-a696-54b70d83203a";
+    fsType = "ext4";
+    options = [ "defaults" "nofail" "x-systemd.device-timeout=10s" ];
+  };
+}
@@ -0,0 +1,130 @@
+# ⚠️ HISTORICAL / DEAD CONFIG — DO NOT DEPLOY. (superseded 2026-05-31)
+# This is the OLD Incus VM (`cc-ci-orchestrator`, tailnet 100.116.55.106) config. The orchestrator
+# now runs on Hetzner — the LIVE config is nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix.
+# Kept only for historical comparison; it runs loops as root and hard-codes the dead cc-ci IP.
+# Delete once the old Incus VM is torn down.
+#
+# cc-ci-orchestrator VM — NixOS config (channel-based: nixos-24.11; deployed to /etc/nixos/configuration.nix)
+#
+# Purpose: a reboot-resilient host for the cc-ci Builder/Adversary loops + watchdog + the orchestrator
+# session, moved off the unstable 905 MiB Pi. See plan-orchestrator-migration.md.
+#
+# STATUS: DRAFT (Phase B). The nix-ld + claude-install bits need on-VM validation (the standalone
+# Claude Code is a Bun ELF binary; NixOS needs nix-ld to run a foreign dynamic binary). The
+# cc-ci-loops supervisor service is defined but NOT enabled until the workspace is staged (Phase C/D).
+{ config, pkgs, lib, modulesPath, ... }:
+{
+  imports = [
+    "${modulesPath}/virtualisation/incus-virtual-machine.nix"
+  ];
+
+  # --- base (mirrors the incus-base-vm) ---
+  virtualisation.incus.agent.enable = true;            # for `incus exec`
+  services.cloud-init = { enable = true; network.enable = true; };
+  services.openssh = { enable = true; settings.PermitRootLogin = "yes"; };
+  networking.useDHCP = true;
+  networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
+  networking.firewall = { enable = true; trustedInterfaces = [ "tailscale0" ]; allowedTCPPorts = [ 22 ]; };
+  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  system.stateVersion = "24.11";
+
+  # --- tailscale (auto-auth from /etc/ts-auth-key, hostname from /etc/ts-hostname; written by cloud-init) ---
+  services.tailscale = {
+    enable = true;
+    authKeyFile = "/etc/ts-auth-key";
+    extraUpFlags = let h = lib.strings.removeSuffix "\n" (builtins.readFile /etc/ts-hostname);
+                   in [ "--hostname=${h}" "--ssh" ];   # --ssh: allow tailscale-ssh as a fallback path
+  };
+
+  # --- swap: the Pi OOM lesson. 2 GB RAM is tight for 3 concurrent claude sessions; 4 GB disk swap
+  #     as a real overflow tier (zram is in-RAM and doesn't add capacity). ---
+  swapDevices = [ { device = "/swapfile"; size = 4096; } ];
+
+  # --- nix-ld: lets the standalone Claude Code (foreign dynamic ELF / Bun) run on NixOS ---
+  programs.nix-ld.enable = true;
+  programs.nix-ld.libraries = with pkgs; [
+    stdenv.cc.cc.lib   # libstdc++ / libgcc_s
+    zlib
+    openssl
+    curl
+    glibc
+  ];
+
+  # --- packages the loops + launch.sh + orchestrator need ---
+  environment.systemPackages = with pkgs; [
+    git tmux python3 jq curl cacert
+    gnused gawk coreutils gnugrep findutils util-linux
+    nettools openssh        # nc, ssh
+    docker-client           # `docker` CLI is not needed (deploys run on cc-ci), but handy for probes
+  ];
+
+  # --- loops user: non-root account for running claude (--dangerously-skip-permissions blocked for root) ---
+  users.users.loops = {
+    isNormalUser = true;
+    home = "/home/loops";
+    shell = pkgs.bash;
+    extraGroups = [ "wheel" ];          # sudo access
+  };
+  security.sudo.wheelNeedsPassword = false;   # passwordless sudo for wheel
+  # Allow loops user to use tmux/claude without a password prompt
+  security.sudo.extraRules = [{
+    users = [ "loops" ];
+    commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
+  }];
+
+  # --- root PATH: ensure ~/.local/bin (where the standalone claude binary lives) is on root's PATH ---
+  environment.variables.PATH = lib.mkForce "/root/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
+
+  # --- root ssh config: reach cc-ci DIRECTLY over the VM's own tailscale (this VM is a tailnet peer,
+  #     so NO SOCKS proxy is needed — unlike the Pi). Key staged at /root/.ssh/cc-ci-root-ed25519. ---
+  system.activationScripts.ccciSshConfig = ''
+    mkdir -p /root/.ssh && chmod 700 /root/.ssh
+    cat > /root/.ssh/config <<'SSHCFG'
+    Host cc-ci cc-nix-test 100.90.116.4
+        HostName 100.90.116.4
+        User root
+        IdentityFile /root/.ssh/cc-ci-root-ed25519
+        IdentitiesOnly yes
+        StrictHostKeyChecking accept-new
+        ServerAliveInterval 30
+    SSHCFG
+    chmod 600 /root/.ssh/config
+  '';
+
+  # --- claude-install: idempotent oneshot — fetch the standalone Claude Code CLI if missing.
+  #     Runs via nix-ld. (Auth is a one-time operator step: `claude auth login` — see migration plan.) ---
+  systemd.services.claude-install = {
+    description = "Install the standalone Claude Code CLI if missing (idempotent)";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "nix-ld.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
+    path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
+    script = ''
+      if [ ! -x /root/.local/bin/claude ]; then
+        echo "installing standalone Claude Code CLI..."
+        curl -fsSL https://claude.ai/install.sh | bash || echo "claude install failed (retry next activation)"
+      fi
+    '';
+  };
+
+  # --- cc-ci-loops supervisor (DEFINED, NOT YET ENABLED). Enabled in Phase D after the workspace
+  #     (/srv/cc-ci: launch.sh, plan, prompts, the loop clones, secrets) is staged. This is the
+  #     reboot-resilience fix: it runs launch.sh start on every boot. Mirrors the Pi's cc-ci-loops.service. ---
+  systemd.services.cc-ci-loops = {
+    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start, RESUME_PHASE)";
+    wantedBy = [ "multi-user.target" ];   # enabled: workspace staged (Phase C/D 2026-05-30)
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot"; RemainAfterExit = true; User = "root";
+      WorkingDirectory = "/srv/cc-ci";
+    };
+    environment = { RESUME_PHASE = "1"; HOME = "/root"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
+        echo "workspace not staged yet (/srv/cc-ci/cc-ci-plan/launch.sh missing) — skipping"
+    '';
+  };
+}