nix: one Hetzner host for the CI server + the orchestrator (#cc-ci)
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).
nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.
README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
Claude Fable 5.1
@@ -1,35 +0,0 @@
|
||||
# cc-ci-orchestrator
|
||||
|
||||
> ⚠️ **HISTORICAL.** This README describes the retired **Incus VM** (`100.116.55.106`). The
|
||||
> orchestrator now runs on **Hetzner** — the live host config is
|
||||
> `nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix`. See
|
||||
> `cc-ci-plan/plan-orchestrator-hetzner-migration.md` for the current setup. Kept for history.
|
||||
|
||||
NixOS config for the **`cc-ci-orchestrator`** Incus VM (b1, project `terraform-ci`, tailnet
|
||||
`100.116.55.106`) — the reboot-resilient host for the cc-ci Builder/Adversary loops + watchdog +
|
||||
orchestrator session, moved off the unstable 905 MiB Pi.
|
||||
|
||||
See `cc-ci-plan/plan-orchestrator-migration.md` for the full migration.
|
||||
|
||||
## Files
|
||||
- `configuration.nix` — the VM's NixOS config (channel-based, `nixos-24.11`). Deployed to
|
||||
`/etc/nixos/configuration.nix` on the VM. Provides: nix-ld (so the standalone Claude Code Bun binary
|
||||
runs), tmux/git/python/jq + tools, a 4 GB swapfile, direct ssh to cc-ci (the VM is a tailnet peer —
|
||||
no SOCKS proxy needed, unlike the Pi), an idempotent `claude-install` oneshot, and the
|
||||
`cc-ci-loops` supervisor service (defined, **enabled in Phase D** once the workspace is staged).
|
||||
|
||||
## Deploy (until this is wired to a flake/auto-pull)
|
||||
```
|
||||
# copy configuration.nix to the VM, then:
|
||||
ssh cc-ci-orchestrator 'nixos-rebuild switch' # or run detached: see below
|
||||
```
|
||||
Over the (currently flaky) Pi→VM link, run the rebuild **detached** on the VM so an ssh/proxy drop
|
||||
doesn't abort it, e.g. `systemd-run --unit=orch-rebuild --collect nixos-rebuild switch` then poll
|
||||
`journalctl -u orch-rebuild`.
|
||||
|
||||
## Status
|
||||
- Phase A: VM created (2 GB / 2 vCPU / 30 GB), on tailnet, ssh-able. ✅
|
||||
- Phase B: this config (DRAFT) — nix-ld/claude validation pending on the VM.
|
||||
- Operator step pending (Phase C): `claude auth login` on the VM (device-code; can't be scripted).
|
||||
- Secrets to stage (Phase C, out-of-band): `/srv/cc-ci/.testenv`, `~/.ssh/cc-ci-root-ed25519`,
|
||||
Incus mTLS certs, the sops master age key.
|
||||
@@ -1,135 +0,0 @@
|
||||
# atproto-likes — the "most-liked accounts" web UI, run as a docker-compose stack
|
||||
#
|
||||
# ⚠️ COPY. Canonical source:
|
||||
# /srv/project-orchestrator/projects/notplants-atproto/nix/atproto-likes.nix
|
||||
# Pure evaluation cannot import a path outside the flake tree. Re-copy after editing.
|
||||
#
|
||||
# Host-specific context that is deliberately NOT in the public project repo:
|
||||
# - the domain is fronted by the gateway (143.244.213.108); it forwards everything here
|
||||
# - this vhost lists 0.0.0.0 only so it never shares an nginx address group with
|
||||
# oc.commoninternet.net (opencode, on the tailscale address, no auth of its own)
|
||||
# - /srv/project-orchestrator/projects/notplants-atproto/tools/check-exposure.sh asserts
|
||||
# what this host serves publicly; run it after any nginx change
|
||||
# behind the host's nginx.
|
||||
#
|
||||
# Import this from the host configuration:
|
||||
# imports = [ /srv/project-orchestrator/projects/notplants-atproto/nix/atproto-likes.nix ];
|
||||
#
|
||||
# What it sets up:
|
||||
# * the Docker daemon
|
||||
# * a systemd unit that builds and runs docker-compose.yml from the project checkout
|
||||
# * an nginx vhost for atproto.commoninternet.net with a Let's Encrypt cert,
|
||||
# proxying to the container on 127.0.0.1:8731
|
||||
# * ports 80/443 opened in the firewall
|
||||
#
|
||||
# DNS / TLS: point the domain's A record at this host (or at a reverse proxy that
|
||||
# forwards to it). ACME HTTP-01 needs the challenge to reach this nginx, so if a
|
||||
# proxy fronts the domain it must forward /.well-known/acme-challenge/ through.
|
||||
# Until a certificate is issued, NixOS installs a self-signed placeholder so nginx
|
||||
# still starts.
|
||||
#
|
||||
# A proxy in front should preserve the Host header (nginx routes by server_name),
|
||||
# pass Upgrade/Connection through (the loading page is a websocket) and allow a
|
||||
# long read timeout (a cold scan can run for minutes).
|
||||
{ config, pkgs, lib, ... }:
|
||||
|
||||
let
|
||||
projectDir = "/srv/project-orchestrator/projects/notplants-atproto";
|
||||
domain = "atproto.commoninternet.net";
|
||||
hostPort = 8731; # must match the ports: mapping in docker-compose.yml
|
||||
compose = "${pkgs.docker-compose}/bin/docker-compose";
|
||||
in
|
||||
{
|
||||
virtualisation.docker = {
|
||||
enable = true;
|
||||
# Reclaim dangling images from repeated `--build` runs.
|
||||
autoPrune = { enable = true; dates = "weekly"; };
|
||||
};
|
||||
|
||||
# Build + run the compose stack. Type=oneshot with RemainAfterExit: compose
|
||||
# detaches, and the containers' own restart policy keeps them alive.
|
||||
systemd.services.atproto-likes = {
|
||||
description = "atproto-likes — most-liked-accounts web UI (docker compose)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "docker.service" "docker.socket" "network-online.target" ];
|
||||
requires = [ "docker.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
path = [ pkgs.docker pkgs.docker-compose ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
WorkingDirectory = projectDir;
|
||||
# A cold `--build` pulls python:3.12-slim and installs pip deps.
|
||||
TimeoutStartSec = "1800";
|
||||
# The container runs unprivileged as uid 10001 (see Dockerfile), but the
|
||||
# page cache is a host bind mount, so its ownership is the host's business.
|
||||
# Without this the app 500s on any actor whose cache files were written by
|
||||
# an earlier root-running container.
|
||||
ExecStartPre = [
|
||||
"${pkgs.coreutils}/bin/mkdir -p ${projectDir}/cache"
|
||||
"${pkgs.coreutils}/bin/chown -R 10001:10001 ${projectDir}/cache"
|
||||
];
|
||||
ExecStart = "${compose} up -d --build --remove-orphans";
|
||||
ExecStop = "${compose} down";
|
||||
Restart = "on-failure";
|
||||
RestartSec = "30s";
|
||||
};
|
||||
};
|
||||
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
virtualHosts.${domain} = {
|
||||
# addSSL, NOT forceSSL. If a reverse proxy fronts this domain and its
|
||||
# back-end hop is plain HTTP, a forced 301 to https://<domain>/ would
|
||||
# resolve straight back to that proxy — a redirect loop. Serving both
|
||||
# schemes lets whatever is in front decide where TLS terminates.
|
||||
addSSL = true;
|
||||
enableACME = true;
|
||||
# Listens on 0.0.0.0 only, and the reject-everything default server below
|
||||
# owns those addresses. nginx groups server blocks by the connection's local
|
||||
# address, so keeping this vhost on the public address alone means it never
|
||||
# shares an address group with anything else the host may serve on another
|
||||
# interface — no other service can be reached by sending this one an
|
||||
# unexpected Host header.
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:${toString hostPort}";
|
||||
# The loading page streams scan progress over a websocket.
|
||||
proxyWebsockets = true;
|
||||
# A cold scan can run for minutes with the socket open; the default 60s
|
||||
# proxy read timeout would cut the loading page off mid-flower.
|
||||
extraConfig = ''
|
||||
proxy_read_timeout 1800s;
|
||||
proxy_send_timeout 1800s;
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
# Strict default server for the public addresses. Without an explicit
|
||||
# default_server, nginx promotes the first server block in the group — so any
|
||||
# vhost added later silently becomes what answers unmatched, absent or spoofed
|
||||
# Host headers from the internet. This closes that door once: only names
|
||||
# deliberately served get a response.
|
||||
#
|
||||
# port 80 -> 444 (close the connection, no response at all)
|
||||
# port 443 -> ssl_reject_handshake, so an unknown SNI never even gets TLS
|
||||
virtualHosts."public-default-reject" = {
|
||||
default = true;
|
||||
serverName = null;
|
||||
rejectSSL = true;
|
||||
listen = [
|
||||
{ addr = "0.0.0.0"; port = 80; ssl = false; }
|
||||
{ addr = "0.0.0.0"; port = 443; ssl = true; }
|
||||
];
|
||||
extraConfig = "return 444;";
|
||||
};
|
||||
};
|
||||
|
||||
security.acme = {
|
||||
acceptTerms = true;
|
||||
defaults.email = "mfowler.email@protonmail.com"; # ACME contact
|
||||
};
|
||||
|
||||
# Public HTTP/HTTPS.
|
||||
networking.firewall.allowedTCPPorts = [ 80 443 ];
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
# ⚠️ HISTORICAL / DEAD CONFIG — DO NOT DEPLOY. (superseded 2026-05-31)
|
||||
# This is the OLD Incus VM (`cc-ci-orchestrator`, tailnet 100.116.55.106) config. The orchestrator
|
||||
# now runs on Hetzner — the LIVE config is nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix.
|
||||
# Kept only for historical comparison; it runs loops as root and hard-codes the dead cc-ci IP.
|
||||
# Delete once the old Incus VM is torn down.
|
||||
#
|
||||
# cc-ci-orchestrator VM — NixOS config (channel-based: nixos-24.11; deployed to /etc/nixos/configuration.nix)
|
||||
#
|
||||
# Purpose: a reboot-resilient host for the cc-ci Builder/Adversary loops + watchdog + the orchestrator
|
||||
# session, moved off the unstable 905 MiB Pi. See plan-orchestrator-migration.md.
|
||||
#
|
||||
# STATUS: DRAFT (Phase B). The nix-ld + claude-install bits need on-VM validation (the standalone
|
||||
# Claude Code is a Bun ELF binary; NixOS needs nix-ld to run a foreign dynamic binary). The
|
||||
# cc-ci-loops supervisor service is defined but NOT enabled until the workspace is staged (Phase C/D).
|
||||
{ config, pkgs, lib, modulesPath, ... }:
|
||||
{
|
||||
imports = [
|
||||
"${modulesPath}/virtualisation/incus-virtual-machine.nix"
|
||||
];
|
||||
|
||||
# --- base (mirrors the incus-base-vm) ---
|
||||
virtualisation.incus.agent.enable = true; # for `incus exec`
|
||||
services.cloud-init = { enable = true; network.enable = true; };
|
||||
services.openssh = { enable = true; settings.PermitRootLogin = "yes"; };
|
||||
networking.useDHCP = true;
|
||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||
networking.firewall = { enable = true; trustedInterfaces = [ "tailscale0" ]; allowedTCPPorts = [ 22 ]; };
|
||||
nix.settings.experimental-features = [ "nix-command" "flakes" ];
|
||||
system.stateVersion = "24.11";
|
||||
|
||||
# --- tailscale (auto-auth from /etc/ts-auth-key, hostname from /etc/ts-hostname; written by cloud-init) ---
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
authKeyFile = "/etc/ts-auth-key";
|
||||
extraUpFlags = let h = lib.strings.removeSuffix "\n" (builtins.readFile /etc/ts-hostname);
|
||||
in [ "--hostname=${h}" "--ssh" ]; # --ssh: allow tailscale-ssh as a fallback path
|
||||
};
|
||||
|
||||
# --- swap: the Pi OOM lesson. 2 GB RAM is tight for 3 concurrent claude sessions; 4 GB disk swap
|
||||
# as a real overflow tier (zram is in-RAM and doesn't add capacity). ---
|
||||
swapDevices = [ { device = "/swapfile"; size = 4096; } ];
|
||||
|
||||
# --- nix-ld: lets the standalone Claude Code (foreign dynamic ELF / Bun) run on NixOS ---
|
||||
programs.nix-ld.enable = true;
|
||||
programs.nix-ld.libraries = with pkgs; [
|
||||
stdenv.cc.cc.lib # libstdc++ / libgcc_s
|
||||
zlib
|
||||
openssl
|
||||
curl
|
||||
glibc
|
||||
];
|
||||
|
||||
# --- packages the loops + launch.sh + orchestrator need ---
|
||||
environment.systemPackages = with pkgs; [
|
||||
git tmux python3 jq curl cacert
|
||||
gnused gawk coreutils gnugrep findutils util-linux
|
||||
nettools openssh # nc, ssh
|
||||
docker-client # `docker` CLI is not needed (deploys run on cc-ci), but handy for probes
|
||||
];
|
||||
|
||||
# --- loops user: non-root account for running claude (--dangerously-skip-permissions blocked for root) ---
|
||||
users.users.loops = {
|
||||
isNormalUser = true;
|
||||
home = "/home/loops";
|
||||
shell = pkgs.bash;
|
||||
extraGroups = [ "wheel" ]; # sudo access
|
||||
};
|
||||
security.sudo.wheelNeedsPassword = false; # passwordless sudo for wheel
|
||||
# Allow loops user to use tmux/claude without a password prompt
|
||||
security.sudo.extraRules = [{
|
||||
users = [ "loops" ];
|
||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
||||
}];
|
||||
|
||||
# --- root PATH: ensure ~/.local/bin (where the standalone claude binary lives) is on root's PATH ---
|
||||
environment.variables.PATH = lib.mkForce "/root/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
||||
|
||||
# --- root ssh config: reach cc-ci DIRECTLY over the VM's own tailscale (this VM is a tailnet peer,
|
||||
# so NO SOCKS proxy is needed — unlike the Pi). Key staged at /root/.ssh/cc-ci-root-ed25519. ---
|
||||
system.activationScripts.ccciSshConfig = ''
|
||||
mkdir -p /root/.ssh && chmod 700 /root/.ssh
|
||||
cat > /root/.ssh/config <<'SSHCFG'
|
||||
Host cc-ci cc-nix-test 100.90.116.4
|
||||
HostName 100.90.116.4
|
||||
User root
|
||||
IdentityFile /root/.ssh/cc-ci-root-ed25519
|
||||
IdentitiesOnly yes
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 30
|
||||
SSHCFG
|
||||
chmod 600 /root/.ssh/config
|
||||
'';
|
||||
|
||||
# --- claude-install: idempotent oneshot — fetch the standalone Claude Code CLI if missing.
|
||||
# Runs via nix-ld. (Auth is a one-time operator step: `claude auth login` — see migration plan.) ---
|
||||
systemd.services.claude-install = {
|
||||
description = "Install the standalone Claude Code CLI if missing (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "nix-ld.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
|
||||
script = ''
|
||||
if [ ! -x /root/.local/bin/claude ]; then
|
||||
echo "installing standalone Claude Code CLI..."
|
||||
curl -fsSL https://claude.ai/install.sh | bash || echo "claude install failed (retry next activation)"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
# --- cc-ci-loops supervisor (DEFINED, NOT YET ENABLED). Enabled in Phase D after the workspace
|
||||
# (/srv/cc-ci: launch.sh, plan, prompts, the loop clones, secrets) is staged. This is the
|
||||
# reboot-resilience fix: it runs launch.sh start on every boot. Mirrors the Pi's cc-ci-loops.service. ---
|
||||
systemd.services.cc-ci-loops = {
|
||||
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start, RESUME_PHASE)";
|
||||
wantedBy = [ "multi-user.target" ]; # enabled: workspace staged (Phase C/D 2026-05-30)
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; RemainAfterExit = true; User = "root";
|
||||
WorkingDirectory = "/srv/cc-ci";
|
||||
};
|
||||
environment = { RESUME_PHASE = "1"; HOME = "/root"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
|
||||
echo "workspace not staged yet (/srv/cc-ci/cc-ci-plan/launch.sh missing) — skipping"
|
||||
'';
|
||||
};
|
||||
}
|
||||
@@ -1,305 +0,0 @@
|
||||
# cc-ci-orchestrator-hetzner — NixOS config for the Hetzner loops runtime host.
|
||||
#
|
||||
# Purpose: run the cc-ci Builder/Adversary/Watchdog loops + orchestrator/assistant sessions
|
||||
# on a Hetzner cpx11 (2 vCPU / 2 GB dedicated AMD / 40 GB NVMe), replacing the slow b1 Incus VM.
|
||||
#
|
||||
# Provision with terraform/ then converge with: nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner
|
||||
# See terraform/README.md for the full Stage 2 procedure.
|
||||
{ config, pkgs, lib, ... }:
|
||||
{
|
||||
# hardware.nix is the nixos-infect generated hardware-configuration.nix (see README Stage 2a).
|
||||
|
||||
# atproto-likes — the notplants-atproto "most-liked accounts" web UI, served at
|
||||
# atproto.commoninternet.net. Brings in Docker + a compose stack + an nginx vhost
|
||||
# with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
|
||||
# the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
|
||||
# because pure evaluation cannot import a path outside the flake.
|
||||
# cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
|
||||
# the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
|
||||
# a general agent box that also serves unrelated projects — can be reasoned about separately
|
||||
# from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
|
||||
imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];
|
||||
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
settings.PermitRootLogin = "yes";
|
||||
};
|
||||
|
||||
# Root SSH access — all keys from the current orchestrator VM's /root/.ssh/authorized_keys.
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526"
|
||||
];
|
||||
networking.useDHCP = true;
|
||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||
networking.firewall = {
|
||||
enable = true;
|
||||
trustedInterfaces = [ "tailscale0" ];
|
||||
# Port 80 open only on the tailscale interface (trusted) — nginx binds there for oc.commoninternet.net.
|
||||
allowedTCPPorts = [ 22 ];
|
||||
};
|
||||
nix.settings.experimental-features = [ "nix-command" "flakes" ];
|
||||
system.stateVersion = "24.11";
|
||||
|
||||
# Tailscale — auth key at /etc/ts-auth-key (placed manually in Stage 2, not in git).
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
authKeyFile = "/etc/ts-auth-key";
|
||||
extraUpFlags = [ "--hostname=cc-ci-orchestrator" ];
|
||||
};
|
||||
|
||||
# 4 GB disk swap — claude session memory safety net (2 GB RAM is tight for 3+ sessions).
|
||||
swapDevices = [ { device = "/swapfile"; size = 4096; } ];
|
||||
|
||||
# nix-ld — lets the standalone Claude Code CLI (foreign dynamic ELF / Bun) run on NixOS.
|
||||
programs.nix-ld.enable = true;
|
||||
programs.nix-ld.libraries = with pkgs; [
|
||||
stdenv.cc.cc.lib
|
||||
zlib
|
||||
openssl
|
||||
curl
|
||||
glibc
|
||||
];
|
||||
|
||||
environment.systemPackages = with pkgs; [
|
||||
git tmux python3 jq curl cacert
|
||||
gnused gawk coreutils gnugrep findutils util-linux
|
||||
nettools openssh
|
||||
age sops # key management (same toolchain as cc-ci server)
|
||||
];
|
||||
|
||||
# loops user — claude sessions run as non-root (--dangerously-skip-permissions blocked for root).
|
||||
users.users.loops = {
|
||||
isNormalUser = true;
|
||||
home = "/home/loops";
|
||||
shell = pkgs.bash;
|
||||
extraGroups = [ "wheel" ];
|
||||
};
|
||||
security.sudo.wheelNeedsPassword = false;
|
||||
security.sudo.extraRules = [{
|
||||
users = [ "loops" ];
|
||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
||||
}];
|
||||
|
||||
# Ensure /home/loops/.local/bin (claude + opencode) is on the loops user PATH.
|
||||
# opencode binary is installed there manually (not yet in nixpkgs); re-install if missing:
|
||||
# curl -sL https://github.com/anomalyco/opencode/releases/download/v1.15.13/opencode-linux-x64.tar.gz \
|
||||
# | tar -xz -C /home/loops/.local/bin opencode && chmod +x /home/loops/.local/bin/opencode
|
||||
environment.variables.PATH = lib.mkForce
|
||||
"/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
||||
|
||||
# SSH config for the loops user — points to the cc-ci Hetzner server via tailnet.
|
||||
# HostName is the Hetzner cc-ci server's tailnet IP (cutover settled 2026-05-31).
|
||||
system.activationScripts.loopsSshConfig = ''
|
||||
mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
|
||||
# Only write if not already present (preserves manual customisation).
|
||||
if [ ! -f /home/loops/.ssh/config ]; then
|
||||
cat > /home/loops/.ssh/config <<'SSHCFG'
|
||||
Host cc-ci
|
||||
HostName 100.95.31.88
|
||||
User root
|
||||
IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
|
||||
IdentitiesOnly yes
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 30
|
||||
SSHCFG
|
||||
chmod 600 /home/loops/.ssh/config
|
||||
chown loops:users /home/loops/.ssh/config
|
||||
fi
|
||||
'';
|
||||
|
||||
# claude-install — fetch the standalone Claude Code CLI for the loops user if missing.
|
||||
systemd.services.claude-install = {
|
||||
description = "Install Claude Code CLI for loops user (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
|
||||
script = ''
|
||||
if [ ! -x "$HOME/.local/bin/claude" ]; then
|
||||
echo "installing Claude Code CLI for loops user..."
|
||||
curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
# opencode web server — one shared instance; agent sessions attach to it for web visibility.
|
||||
# Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
|
||||
# Public provider configuration is tracked in the cc-ci repo; credentials
|
||||
# remain in /srv/cc-ci/.testenv.
|
||||
systemd.services.opencode-web = {
|
||||
description = "opencode web server for cc-ci agents";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "simple";
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
||||
EnvironmentFile = [
|
||||
"-/srv/cc-ci/cc-ci/.env.public"
|
||||
"/srv/cc-ci/.testenv"
|
||||
];
|
||||
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
||||
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
||||
Restart = "on-failure";
|
||||
RestartSec = "5s";
|
||||
};
|
||||
environment = {
|
||||
HOME = "/home/loops";
|
||||
PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/home/loops/.nix-profile/bin:/nix/profile/bin:/home/loops/.local/state/nix/profile/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
|
||||
};
|
||||
path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
|
||||
};
|
||||
|
||||
# nginx — reverse-proxy oc.commoninternet.net → opencode web server.
|
||||
# Bound to the tailscale IP so it is only reachable on the tailnet.
|
||||
# DNS: add A record oc.commoninternet.net → 100.84.190.30 (operator step if hostname access is wanted).
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
virtualHosts."oc.commoninternet.net" = {
|
||||
# Listen on the tailscale interface only — not the public IP. Both 80 and 443.
|
||||
# 443 uses a SELF-SIGNED cert (below): this name resolves to a CGNAT tailscale
|
||||
# IP, so Let's Encrypt HTTP-01 can't validate it and there is no DNS-01 provider
|
||||
# configured on this host. The trusted *.ci.commoninternet.net wildcard lives on
|
||||
# the coop-cloud swarm (91.98.47.73), not here, and doesn't cover this label.
|
||||
# Self-signed is fine because the vhost is tailnet-only (trusted network); the
|
||||
# browser shows a one-time trust prompt. (Chosen by operator, 2026-08-03.)
|
||||
#
|
||||
# The cert/key are created out of band — same convention as oc-htpasswd, NOT in
|
||||
# git and NOT in the nix store (a store path would be world-readable):
|
||||
#
|
||||
# /etc/nginx/oc-selfsigned.crt root:nginx 0644
|
||||
# /etc/nginx/oc-selfsigned.key root:nginx 0640
|
||||
#
|
||||
# Regenerate (10y, SANs = hostname + tailscale IP + ts.net name) with:
|
||||
# openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
|
||||
# -keyout /etc/nginx/oc-selfsigned.key -out /etc/nginx/oc-selfsigned.crt \
|
||||
# -subj /CN=oc.commoninternet.net \
|
||||
# -addext "subjectAltName=DNS:oc.commoninternet.net,DNS:cc-ci-orchestrator-1.taila4a0bf.ts.net,IP:100.84.190.30"
|
||||
# sudo chown root:nginx /etc/nginx/oc-selfsigned.{crt,key}
|
||||
# sudo chmod 644 /etc/nginx/oc-selfsigned.crt && sudo chmod 640 /etc/nginx/oc-selfsigned.key
|
||||
#
|
||||
# NOTE: like oc-htpasswd, if these files go missing nginx fails to START — which
|
||||
# would take the atproto vhost down with it. Recreate them before rebuilding on
|
||||
# a fresh host.
|
||||
listen = [
|
||||
{ addr = "100.84.190.30"; port = 80; ssl = false; }
|
||||
{ addr = "100.84.190.30"; port = 443; ssl = true; }
|
||||
];
|
||||
# addSSL (serve BOTH http+https) is required, not cosmetic: the NixOS nginx
|
||||
# module only renders the `ssl_certificate` directives when a vhost is flagged
|
||||
# as an SSL vhost. An explicit `listen ... ssl` + sslCertificate WITHOUT this
|
||||
# flag produces an SSL listener with no cert → `nginx -t` fails and the whole
|
||||
# service (atproto included) won't start.
|
||||
addSSL = true;
|
||||
sslCertificate = "/etc/nginx/oc-selfsigned.crt";
|
||||
sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
|
||||
|
||||
# HTTP basic auth in front of opencode. The opencode web UI has no
|
||||
# authentication of its own and can drive agent sessions, so since this host
|
||||
# started serving 80/443 publicly (atproto-likes, 2026-08-01) it is worth a
|
||||
# second layer even though this vhost is tailnet-only.
|
||||
#
|
||||
# basicAuthFile, NOT basicAuth: the `basicAuth` attrset writes the password
|
||||
# into the nix store, which is world-readable. This file is created out of
|
||||
# band — same convention as /etc/ts-auth-key — and is not in git:
|
||||
#
|
||||
# /etc/nginx/oc-htpasswd root:nginx 0640, `oc:<bcrypt hash>`
|
||||
# /secrets/files/oc-basic-auth.txt the plaintext, loops-only 0600
|
||||
#
|
||||
# Rotate with:
|
||||
# P=$(python3 -c "import secrets,string;a=string.ascii_letters+string.digits;print(''.join(secrets.choice(a) for _ in range(32)))")
|
||||
# printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" | sudo tee /etc/nginx/oc-htpasswd
|
||||
# sudo chown root:nginx /etc/nginx/oc-htpasswd && sudo chmod 640 /etc/nginx/oc-htpasswd
|
||||
# sudo systemctl reload nginx
|
||||
#
|
||||
# NOTE: if this file goes missing, nginx fails to START — which would take
|
||||
# the atproto vhost down with it. Recreate it before rebuilding on a fresh
|
||||
# host.
|
||||
basicAuthFile = "/etc/nginx/oc-htpasswd";
|
||||
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:4096";
|
||||
proxyWebsockets = true;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
|
||||
# p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
|
||||
# (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
|
||||
# Control session + watchdog + pipeline via `engine/agents.py up`. Added 2026-07-08 after a reboot
|
||||
# (Hetzner rollback) left this orchestrator down while cc-ci-loops auto-recovered.
|
||||
# NOTE: still points at the /home path — will be re-pointed to /srv when that migration happens.
|
||||
systemd.services.lichen-orchestrator = {
|
||||
description = "p-lichen-orchestrator (lichen.page testing) — orchestrator + watchdog + pipeline";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/lichen-orchestrator";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/lichen-orchestrator"
|
||||
# boot marker (best-effort; boot_id-gated logging can be added later)
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — lichen-orchestrator up" \
|
||||
>> "$proj/.ao-state/boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 engine/agents.py up || echo "p-lichen agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
# project-orchestrator (fleet manager) — always-on so the operator can reach it over Remote
|
||||
# Control at any time (operator 2026-08-01). Same shape as lichen-orchestrator above; the PO's
|
||||
# own agents.toml declares NO `wake`, so the watchdog only heals a dead session — it never sends
|
||||
# periodic prompts. Starting it is `agents.py up`; that also starts its watchdog.
|
||||
systemd.services.project-orchestrator = {
|
||||
description = "project-orchestrator (fleet manager) — PO agent + watchdog, remote-control always up";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
|
||||
# default (control-group) systemd kills every leftover process in the cgroup when the unit
|
||||
# stops — and since one tmux server hosts every agent session on this host, a rebuild that
|
||||
# merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
|
||||
# exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
|
||||
KillMode = "process";
|
||||
Type = "oneshot"; RemainAfterExit = true;
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/project-orchestrator";
|
||||
};
|
||||
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
|
||||
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
|
||||
script = ''
|
||||
export PATH="/home/loops/.local/bin:$PATH"
|
||||
proj="/srv/project-orchestrator"
|
||||
echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — project-orchestrator up" \
|
||||
>> "$proj/.ao-state/boot.log" 2>/dev/null || true
|
||||
cd "$proj" && python3 engine/agents.py up || echo "PO agents.py up failed"
|
||||
'';
|
||||
};
|
||||
|
||||
|
||||
|
||||
|
||||
}
|
||||
@@ -1,33 +0,0 @@
|
||||
{ modulesPath, ... }:
|
||||
{
|
||||
imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
|
||||
boot.loader = {
|
||||
efi.efiSysMountPoint = "/boot/efi";
|
||||
grub = {
|
||||
efiSupport = true;
|
||||
efiInstallAsRemovable = true;
|
||||
device = "nodev";
|
||||
};
|
||||
};
|
||||
fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/39A5-C7B9"; fsType = "vfat"; };
|
||||
boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
|
||||
boot.initrd.kernelModules = [ "nvme" ];
|
||||
fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
|
||||
|
||||
# 150G Hetzner volume (scsi-0HC_Volume_106342723) — holds the Rust build trees, which repeatedly
|
||||
# filled the 75G root. A full disk here does NOT error: it silently truncates whatever is being
|
||||
# written (it destroyed a plan file mid-write before anyone noticed).
|
||||
#
|
||||
# by-uuid, not /dev/sdb: device names are not stable across reboots and attaching another volume
|
||||
# would silently mount the wrong disk here.
|
||||
#
|
||||
# nofail is REQUIRED, not decoration: without it, a detached/failed volume makes the mount unit a
|
||||
# boot dependency, systemd drops to emergency mode, and the box comes up WITHOUT SSH. That failure
|
||||
# has already happened once on this host from a bad nixos config, and it needed a recovery to the
|
||||
# previous generation. The build cache is not worth risking access to the machine.
|
||||
fileSystems."/mnt/data" = {
|
||||
device = "/dev/disk/by-uuid/16cd6650-1399-4cb9-a696-54b70d83203a";
|
||||
fsType = "ext4";
|
||||
options = [ "defaults" "nofail" "x-systemd.device-timeout=10s" ];
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
# cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
|
||||
#
|
||||
# This file is only what is physical or identity about the machine: hardware, networking, the
|
||||
# tailscale node, root SSH keys, swap, stateVersion. Everything functional comes from modules:
|
||||
# cc-ci.nixosModules.cc-ci-server recipe-maintainers/cc-ci — swarm, traefik, drone,
|
||||
# runner, bridge, dashboard, reports, acme-dns, harness
|
||||
# self.nixosModules.cc-ci-orchestrator nix/modules/cc-ci.nix — loops, orchestrator, timers
|
||||
# self.nixosModules.orchestrator-host nix/modules/orchestrator-host.nix — loops user, CLIs
|
||||
# See README.md for provisioning (Hetzner Debian → nixos-infect → this flake) and staging.
|
||||
{ lib, pkgs, ... }:
|
||||
{
|
||||
imports = [
|
||||
./hardware.nix
|
||||
./networking.nix
|
||||
];
|
||||
|
||||
networking.hostName = "cc-ci";
|
||||
|
||||
# ---- cc-ci server identity --------------------------------------------------------------
|
||||
# Public address: acme-dns binds to it and publishes it as the `ns-acme` glue record; the
|
||||
# Gandi A records for ci / *.ci / ns-acme .commoninternet.net point here.
|
||||
cc-ci.publicIPv4 = "195.201.88.249";
|
||||
# cc-ci is a plain flake input here (no private submodule), so the sops file is the one in
|
||||
# the deployed --recursive checkout the weekly sweep runs from (README "Stage the workspace").
|
||||
cc-ci.sopsFile = "/etc/cc-ci/secrets/secrets.yaml";
|
||||
|
||||
# ---- orchestrator identity --------------------------------------------------------------
|
||||
# The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
|
||||
cc-ci-orchestrator.ciSshHost = "127.0.0.1";
|
||||
|
||||
# ---- tailscale — auth key staged out of band at /etc/ts-auth-key -----------------------
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
authKeyFile = "/etc/ts-auth-key";
|
||||
extraUpFlags = [ "--hostname=cc-ci" ];
|
||||
};
|
||||
|
||||
# ---- ssh ----------------------------------------------------------------------------------
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
settings.PermitRootLogin = "yes";
|
||||
};
|
||||
# Root keys: PUBLIC keys, tracked deliberately in ./ssh-keys (one per line, blank lines ok).
|
||||
users.users.root.openssh.authorizedKeys.keys =
|
||||
builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
|
||||
# The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
|
||||
users.users.loops.openssh.authorizedKeys.keys =
|
||||
builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
|
||||
|
||||
# ---- firewall -------------------------------------------------------------------------------
|
||||
# 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. The tailscale
|
||||
# interface is trusted, which is what makes the opencode UI on 8443 tailnet-only.
|
||||
networking.firewall = {
|
||||
enable = true;
|
||||
trustedInterfaces = [ "tailscale0" ];
|
||||
allowedTCPPorts = [ 22 ];
|
||||
};
|
||||
networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
|
||||
|
||||
# ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------
|
||||
swapDevices = [ { device = "/swapfile"; size = 8192; } ];
|
||||
|
||||
# ssh client for root (the orchestrator's `ssh cc-ci` goes through the loops user's own config).
|
||||
environment.systemPackages = [ pkgs.openssh ];
|
||||
|
||||
# Fresh NixOS 26.05 install (nixos-infect, 2026-09-07). Never change this on an existing host.
|
||||
system.stateVersion = "26.05";
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
# PROVISIONAL — copied from the old cc-ci server (same Hetzner cloud layout) so the flake can be
|
||||
# evaluated before the new box yielded its own file. MUST be replaced by /etc/nixos/hardware-configuration.nix
|
||||
# from the new host (README §3): the ESP UUID below belongs to the OLD machine.
|
||||
# Hardware configuration for cc-ci on Hetzner Cloud (cpx32: AMD 4 vCPU / 8 GB / x86_64).
|
||||
# Generated by nixos-infect from a Debian 12 base image, then committed here.
|
||||
#
|
||||
# nixos-infect uses GRUB + EFI on Hetzner (not systemd-boot), with a qemu-guest profile
|
||||
# because Hetzner Cloud uses KVM virtualisation.
|
||||
#
|
||||
# IMPORTANT: networking.nix (below) contains the server's static public IP.
|
||||
# When provisioning a new server via `terraform apply`, copy the fresh networking.nix
|
||||
# from /etc/nixos/networking.nix on the new host and commit it here before rebuilding.
|
||||
{ modulesPath, ... }:
|
||||
{
|
||||
imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
|
||||
|
||||
boot = {
|
||||
loader = {
|
||||
efi.efiSysMountPoint = "/boot/efi";
|
||||
grub = {
|
||||
efiSupport = true;
|
||||
efiInstallAsRemovable = true;
|
||||
device = "nodev";
|
||||
};
|
||||
};
|
||||
initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
|
||||
initrd.kernelModules = [ "nvme" ];
|
||||
};
|
||||
|
||||
fileSystems."/boot/efi" = {
|
||||
device = "/dev/disk/by-uuid/D978-69EE";
|
||||
fsType = "vfat";
|
||||
};
|
||||
|
||||
fileSystems."/" = {
|
||||
device = "/dev/sda1";
|
||||
fsType = "ext4";
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
# PROVISIONAL — derived from the old cc-ci server networking.nix with the new address; replace with the
|
||||
# nixos-infect output of the new host (README §3), keeping defaultGateway.interface.
|
||||
# Hetzner static networking — generated by nixos-infect at provision time.
|
||||
#
|
||||
# This file is server-specific: the IP, gateway, and MAC address are tied to a
|
||||
# particular Hetzner instance. When provisioning a new server:
|
||||
# 1. After `terraform apply` + nixos-infect completes, run:
|
||||
# ssh root@<new-ip> 'cat /etc/nixos/networking.nix'
|
||||
# 2. Replace this file's contents with the output and commit.
|
||||
# 3. Then: `nixos-rebuild switch --flake .#cc-ci-hetzner --target-host root@<new-ip>`
|
||||
#
|
||||
# Current instance: 195.201.88.249 (fsn1, Hetzner server 134485294, provisioned 2026-05-31).
|
||||
{ lib, ... }: {
|
||||
networking = {
|
||||
nameservers = [
|
||||
"185.12.64.1"
|
||||
"185.12.64.2"
|
||||
];
|
||||
# The interface MUST be explicit here. Since NixOS 25.05 the scripted-networking
|
||||
# module installs the default route from the gateway interface's
|
||||
# network-addresses-<iface>.service, and it finds that interface either by
|
||||
# `defaultGateway.interface` or by the gateway address being inside one of the
|
||||
# interface's subnets. With Hetzner's off-subnet point-to-point gateway
|
||||
# (195.201.88.249/32 on eth0, gateway 172.31.1.1) neither matched when this was a
|
||||
# bare string, so NO default route was installed and the 26.05 rebuild on
|
||||
# 2026-08-03 took the host off the network (recovered via rescue mode).
|
||||
defaultGateway = {
|
||||
address = "172.31.1.1";
|
||||
interface = "eth0";
|
||||
};
|
||||
# No IPv6 on this Hetzner instance (link-local only) — nixos-infect emitted an empty
|
||||
# defaultGateway6/ipv6.route which made network-addresses-eth0.service fail
|
||||
# ("ip route add /128" with no prefix). v4-only box, so no IPv6 gateway/route declared.
|
||||
dhcpcd.enable = false;
|
||||
usePredictableInterfaceNames = lib.mkForce false;
|
||||
interfaces = {
|
||||
eth0 = {
|
||||
ipv4.addresses = [
|
||||
{ address = "195.201.88.249"; prefixLength = 32; }
|
||||
];
|
||||
ipv4.routes = [{ address = "172.31.1.1"; prefixLength = 32; }];
|
||||
};
|
||||
};
|
||||
};
|
||||
services.udev.extraRules = ''
|
||||
ATTR{address}=="00:00:00:00:00:00", NAME="eth0"
|
||||
'';
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZGp/DQTFuD1GvsyTzCVBUTmoWqcb5T+Z7zZo5nYLXO
|
||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhgo41nt8/L+Cr0PKd8jQK45mw/A+h041j6LQ8JWZisEVaQOzr6s9rxPL8VT5ML4P3/4bMblzdDiXWlJxymcb+yk5S5TnVrMavzHEDhWHwEvTRMe6xNTmsU6cmmhRw7PJqqQ+0GTlQalu3I4jkC0kTF7kuPwduUOgUuSpJqxvDTwYiXoyVnOQHAIygh+BmQvYUz0PBfQgIhgcbYmGZ++T0DnMzdGFzW2UB/iy5mymnpmbaZCgLy0w8AoDE+0YLtUc4gwTXc183nvqO1i7LQr+3jBYkv5ZthCCc52vXFHDSw9xZ5ohsOrBvoi5foRbqinmU5/t0aTK7SSrat7xXm/odIOyS+S7PJyeEcsXN6d5zdxbabAy5vLfodEaKGZd4rqQeDCxOTPAS/BlrBV/EV714n4E+fSOAllAuMBO4IibJM/gLJrh2Dql3co50QW9HEDeSC7iqp2lxRBDxvUs3rIEzy7o4HSN8chqBUK1bbBY6B17fuNHIpBAw4akRVVvPnVM= trav@trav480sweet
|
||||
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6jrKj7iZUNRLBTZG0vZM1D/BXtARhhB4+GrvpyuqmPb9iw2ifT9YqRUwgyGrOW9U6nIAR9yFnfp9+FkyhEKWByqEBbe/zYKlGLRGjfsIdDdW29QQ3hvmqNyboCkXLxZGat93poYhnoomqicmGD/xST4s0OUhcK9E494lUmenlD9dcMZW1aKpJ+9O4Dq6A7nk2z1e4KFcZdrZDI2Hgg+gfEdsKZQqd/R3Mls/eVKpzhfv3Y8BiNoHssUChVf8IGESqTOBOR7Dk7FsU5Z2ZcnQ1coxY7VlBn4fPjTWmz/Ac0jLqgcpCLpNyQzFPDVMYZKYrPVoqBeKVhN5YnfwR5OVP8YsakT/obLwC43sx/esXfjhVGcsRoGpiLOfazzNw/eC8s6FlS8cesOubEM37a7F25z4UEG3d487oM7EjQ39gBCCj/KRgUimCKMWsm6yIas4OSctBWEAo/NhZp0gwulSRxleW6eJCNNwzOmWjdzYIVWoVP0EIeM95Tq8PVUN7gpc= aadil@t480
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMyHSi12R0+HCVBz7+d9fyOBnoJi8Nsj5D7vQ9UQO8a5
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQFuqUB2qNZSDNjDsjjhVA/WnnQNVAMmsUscW6OgMDN
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS notplants-orchestrator
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
|
||||
+16
-16
@@ -1,15 +1,16 @@
|
||||
# cc-ci.nix — everything on this host that exists FOR cc-ci, and nothing else.
|
||||
# cc-ci.nix — the cc-ci ORCHESTRATOR: the Builder/Adversary loops supervisor, the operator's
|
||||
# steering session, and the weekly-upgrade + hourly-supervisor timers. Nothing else.
|
||||
#
|
||||
# Split out of the orchestrator host config on 2026-08-20. The host it runs on is a general
|
||||
# agent/orchestration box that also serves several unrelated projects; this module is the cc-ci
|
||||
# part of it, so that the two can evolve (and be reviewed) independently. It is exported from this
|
||||
# repo's flake as `nixosModules.cc-ci` and imported by whichever host runs cc-ci.
|
||||
# Exported from this repo's flake as `nixosModules.cc-ci-orchestrator` (and, for the host that
|
||||
# used to import it under the old name, `nixosModules.cc-ci`). Split out of the shared agent
|
||||
# host config on 2026-08-20; since 2026-09 it runs on the same Hetzner host as the CI server
|
||||
# itself (`#cc-ci` in flake.nix), next to recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
|
||||
#
|
||||
# All of it assumes the cc-ci workspaces exist on the host:
|
||||
# /srv/cc-ci the loops workspace (+ .cc-ci-logs, upgrader.env)
|
||||
# /srv/cc-ci-orch this repo (the orchestrator's own working dir)
|
||||
# /srv/cc-ci the loops workspace (+ .cc-ci-logs, upgrader.env) — a symlink to
|
||||
# /srv/cc-ci-orch this repo (the orchestrator's own working dir), with cc-ci/ checked out
|
||||
# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
|
||||
# those are host concerns, provided by the host config, not by this module.
|
||||
# those are host concerns, provided by nix/modules/orchestrator-host.nix, not by this module.
|
||||
{ config, pkgs, lib, ... }:
|
||||
{
|
||||
# cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
|
||||
@@ -48,15 +49,14 @@
|
||||
# cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
|
||||
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
|
||||
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
|
||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
|
||||
# the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
|
||||
# attaches to the shared opencode web server (opencode-web.service below) and is reachable for
|
||||
# Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
|
||||
# watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
|
||||
# supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
|
||||
# the same reboot-resilience the other two orchestrators already have.
|
||||
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
|
||||
# (backend/model chosen there — Claude Code under Remote Control since 2026-09-07; before that
|
||||
# opencode/glm-5.2 attached to the shared opencode web server, opencode-web.service in
|
||||
# orchestrator-host.nix, which the upgrader still uses). The harness watchdog (started by
|
||||
# `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
|
||||
# be killed just for idling). Added 2026-08-03 for reboot-resilience.
|
||||
systemd.services.cc-ci-orchestrator = {
|
||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
|
||||
description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
|
||||
# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
|
||||
# the standalone claude/opencode CLIs, the shared opencode web server and its tailnet-only UI,
|
||||
# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
|
||||
#
|
||||
# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
|
||||
# `nixosModules.cc-ci-orchestrator`; the combined CI-server + orchestrator host (`#cc-ci`) also
|
||||
# imports recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
|
||||
#
|
||||
# History: until 2026-09 this lived (twice, drifting) in nix/hosts/cc-ci-orchestrator-hetzner/
|
||||
# configuration.nix here and in notplants-nix's hosts/notplants-orchestrator/configuration.nix,
|
||||
# the shared agent box that also ran lichen + project-orchestrator. The cc-ci half moved to its
|
||||
# own host; this file is that half.
|
||||
{ config, lib, pkgs, ... }:
|
||||
let
|
||||
cfg = config.cc-ci-orchestrator;
|
||||
in
|
||||
{
|
||||
options.cc-ci-orchestrator = {
|
||||
ciSshHost = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "127.0.0.1";
|
||||
example = "100.95.31.88";
|
||||
description = ''
|
||||
Where `ssh cc-ci` (used by every skill and script that drives the CI server) connects to,
|
||||
as root with ~loops/.ssh/cc-ci-root-ed25519. On the combined host the CI server IS this
|
||||
machine, so the default is loopback; a standalone orchestrator points it at the CI
|
||||
server's tailnet address.
|
||||
'';
|
||||
};
|
||||
|
||||
opencodeUiPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 8443;
|
||||
description = ''
|
||||
TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host
|
||||
Traefik (docker swarm) owns 80/443. The port is not opened in the firewall, so it is
|
||||
reachable only over the trusted tailscale interface.
|
||||
'';
|
||||
};
|
||||
|
||||
opencodeUiHost = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "oc.commoninternet.net";
|
||||
description = "nginx server_name for the opencode web UI (self-signed, basic auth).";
|
||||
};
|
||||
};
|
||||
|
||||
config = {
|
||||
# ---- the loops user -------------------------------------------------------------------
|
||||
# claude sessions run as non-root (--dangerously-skip-permissions is refused for root).
|
||||
users.users.loops = {
|
||||
isNormalUser = true;
|
||||
home = "/home/loops";
|
||||
shell = pkgs.bash;
|
||||
extraGroups = [ "wheel" "docker" ];
|
||||
};
|
||||
security.sudo.wheelNeedsPassword = false;
|
||||
security.sudo.extraRules = [{
|
||||
users = [ "loops" ];
|
||||
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
|
||||
}];
|
||||
|
||||
# /home/loops/.local/bin holds the standalone claude + opencode binaries; it must be first on
|
||||
# every PATH (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
|
||||
environment.variables.PATH = lib.mkForce
|
||||
"/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
|
||||
|
||||
# ---- nix-ld: the standalone Claude Code / opencode CLIs are foreign dynamic ELF binaries ---
|
||||
programs.nix-ld.enable = true;
|
||||
programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
|
||||
|
||||
# ---- the toolbox every agent on this box gets ----------------------------------------
|
||||
# Bar for adding something: an agent doing ordinary work would otherwise waste a turn
|
||||
# discovering it is absent.
|
||||
environment.systemPackages = with pkgs; [
|
||||
git tmux python3 jq curl cacert
|
||||
gnused gawk coreutils gnugrep findutils util-linux nettools openssh
|
||||
age sops ssh-to-age
|
||||
wget gnutar gzip unzip zip xz
|
||||
ripgrep fd tree file less which
|
||||
procps psmisc htop lsof strace ncdu
|
||||
dnsutils socat netcat-gnu iproute2 iputils
|
||||
openssl gnumake gcc pkg-config
|
||||
yq-go diffutils patch rsync bubblewrap
|
||||
];
|
||||
|
||||
# ---- ssh config for the loops user: `ssh cc-ci` = the CI server (root) -----------------
|
||||
# Written only if absent so a manual customisation survives rebuilds.
|
||||
system.activationScripts.loopsSshConfig = ''
|
||||
mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
|
||||
if [ ! -f /home/loops/.ssh/config ]; then
|
||||
cat > /home/loops/.ssh/config <<'SSHCFG'
|
||||
Host cc-ci
|
||||
HostName ${cfg.ciSshHost}
|
||||
User root
|
||||
IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
|
||||
IdentitiesOnly yes
|
||||
StrictHostKeyChecking accept-new
|
||||
ServerAliveInterval 30
|
||||
|
||||
Host git.autonomic.zone
|
||||
HostName git.autonomic.zone
|
||||
Port 2222
|
||||
User git
|
||||
IdentityFile /home/loops/.ssh/autonomic-bot-gitea-ed25519
|
||||
IdentitiesOnly yes
|
||||
|
||||
Host tangled.org
|
||||
IdentityFile /home/loops/.ssh/tangled-ed25519
|
||||
IdentitiesOnly yes
|
||||
SSHCFG
|
||||
chmod 600 /home/loops/.ssh/config
|
||||
chown loops:users /home/loops/.ssh/config
|
||||
fi
|
||||
'';
|
||||
|
||||
# ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
|
||||
systemd.services.claude-install = {
|
||||
description = "Install Claude Code CLI for loops user (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
|
||||
script = ''
|
||||
if [ ! -x "$HOME/.local/bin/claude" ]; then
|
||||
echo "installing Claude Code CLI for loops user..."
|
||||
curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
systemd.services.opencode-install = {
|
||||
description = "Install opencode CLI for loops user (idempotent)";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
|
||||
environment = { HOME = "/home/loops"; };
|
||||
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
|
||||
script = ''
|
||||
if [ ! -x "$HOME/.local/bin/opencode" ]; then
|
||||
echo "installing opencode CLI for loops user..."
|
||||
curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation"
|
||||
fi
|
||||
'';
|
||||
};
|
||||
|
||||
# ---- opencode web server: one shared instance the opencode-backed agents attach to -------
|
||||
# Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
|
||||
systemd.services.opencode-web = {
|
||||
description = "opencode web server for cc-ci agents";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
after = [ "network-online.target" "tailscaled.service" "opencode-install.service" ];
|
||||
wants = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "simple";
|
||||
User = "loops"; Group = "users";
|
||||
WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
|
||||
EnvironmentFile = [ "-/srv/cc-ci/cc-ci/.env.public" "/srv/cc-ci/.testenv" ];
|
||||
ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
|
||||
ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
|
||||
Restart = "on-failure";
|
||||
RestartSec = "5s";
|
||||
};
|
||||
environment = {
|
||||
HOME = "/home/loops";
|
||||
PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
|
||||
};
|
||||
path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
|
||||
};
|
||||
|
||||
# ---- tailnet-only nginx front door for the opencode UI -------------------------------
|
||||
# Self-signed cert + basic auth, both created out of band (a store path would be world
|
||||
# readable) — see README "Secrets to stage". nginx FAILS TO START if they are missing.
|
||||
# /etc/nginx/oc-selfsigned.crt root:nginx 0644
|
||||
# /etc/nginx/oc-selfsigned.key root:nginx 0640
|
||||
# /etc/nginx/oc-htpasswd root:nginx 0640 (`oc:<bcrypt>`; plaintext in /secrets)
|
||||
services.nginx = {
|
||||
enable = true;
|
||||
recommendedProxySettings = true;
|
||||
virtualHosts.${cfg.opencodeUiHost} = {
|
||||
listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ];
|
||||
# onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener.
|
||||
onlySSL = true;
|
||||
sslCertificate = "/etc/nginx/oc-selfsigned.crt";
|
||||
sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
|
||||
basicAuthFile = "/etc/nginx/oc-htpasswd";
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:4096";
|
||||
proxyWebsockets = true;
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user