From 8f7f125fc07336392c19f1ad0cdc55dfbc691280 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Mon, 7 Sep 2026 20:45:59 +0000 Subject: [PATCH] cc-ci host: no tailscale; fail2ban; opencode UI public on 8443 with the LE wildcard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator 2026-09-07: the combined host stays off the tailnet. ssh is key-only (password + keyboard-interactive auth off), the opencode UI is served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's *.ci.commoninternet.net ACME cert (new orchestrator-host option opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail). tailscaled ordering dropped from the orchestrator units. README updated. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz --- README.md | 36 +++++++++++++------------ nix/hosts/cc-ci/configuration.nix | 44 ++++++++++++++++++++++--------- nix/modules/cc-ci.nix | 8 +++--- nix/modules/orchestrator-host.nix | 42 +++++++++++++++++++++-------- 4 files changed, 86 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index b95644b..c6d8b2b 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) rep | Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` | | The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` | | The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) | -| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` | +| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` | | Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) | | Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` | | How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` | @@ -31,7 +31,8 @@ can repeat it. Read it once before starting; the order matters. - A **Hetzner Cloud** project you can create servers in (console login or an API token). - **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The - public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). + public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password + login is disabled and fail2ban watches sshd, so a key is the only way in. - Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key, `autonomic-bot-gitea-ed25519`, has it). @@ -153,12 +154,11 @@ Commit on a branch; the rebuild in §5 can use the local checkout before the PR Everything in this section is **outside git**. Do it as root over SSH, in this order. -### 4a. Tailscale +### 4a. No tailscale -```bash -# a reusable (or fresh) tailnet auth key from the tailscale admin console -install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…' -``` +The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the +public IP, the CI front doors are public via traefik, and the opencode UI is public on 8443 +behind TLS + basic auth, all guarded by fail2ban (`nix/hosts/cc-ci/configuration.nix`). ### 4b. The CI server's checkout and its one out-of-band secret @@ -210,7 +210,7 @@ Then the files below (`install -m600 -o loops -g users`): | `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host | | `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) | | `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` | -| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) | +| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net:8443`, TLS from the CI wildcard cert); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:` line via `mkpasswd -m bcrypt` | `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` → `127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present. @@ -220,7 +220,7 @@ Then the files below (`install -m600 -o loops -g users`): From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo): ```bash -# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it). +# as root, detached (the activation restarts sshd; a dropped session must not kill it). # Three things the FIRST rebuild on a bare infect system needs, none of which the converged # host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect # system has none — hence nix-shell), HOME=/root (so root's `git config --global @@ -234,6 +234,9 @@ journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two `test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks networking or sshd a reboot from the Hetzner console lands on the last known-good generation. +(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation +restarted tailscale under the session; this one is plain public ssh, but detached is still the +safe habit.) Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout. The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge @@ -243,9 +246,9 @@ deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify: ```bash systemctl is-system-running # running — or list-units --failed and read journalctl -u -tailscale status | head -3 docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1 -systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns +systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban +fail2ban-client status sshd; fail2ban-client status nginx-http-auth systemctl list-timers 'cc-ci-*' nightly-sweep sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active) # the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once @@ -256,10 +259,8 @@ curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{h dig +short @ ns-acme.commoninternet.net # acme-dns answering on the public 53 ``` -Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` — -the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put -it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it -during the install; the box is reachable on its public IP throughout. +The opencode UI: `curl -sk --resolve oc.ci.commoninternet.net:8443: -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net:8443/` +→ 401 without credentials, 200 with `-u oc:`. When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot default). **If you are migrating from another host, do §6 before letting it serve anything**: right @@ -273,7 +274,7 @@ systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer ## 6. Migrating: restore state from the previous host -Over tailscale (`rsync -aHAX --numeric-ids root@: `), with the matching service +Over ssh between the hosts (`ssh root@ 'tar -C / -cf - ' | ssh root@ 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service stopped on the new host while its directory is copied: | path | holds | notes | @@ -305,7 +306,8 @@ stopped on the new host while its directory is copied: 5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator` (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count. -6. Keep the old host as a cold standby for a week, then delete it and its tailnet node. +6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the + old servers were on the tailnet; the new one is not). ## 8. Day 2 diff --git a/nix/hosts/cc-ci/configuration.nix b/nix/hosts/cc-ci/configuration.nix index d35a64b..fe4f44f 100644 --- a/nix/hosts/cc-ci/configuration.nix +++ b/nix/hosts/cc-ci/configuration.nix @@ -1,7 +1,7 @@ # cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator. # -# This file is only what is physical or identity about the machine: hardware, networking, the -# tailscale node, root SSH keys, swap, stateVersion. Everything functional comes from modules: +# This file is only what is physical or identity about the machine: hardware, networking, +# root SSH keys, firewall + fail2ban, swap, stateVersion. Everything functional comes from modules: # cc-ci.nixosModules.cc-ci-server recipe-maintainers/cc-ci — swarm, traefik, drone, # runner, bridge, dashboard, reports, acme-dns, harness # self.nixosModules.cc-ci-orchestrator nix/modules/cc-ci.nix — loops, orchestrator, timers @@ -27,13 +27,13 @@ # ---- orchestrator identity -------------------------------------------------------------- # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default). cc-ci-orchestrator.ciSshHost = "127.0.0.1"; + # The opencode UI rides the CI server's Let's Encrypt wildcard (*.ci.commoninternet.net). + cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net"; + cc-ci-orchestrator.opencodeUiAcmeHost = "ci.commoninternet.net"; - # ---- tailscale — auth key staged out of band at /etc/ts-auth-key ----------------------- - services.tailscale = { - enable = true; - authKeyFile = "/etc/ts-auth-key"; - extraUpFlags = [ "--hostname=cc-ci" ]; - }; + # ---- no tailscale on this host (operator 2026-09-07) -------------------------------------- + # Everything is reached over the public IP: ssh (keys only), the CI front doors via traefik, + # and the opencode UI on 8443 behind TLS + basic auth. fail2ban below guards the two logins. # ---- ssh ---------------------------------------------------------------------------------- services.openssh = { @@ -46,15 +46,35 @@ # The loops user can also be reached directly (same keys) — handy for rsync of its workspace. users.users.loops.openssh.authorizedKeys.keys = builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys)); + services.openssh.settings.PasswordAuthentication = false; + services.openssh.settings.KbdInteractiveAuthentication = false; # ---- firewall ------------------------------------------------------------------------------- - # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. The tailscale - # interface is trusted, which is what makes the opencode UI on 8443 tailnet-only. + # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. 8443 is the + # opencode UI (nginx, LE cert, basic auth — see orchestrator-host.nix). networking.firewall = { enable = true; - trustedInterfaces = [ "tailscale0" ]; - allowedTCPPorts = [ 22 ]; + allowedTCPPorts = [ 22 8443 ]; }; + + # ---- fail2ban: sshd (password auth is off, this stops the log noise and slow brute force) and + # the opencode UI's basic auth (nginx logs 401s to its error log; the built-in + # nginx-http-auth filter matches them). + services.fail2ban = { + enable = true; + maxretry = 5; + bantime = "1h"; + bantime-increment = { enable = true; maxtime = "48h"; factor = "4"; }; + ignoreIP = [ "127.0.0.0/8" "::1" ]; + jails.nginx-http-auth.settings = { + enabled = true; + filter = "nginx-http-auth"; + logpath = "/var/log/nginx/error.log"; + backend = "auto"; + port = "8443"; + }; + }; + networking.nameservers = [ "1.1.1.1" "8.8.8.8" ]; # ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions --------- diff --git a/nix/modules/cc-ci.nix b/nix/modules/cc-ci.nix index 8842ade..357c085 100644 --- a/nix/modules/cc-ci.nix +++ b/nix/modules/cc-ci.nix @@ -17,7 +17,7 @@ systemd.services.cc-ci-loops = { description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)"; wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover) - after = [ "network-online.target" "tailscaled.service" "claude-install.service" ]; + after = [ "network-online.target" "claude-install.service" ]; wants = [ "network-online.target" ]; serviceConfig = { # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the @@ -58,7 +58,7 @@ systemd.services.cc-ci-orchestrator = { description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator"; wantedBy = [ "multi-user.target" ]; - after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ]; + after = [ "network-online.target" "opencode-web.service" ]; wants = [ "network-online.target" ]; serviceConfig = { # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit @@ -85,7 +85,7 @@ # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule. systemd.services.cc-ci-upgrade-all = { description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)"; - after = [ "network-online.target" "tailscaled.service" "claude-install.service" ]; + after = [ "network-online.target" "claude-install.service" ]; wants = [ "network-online.target" ]; serviceConfig = { Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns @@ -129,7 +129,7 @@ # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE. systemd.services.cc-ci-upgrade-supervisor = { description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)"; - after = [ "network-online.target" "tailscaled.service" ]; + after = [ "network-online.target" ]; wants = [ "network-online.target" ]; serviceConfig = { Type = "oneshot"; # launch-supervisor.py check: gate now, spawn the agent into tmux, return diff --git a/nix/modules/orchestrator-host.nix b/nix/modules/orchestrator-host.nix index e9b7479..584dfca 100644 --- a/nix/modules/orchestrator-host.nix +++ b/nix/modules/orchestrator-host.nix @@ -34,15 +34,26 @@ in default = 8443; description = '' TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host - Traefik (docker swarm) owns 80/443. The port is not opened in the firewall, so it is - reachable only over the trusted tailscale interface. + Traefik (docker swarm) owns 80/443. The host decides whether to open it in the firewall. ''; }; opencodeUiHost = lib.mkOption { type = lib.types.str; default = "oc.commoninternet.net"; - description = "nginx server_name for the opencode web UI (self-signed, basic auth)."; + description = "nginx server_name for the opencode web UI (TLS + basic auth)."; + }; + + opencodeUiAcmeHost = lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + example = "ci.commoninternet.net"; + description = '' + Name of a `security.acme.certs` entry whose certificate covers opencodeUiHost (the + combined host has the *.ci.commoninternet.net wildcard). When set, nginx serves that + certificate; when null, the self-signed pair /etc/nginx/oc-selfsigned.{crt,key} staged + out of band is used. + ''; }; }; @@ -158,7 +169,7 @@ SSHCFG systemd.services.opencode-web = { description = "opencode web server for cc-ci agents"; wantedBy = [ "multi-user.target" ]; - after = [ "network-online.target" "tailscaled.service" "opencode-install.service" ]; + after = [ "network-online.target" "opencode-install.service" ]; wants = [ "network-online.target" ]; serviceConfig = { Type = "simple"; @@ -177,12 +188,20 @@ SSHCFG path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ]; }; - # ---- tailnet-only nginx front door for the opencode UI ------------------------------- - # Self-signed cert + basic auth, both created out of band (a store path would be world - # readable) — see README "Secrets to stage". nginx FAILS TO START if they are missing. - # /etc/nginx/oc-selfsigned.crt root:nginx 0644 + # ---- nginx front door for the opencode UI -------------------------------------------- + # TLS (LE via opencodeUiAcmeHost, else the self-signed pair below) + HTTP basic auth. The + # opencode web UI has no auth of its own and can drive agent sessions, so the htpasswd is + # mandatory. Files created out of band (a store path would be world readable) — nginx + # FAILS TO START without them, and its config check runs as the nginx user: + # /etc/nginx/oc-htpasswd root:nginx 0640 (`oc:`; plaintext kept in /secrets) + # /etc/nginx/oc-selfsigned.crt root:nginx 0644 (only when opencodeUiAcmeHost is null) # /etc/nginx/oc-selfsigned.key root:nginx 0640 - # /etc/nginx/oc-htpasswd root:nginx 0640 (`oc:`; plaintext in /secrets) + # Rotate the password with: + # printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx + # nginx must be able to read the ACME-issued key (the acme group owns it). + users.users.nginx.extraGroups = lib.mkIf (cfg.opencodeUiAcmeHost != null) + [ config.security.acme.certs.${cfg.opencodeUiAcmeHost}.group ]; + services.nginx = { enable = true; recommendedProxySettings = true; @@ -190,8 +209,9 @@ SSHCFG listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ]; # onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener. onlySSL = true; - sslCertificate = "/etc/nginx/oc-selfsigned.crt"; - sslCertificateKey = "/etc/nginx/oc-selfsigned.key"; + useACMEHost = cfg.opencodeUiAcmeHost; + sslCertificate = lib.mkIf (cfg.opencodeUiAcmeHost == null) "/etc/nginx/oc-selfsigned.crt"; + sslCertificateKey = lib.mkIf (cfg.opencodeUiAcmeHost == null) "/etc/nginx/oc-selfsigned.key"; basicAuthFile = "/etc/nginx/oc-htpasswd"; locations."/" = { proxyPass = "http://127.0.0.1:4096";