cc-ci host: no tailscale; fail2ban; opencode UI public on 8443 with the LE wildcard

Operator 2026-09-07: the combined host stays off the tailnet. ssh is
key-only (password + keyboard-interactive auth off), the opencode UI is
served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's
*.ci.commoninternet.net ACME cert (new orchestrator-host option
opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic
auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail).
tailscaled ordering dropped from the orchestrator units. README updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz

README.md

@@ -11,7 +11,7 @@ module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) rep
 | Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
 | The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
 | The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
-| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
+| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
 | Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
 | Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
 | How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
@@ -31,7 +31,8 @@ can repeat it. Read it once before starting; the order matters.
 
 - A **Hetzner Cloud** project you can create servers in (console login or an API token).
 - **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
-  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
+  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
+  login is disabled and fail2ban watches sshd, so a key is the only way in.
 - Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
   read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
   `autonomic-bot-gitea-ed25519`, has it).
@@ -153,12 +154,11 @@ Commit on a branch; the rebuild in §5 can use the local checkout before the PR
 
 Everything in this section is **outside git**. Do it as root over SSH, in this order.
 
-### 4a. Tailscale
+### 4a. No tailscale
 
-```bash
+The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
-# a reusable (or fresh) tailnet auth key from the tailscale admin console
+public IP, the CI front doors are public via traefik, and the opencode UI is public on 8443
-install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
+behind TLS + basic auth, all guarded by fail2ban (`nix/hosts/cc-ci/configuration.nix`).
-```
 
 ### 4b. The CI server's checkout and its one out-of-band secret
 
@@ -210,7 +210,7 @@ Then the files below (`install -m600 -o loops -g users`):
 | `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
 | `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
 | `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
-| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
+| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net:8443`, TLS from the CI wildcard cert); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
 
 `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
 `127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
@@ -220,7 +220,7 @@ Then the files below (`install -m600 -o loops -g users`):
 From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
 
 ```bash
-# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
+# as root, detached (the activation restarts sshd; a dropped session must not kill it).
 # Three things the FIRST rebuild on a bare infect system needs, none of which the converged
 # host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
 # system has none — hence nix-shell), HOME=/root (so root's `git config --global
@@ -234,6 +234,9 @@ journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two
 
 `test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
 networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
+(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
+restarted tailscale under the session; this one is plain public ssh, but detached is still the
+safe habit.)
 Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
 
 The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
@@ -243,9 +246,9 @@ deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
 
 ```bash
 systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
-tailscale status | head -3
 docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
-systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
+systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
+fail2ban-client status sshd; fail2ban-client status nginx-http-auth
 systemctl list-timers 'cc-ci-*' nightly-sweep
 sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
 # the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
@@ -256,10 +259,8 @@ curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{h
 dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
 ```
 
-Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
+The opencode UI: `curl -sk --resolve oc.ci.commoninternet.net:8443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net:8443/`
-the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
+→ 401 without credentials, 200 with `-u oc:<password>`.
-it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
-during the install; the box is reachable on its public IP throughout.
 
 When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
 default). **If you are migrating from another host, do §6 before letting it serve anything**: right
@@ -273,7 +274,7 @@ systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
 
 ## 6. Migrating: restore state from the previous host
 
-Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
+Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
 stopped on the new host while its directory is copied:
 
 | path | holds | notes |
@@ -305,7 +306,8 @@ stopped on the new host while its directory is copied:
 5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
    (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
    claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
-6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
+6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
+   old servers were on the tailnet; the new one is not).
 
 ## 8. Day 2
 

nix/hosts/cc-ci/configuration.nix

@@ -1,7 +1,7 @@
 # cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
 #
-# This file is only what is physical or identity about the machine: hardware, networking, the
+# This file is only what is physical or identity about the machine: hardware, networking,
-# tailscale node, root SSH keys, swap, stateVersion. Everything functional comes from modules:
+# root SSH keys, firewall + fail2ban, swap, stateVersion. Everything functional comes from modules:
 #   cc-ci.nixosModules.cc-ci-server        recipe-maintainers/cc-ci — swarm, traefik, drone,
 #                                          runner, bridge, dashboard, reports, acme-dns, harness
 #   self.nixosModules.cc-ci-orchestrator   nix/modules/cc-ci.nix — loops, orchestrator, timers
@@ -27,13 +27,13 @@
   # ---- orchestrator identity --------------------------------------------------------------
   # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
   cc-ci-orchestrator.ciSshHost = "127.0.0.1";
+  # The opencode UI rides the CI server's Let's Encrypt wildcard (*.ci.commoninternet.net).
+  cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
+  cc-ci-orchestrator.opencodeUiAcmeHost = "ci.commoninternet.net";
 
-  # ---- tailscale — auth key staged out of band at /etc/ts-auth-key -----------------------
+  # ---- no tailscale on this host (operator 2026-09-07) --------------------------------------
-  services.tailscale = {
+  # Everything is reached over the public IP: ssh (keys only), the CI front doors via traefik,
-    enable = true;
+  # and the opencode UI on 8443 behind TLS + basic auth. fail2ban below guards the two logins.
-    authKeyFile = "/etc/ts-auth-key";
-    extraUpFlags = [ "--hostname=cc-ci" ];
-  };
 
   # ---- ssh ----------------------------------------------------------------------------------
   services.openssh = {
@@ -46,15 +46,35 @@
   # The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
   users.users.loops.openssh.authorizedKeys.keys =
     builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+  services.openssh.settings.PasswordAuthentication = false;
+  services.openssh.settings.KbdInteractiveAuthentication = false;
 
   # ---- firewall -------------------------------------------------------------------------------
-  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. The tailscale
+  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. 8443 is the
-  # interface is trusted, which is what makes the opencode UI on 8443 tailnet-only.
+  # opencode UI (nginx, LE cert, basic auth — see orchestrator-host.nix).
   networking.firewall = {
     enable = true;
-    trustedInterfaces = [ "tailscale0" ];
+    allowedTCPPorts = [ 22 8443 ];
-    allowedTCPPorts = [ 22 ];
   };
+
+  # ---- fail2ban: sshd (password auth is off, this stops the log noise and slow brute force) and
+  # the opencode UI's basic auth (nginx logs 401s to its error log; the built-in
+  # nginx-http-auth filter matches them).
+  services.fail2ban = {
+    enable = true;
+    maxretry = 5;
+    bantime = "1h";
+    bantime-increment = { enable = true; maxtime = "48h"; factor = "4"; };
+    ignoreIP = [ "127.0.0.0/8" "::1" ];
+    jails.nginx-http-auth.settings = {
+      enabled = true;
+      filter = "nginx-http-auth";
+      logpath = "/var/log/nginx/error.log";
+      backend = "auto";
+      port = "8443";
+    };
+  };
+
   networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
 
   # ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------

nix/modules/cc-ci.nix

@@ -17,7 +17,7 @@
   systemd.services.cc-ci-loops = {
     description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
     wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    after = [ "network-online.target" "claude-install.service" ];
     wants = [ "network-online.target" ];
     serviceConfig = {
       # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
@@ -58,7 +58,7 @@
   systemd.services.cc-ci-orchestrator = {
     description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
     wantedBy = [ "multi-user.target" ];
-    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
+    after = [ "network-online.target" "opencode-web.service" ];
     wants = [ "network-online.target" ];
     serviceConfig = {
       # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
@@ -85,7 +85,7 @@
   # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
   systemd.services.cc-ci-upgrade-all = {
     description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    after = [ "network-online.target" "claude-install.service" ];
     wants = [ "network-online.target" ];
     serviceConfig = {
       Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
@@ -129,7 +129,7 @@
   # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
   systemd.services.cc-ci-upgrade-supervisor = {
     description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
-    after = [ "network-online.target" "tailscaled.service" ];
+    after = [ "network-online.target" ];
     wants = [ "network-online.target" ];
     serviceConfig = {
       Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return

nix/modules/orchestrator-host.nix

@@ -34,15 +34,26 @@ in
       default = 8443;
       description = ''
         TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host
-        Traefik (docker swarm) owns 80/443. The port is not opened in the firewall, so it is
+        Traefik (docker swarm) owns 80/443. The host decides whether to open it in the firewall.
-        reachable only over the trusted tailscale interface.
       '';
     };
 
     opencodeUiHost = lib.mkOption {
       type = lib.types.str;
       default = "oc.commoninternet.net";
-      description = "nginx server_name for the opencode web UI (self-signed, basic auth).";
+      description = "nginx server_name for the opencode web UI (TLS + basic auth).";
+    };
+
+    opencodeUiAcmeHost = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "ci.commoninternet.net";
+      description = ''
+        Name of a `security.acme.certs` entry whose certificate covers opencodeUiHost (the
+        combined host has the *.ci.commoninternet.net wildcard). When set, nginx serves that
+        certificate; when null, the self-signed pair /etc/nginx/oc-selfsigned.{crt,key} staged
+        out of band is used.
+      '';
     };
   };
 
@@ -158,7 +169,7 @@ SSHCFG
     systemd.services.opencode-web = {
       description = "opencode web server for cc-ci agents";
       wantedBy = [ "multi-user.target" ];
-      after = [ "network-online.target" "tailscaled.service" "opencode-install.service" ];
+      after = [ "network-online.target" "opencode-install.service" ];
       wants = [ "network-online.target" ];
       serviceConfig = {
         Type = "simple";
@@ -177,12 +188,20 @@ SSHCFG
       path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
     };
 
-    # ---- tailnet-only nginx front door for the opencode UI -------------------------------
+    # ---- nginx front door for the opencode UI --------------------------------------------
-    # Self-signed cert + basic auth, both created out of band (a store path would be world
+    # TLS (LE via opencodeUiAcmeHost, else the self-signed pair below) + HTTP basic auth. The
-    # readable) — see README "Secrets to stage". nginx FAILS TO START if they are missing.
+    # opencode web UI has no auth of its own and can drive agent sessions, so the htpasswd is
-    #   /etc/nginx/oc-selfsigned.crt  root:nginx 0644
+    # mandatory. Files created out of band (a store path would be world readable) — nginx
+    # FAILS TO START without them, and its config check runs as the nginx user:
+    #   /etc/nginx/oc-htpasswd        root:nginx 0640   (`oc:<bcrypt>`; plaintext kept in /secrets)
+    #   /etc/nginx/oc-selfsigned.crt  root:nginx 0644   (only when opencodeUiAcmeHost is null)
     #   /etc/nginx/oc-selfsigned.key  root:nginx 0640
-    #   /etc/nginx/oc-htpasswd        root:nginx 0640   (`oc:<bcrypt>`; plaintext in /secrets)
+    # Rotate the password with:
+    #   printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx
+    # nginx must be able to read the ACME-issued key (the acme group owns it).
+    users.users.nginx.extraGroups = lib.mkIf (cfg.opencodeUiAcmeHost != null)
+      [ config.security.acme.certs.${cfg.opencodeUiAcmeHost}.group ];
+
     services.nginx = {
       enable = true;
       recommendedProxySettings = true;
@@ -190,8 +209,9 @@ SSHCFG
         listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ];
         # onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener.
         onlySSL = true;
-        sslCertificate = "/etc/nginx/oc-selfsigned.crt";
+        useACMEHost = cfg.opencodeUiAcmeHost;
-        sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
+        sslCertificate = lib.mkIf (cfg.opencodeUiAcmeHost == null) "/etc/nginx/oc-selfsigned.crt";
+        sslCertificateKey = lib.mkIf (cfg.opencodeUiAcmeHost == null) "/etc/nginx/oc-selfsigned.key";
         basicAuthFile = "/etc/nginx/oc-htpasswd";
         locations."/" = {
           proxyPass = "http://127.0.0.1:4096";