opencode UI on 443 via traefik (Host routing), nginx basic auth on the docker bridge

Operator: standard 443, routed by domain. Traefik already owns 443 with the
*.ci.commoninternet.net cert, so opencode-ui-route.service drops a dynamic
config into the traefik stack's watched file-providers volume routing
oc.ci.commoninternet.net -> http://172.18.0.1:8080, where nginx (reachable
only on docker_gwbridge) enforces the basic auth and logs 401s with the
real client IP (real_ip from X-Forwarded-For). The fail2ban nginx jail bans
in DOCKER-USER, since that traffic is forwarded, not INPUT. 8443 and the
ACME-host variant are gone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz

README.md

@@ -157,8 +157,12 @@ Everything in this section is **outside git**. Do it as root over SSH, in this o
 ### 4a. No tailscale
 
 The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
-public IP, the CI front doors are public via traefik, and the opencode UI is public on 8443
+public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
-behind TLS + basic auth, all guarded by fail2ban (`nix/hosts/cc-ci/configuration.nix`).
+443 — traefik routes `oc.ci.commoninternet.net` (a dynamic-config file that
+`opencode-ui-route.service` drops into the traefik stack's file-providers volume) to an nginx
+vhost on the docker bridge that enforces basic auth. fail2ban guards sshd and that login
+(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
+traffic is docker-forwarded, not host INPUT).
 
 ### 4b. The CI server's checkout and its one out-of-band secret
 
@@ -210,7 +214,7 @@ Then the files below (`install -m600 -o loops -g users`):
 | `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
 | `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
 | `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
-| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net:8443`, TLS from the CI wildcard cert); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
+| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
 
 `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
 `127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
@@ -259,8 +263,9 @@ curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{h
 dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
 ```
 
-The opencode UI: `curl -sk --resolve oc.ci.commoninternet.net:8443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net:8443/`
+The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
-→ 401 without credentials, 200 with `-u oc:<password>`.
+→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
+counts the failures.
 
 When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
 default). **If you are migrating from another host, do §6 before letting it serve anything**: right

nix/hosts/cc-ci/configuration.nix

@@ -27,13 +27,14 @@
   # ---- orchestrator identity --------------------------------------------------------------
   # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
   cc-ci-orchestrator.ciSshHost = "127.0.0.1";
-  # The opencode UI rides the CI server's Let's Encrypt wildcard (*.ci.commoninternet.net).
+  # The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth.
   cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
-  cc-ci-orchestrator.opencodeUiAcmeHost = "ci.commoninternet.net";
+  cc-ci-orchestrator.opencodeUiTraefikProvidersDir =
+    "/var/lib/docker/volumes/traefik_ci_commoninternet_net_file-providers/_data";
 
   # ---- no tailscale on this host (operator 2026-09-07) --------------------------------------
   # Everything is reached over the public IP: ssh (keys only), the CI front doors via traefik,
-  # and the opencode UI on 8443 behind TLS + basic auth. fail2ban below guards the two logins.
+  # and the opencode UI on 443 (traefik → nginx basic auth). fail2ban below guards the two logins.
 
   # ---- ssh ----------------------------------------------------------------------------------
   services.openssh = {
@@ -50,16 +51,18 @@
   services.openssh.settings.KbdInteractiveAuthentication = false;
 
   # ---- firewall -------------------------------------------------------------------------------
-  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. 8443 is the
+  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module; the opencode UI
-  # opencode UI (nginx, LE cert, basic auth — see orchestrator-host.nix).
+  # rides 443 through traefik (orchestrator-host.nix), so only ssh is opened here.
   networking.firewall = {
     enable = true;
-    allowedTCPPorts = [ 22 8443 ];
+    allowedTCPPorts = [ 22 ];
   };
 
   # ---- fail2ban: sshd (password auth is off, this stops the log noise and slow brute force) and
-  # the opencode UI's basic auth (nginx logs 401s to its error log; the built-in
+  # the opencode UI's basic auth (nginx logs 401s with the real client IP to its error log; the
-  # nginx-http-auth filter matches them).
+  # built-in nginx-http-auth filter matches them). Those clients arrive through traefik's
+  # docker-published 443, which iptables FORWARDs rather than INPUTs, so the ban for that jail
+  # goes into the DOCKER-USER chain — an INPUT rule would never see the traffic.
   services.fail2ban = {
     enable = true;
     maxretry = 5;
@@ -71,7 +74,8 @@
       filter = "nginx-http-auth";
       logpath = "/var/log/nginx/error.log";
       backend = "auto";
-      port = "8443";
+      banaction = "iptables-allports";
+      chain = "DOCKER-USER";
     };
   };
 

nix/modules/orchestrator-host.nix

@@ -29,12 +29,25 @@ in
       '';
     };
 
-    opencodeUiPort = lib.mkOption {
+    opencodeUiBackendPort = lib.mkOption {
       type = lib.types.port;
-      default = 8443;
+      default = 8080;
       description = ''
-        TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host
+        Plain-HTTP port nginx listens on for the opencode UI, reachable ONLY from the docker
-        Traefik (docker swarm) owns 80/443. The host decides whether to open it in the firewall.
+        bridge (firewall rule on docker_gwbridge). Traefik — which owns the public 443 on the
+        combined host — terminates TLS for opencodeUiHost and forwards here; nginx adds the basic
+        auth and logs failures for fail2ban with the real client IP.
+      '';
+    };
+
+    opencodeUiTraefikProvidersDir = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "/var/lib/docker/volumes/traefik_ci_commoninternet_net_file-providers/_data";
+      description = ''
+        Host path of the traefik stack's file-providers directory (watched by traefik). When set,
+        a dynamic-config file routing opencodeUiHost on the `web-secure` entrypoint to nginx is
+        written there after deploy-proxy. null = no traefik route (serve some other way).
       '';
     };
 
@@ -44,17 +57,6 @@ in
       description = "nginx server_name for the opencode web UI (TLS + basic auth).";
     };
 
-    opencodeUiAcmeHost = lib.mkOption {
-      type = lib.types.nullOr lib.types.str;
-      default = null;
-      example = "ci.commoninternet.net";
-      description = ''
-        Name of a `security.acme.certs` entry whose certificate covers opencodeUiHost (the
-        combined host has the *.ci.commoninternet.net wildcard). When set, nginx serves that
-        certificate; when null, the self-signed pair /etc/nginx/oc-selfsigned.{crt,key} staged
-        out of band is used.
-      '';
-    };
   };
 
   config = {
@@ -188,36 +190,61 @@ SSHCFG
       path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
     };
 
-    # ---- nginx front door for the opencode UI --------------------------------------------
+    # ---- nginx: basic auth for the opencode UI, behind traefik -----------------------------
-    # TLS (LE via opencodeUiAcmeHost, else the self-signed pair below) + HTTP basic auth. The
+    # Traefik (public 443, the CI wildcard cert) routes opencodeUiHost to this plain-HTTP vhost
-    # opencode web UI has no auth of its own and can drive agent sessions, so the htpasswd is
+    # on the docker bridge address. nginx enforces HTTP basic auth (the opencode web UI has no
-    # mandatory. Files created out of band (a store path would be world readable) — nginx
+    # auth of its own and can drive agent sessions), and — via real_ip from traefik's
-    # FAILS TO START without them, and its config check runs as the nginx user:
+    # X-Forwarded-For — logs the CLIENT address on a 401, which is what the fail2ban jail bans.
-    #   /etc/nginx/oc-htpasswd        root:nginx 0640   (`oc:<bcrypt>`; plaintext kept in /secrets)
+    # The htpasswd is created out of band (a store path would be world readable); nginx FAILS TO
-    #   /etc/nginx/oc-selfsigned.crt  root:nginx 0644   (only when opencodeUiAcmeHost is null)
+    # START without it, and its config check runs as the nginx user:
-    #   /etc/nginx/oc-selfsigned.key  root:nginx 0640
+    #   /etc/nginx/oc-htpasswd   root:nginx 0640   (`oc:<bcrypt>`; plaintext kept in /secrets)
-    # Rotate the password with:
+    # Rotate with: printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx
-    #   printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx
-    # nginx must be able to read the ACME-issued key (the acme group owns it).
-    users.users.nginx.extraGroups = lib.mkIf (cfg.opencodeUiAcmeHost != null)
-      [ config.security.acme.certs.${cfg.opencodeUiAcmeHost}.group ];
-
     services.nginx = {
       enable = true;
       recommendedProxySettings = true;
       virtualHosts.${cfg.opencodeUiHost} = {
-        listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ];
+        listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiBackendPort; } ];
-        # onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener.
-        onlySSL = true;
-        useACMEHost = cfg.opencodeUiAcmeHost;
-        sslCertificate = lib.mkIf (cfg.opencodeUiAcmeHost == null) "/etc/nginx/oc-selfsigned.crt";
-        sslCertificateKey = lib.mkIf (cfg.opencodeUiAcmeHost == null) "/etc/nginx/oc-selfsigned.key";
         basicAuthFile = "/etc/nginx/oc-htpasswd";
+        extraConfig = ''
+          # traefik sits on the docker networks (ingress 10.0.0.0/24, gwbridge 172.18.0.0/16)
+          set_real_ip_from 172.16.0.0/12;
+          set_real_ip_from 10.0.0.0/8;
+          real_ip_header X-Forwarded-For;
+        '';
         locations."/" = {
           proxyPass = "http://127.0.0.1:4096";
           proxyWebsockets = true;
         };
       };
     };
+    # Only docker's bridge may reach the plain-HTTP backend; the public interface stays closed.
+    networking.firewall.interfaces.docker_gwbridge.allowedTCPPorts = [ cfg.opencodeUiBackendPort ];
+
+    # The traefik side of the route: a dynamic-config file in the stack's watched providers dir.
+    systemd.services.opencode-ui-route = lib.mkIf (cfg.opencodeUiTraefikProvidersDir != null) {
+      description = "traefik route: ${cfg.opencodeUiHost} -> nginx (opencode UI basic auth)";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "deploy-proxy.service" "docker.service" ];
+      wants = [ "deploy-proxy.service" ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
+      script = ''
+        install -d -m 0755 "${cfg.opencodeUiTraefikProvidersDir}"
+        install -m 0644 ${pkgs.writeText "opencode-ui.yml" ''
+          # Written by opencode-ui-route.service (nix/modules/orchestrator-host.nix). Do not edit.
+          http:
+            routers:
+              opencode-ui:
+                rule: Host(`${cfg.opencodeUiHost}`)
+                entryPoints: [web-secure]
+                service: opencode-ui
+                tls: {}
+            services:
+              opencode-ui:
+                loadBalancer:
+                  servers:
+                    - url: http://172.18.0.1:${toString cfg.opencodeUiBackendPort}
+        ''} "${cfg.opencodeUiTraefikProvidersDir}/opencode-ui.yml"
+      '';
+    };
   };
 }