cc-ci host: no tailscale; fail2ban; opencode UI public on 8443 with the LE wildcard
Operator 2026-09-07: the combined host stays off the tailnet. ssh is key-only (password + keyboard-interactive auth off), the opencode UI is served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's *.ci.commoninternet.net ACME cert (new orchestrator-host option opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail). tailscaled ordering dropped from the orchestrator units. README updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
Claude Fable 5.1
@@ -11,7 +11,7 @@ module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) rep
|
||||
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
|
||||
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
|
||||
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
|
||||
| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
||||
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
||||
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
|
||||
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
|
||||
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
|
||||
@@ -31,7 +31,8 @@ can repeat it. Read it once before starting; the order matters.
|
||||
|
||||
- A **Hetzner Cloud** project you can create servers in (console login or an API token).
|
||||
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
|
||||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
|
||||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
|
||||
login is disabled and fail2ban watches sshd, so a key is the only way in.
|
||||
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
|
||||
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
|
||||
`autonomic-bot-gitea-ed25519`, has it).
|
||||
@@ -153,12 +154,11 @@ Commit on a branch; the rebuild in §5 can use the local checkout before the PR
|
||||
|
||||
Everything in this section is **outside git**. Do it as root over SSH, in this order.
|
||||
|
||||
### 4a. Tailscale
|
||||
### 4a. No tailscale
|
||||
|
||||
```bash
|
||||
# a reusable (or fresh) tailnet auth key from the tailscale admin console
|
||||
install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
|
||||
```
|
||||
The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
|
||||
public IP, the CI front doors are public via traefik, and the opencode UI is public on 8443
|
||||
behind TLS + basic auth, all guarded by fail2ban (`nix/hosts/cc-ci/configuration.nix`).
|
||||
|
||||
### 4b. The CI server's checkout and its one out-of-band secret
|
||||
|
||||
@@ -210,7 +210,7 @@ Then the files below (`install -m600 -o loops -g users`):
|
||||
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
|
||||
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
|
||||
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
|
||||
| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
|
||||
| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net:8443`, TLS from the CI wildcard cert); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
|
||||
|
||||
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
|
||||
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
|
||||
@@ -220,7 +220,7 @@ Then the files below (`install -m600 -o loops -g users`):
|
||||
From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
|
||||
|
||||
```bash
|
||||
# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
|
||||
# as root, detached (the activation restarts sshd; a dropped session must not kill it).
|
||||
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
|
||||
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
|
||||
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
|
||||
@@ -234,6 +234,9 @@ journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two
|
||||
|
||||
`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
|
||||
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
|
||||
(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
|
||||
restarted tailscale under the session; this one is plain public ssh, but detached is still the
|
||||
safe habit.)
|
||||
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
|
||||
|
||||
The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
|
||||
@@ -243,9 +246,9 @@ deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
|
||||
|
||||
```bash
|
||||
systemctl is-system-running # running — or list-units --failed and read journalctl -u <unit>
|
||||
tailscale status | head -3
|
||||
docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
|
||||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
|
||||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
|
||||
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
|
||||
systemctl list-timers 'cc-ci-*' nightly-sweep
|
||||
sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active)
|
||||
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
|
||||
@@ -256,10 +259,8 @@ curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{h
|
||||
dig +short @<ip> ns-acme.commoninternet.net # acme-dns answering on the public 53
|
||||
```
|
||||
|
||||
Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
|
||||
the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
|
||||
it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
|
||||
during the install; the box is reachable on its public IP throughout.
|
||||
The opencode UI: `curl -sk --resolve oc.ci.commoninternet.net:8443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net:8443/`
|
||||
→ 401 without credentials, 200 with `-u oc:<password>`.
|
||||
|
||||
When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
|
||||
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
|
||||
@@ -273,7 +274,7 @@ systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
|
||||
|
||||
## 6. Migrating: restore state from the previous host
|
||||
|
||||
Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
|
||||
Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
|
||||
stopped on the new host while its directory is copied:
|
||||
|
||||
| path | holds | notes |
|
||||
@@ -305,7 +306,8 @@ stopped on the new host while its directory is copied:
|
||||
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
|
||||
(or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
|
||||
claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
|
||||
6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
|
||||
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
|
||||
old servers were on the tailnet; the new one is not).
|
||||
|
||||
## 8. Day 2
|
||||
|
||||
|
||||
Reference in New Issue
Block a user