diff --git a/README.md b/README.md index 23d5fe1..b95644b 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ Then the files below (`install -m600 -o loops -g users`): | `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host | | `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) | | `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` | -| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them** | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) | +| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) | `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` → `127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present. @@ -220,15 +220,21 @@ Then the files below (`install -m600 -o loops -g users`): From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo): ```bash -sudo -iu loops -cd /srv/cc-ci-orch && git checkout -sudo nixos-rebuild test --flake .#cc-ci # activates WITHOUT touching the bootloader +# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it). +# Three things the FIRST rebuild on a bare infect system needs, none of which the converged +# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect +# system has none — hence nix-shell), HOME=/root (so root's `git config --global +# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for +# NIX_SSL_CERT_FILE and friends from /etc/set-environment). +git config --global --add safe.directory '*' +systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \ + bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"' +journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two OCI image builds) ``` -`test` first, always: if the activation breaks networking or sshd, a reboot from the Hetzner -console lands on the last known-good generation. Over an SSH session that itself rides tailscale, -run it detached (`systemd-run --unit=ccci-sw --collect nixos-rebuild test --flake .#cc-ci`) so the -tailscale restart during activation does not abort it. +`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks +networking or sshd a reboot from the Hetzner console lands on the last known-good generation. +Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout. The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile @@ -242,8 +248,19 @@ docker service ls # traefik app+socket-proxy, drone, bridge, da systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns systemctl list-timers 'cc-ci-*' nightly-sweep sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active) +# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once +# /var/lib/acme is restored or a cert has been issued): +curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/ +curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/ +curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/ +dig +short @ ns-acme.commoninternet.net # acme-dns answering on the public 53 ``` +Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` — +the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put +it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it +during the install; the box is reachable on its public IP throughout. + When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot default). **If you are migrating from another host, do §6 before letting it serve anything**: right after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so diff --git a/nix/modules/orchestrator-host.nix b/nix/modules/orchestrator-host.nix index ef7d420..e9b7479 100644 --- a/nix/modules/orchestrator-host.nix +++ b/nix/modules/orchestrator-host.nix @@ -145,6 +145,10 @@ SSHCFG if [ ! -x "$HOME/.local/bin/opencode" ]; then echo "installing opencode CLI for loops user..." curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation" + # The installer puts the binary in ~/.opencode/bin; every unit here expects ~/.local/bin. + if [ -x "$HOME/.opencode/bin/opencode" ]; then + mkdir -p "$HOME/.local/bin" && ln -sfn "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode" + fi fi ''; };