orchestrator-host: symlink the opencode installer's binary into ~/.local/bin; README: nginx file perms, first-rebuild wrapper, pre-cutover checks

Found on the first activation of #cc-ci on 195.201.88.249: the opencode
installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and
nginx's pre-start config check runs as the nginx user, so the staged
cert/htpasswd must be root:nginx 0640, not root:root 0600.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz

README.md

@@ -210,7 +210,7 @@ Then the files below (`install -m600 -o loops -g users`):
 | `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
 | `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
 | `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
-| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them** | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
+| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
 
 `~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
 `127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
@@ -220,15 +220,21 @@ Then the files below (`install -m600 -o loops -g users`):
 From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
 
 ```bash
-sudo -iu loops
+# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
-cd /srv/cc-ci-orch && git checkout <branch>
+# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
-sudo nixos-rebuild test --flake .#cc-ci     # activates WITHOUT touching the bootloader
+# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
+# system has none — hence nix-shell), HOME=/root (so root's `git config --global
+# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
+# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
+git config --global --add safe.directory '*'
+systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
+  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
+journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)
 ```
 
-`test` first, always: if the activation breaks networking or sshd, a reboot from the Hetzner
+`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
-console lands on the last known-good generation. Over an SSH session that itself rides tailscale,
+networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
-run it detached (`systemd-run --unit=ccci-sw --collect nixos-rebuild test --flake .#cc-ci`) so the
+Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
-tailscale restart during activation does not abort it.
 
 The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
 and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
@@ -242,8 +248,19 @@ docker service ls                  # traefik app+socket-proxy, drone, bridge, da
 systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
 systemctl list-timers 'cc-ci-*' nightly-sweep
 sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
+# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
+# /var/lib/acme is restored or a cert has been issued):
+curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
+curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
+curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
+dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
 ```
 
+Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
+the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
+it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
+during the install; the box is reachable on its public IP throughout.
+
 When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
 default). **If you are migrating from another host, do §6 before letting it serve anything**: right
 after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so

nix/modules/orchestrator-host.nix

@@ -145,6 +145,10 @@ SSHCFG
         if [ ! -x "$HOME/.local/bin/opencode" ]; then
           echo "installing opencode CLI for loops user..."
           curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation"
+          # The installer puts the binary in ~/.opencode/bin; every unit here expects ~/.local/bin.
+          if [ -x "$HOME/.opencode/bin/opencode" ]; then
+            mkdir -p "$HOME/.local/bin" && ln -sfn "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode"
+          fi
         fi
       '';
     };