plan + journal: cutover done 2026-09-07, orchestrator moved, remaining items

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz

cc-ci-plan/JOURNAL.md

@@ -1061,3 +1061,13 @@ Plan + live log: `cc-ci-plan/plan-cc-ci-combined-host.md` (on the branch; copy h
   .sops/master-age.txt, .cc-ci-logs; nginx oc-* files (root:nginx 0640).
 - Open: tailscale auth key revoked (`invalid key: API key does not exist`) → operator issues a
   new one. DNS cutover at Gandi (ci, *.ci, ns-acme → 195.201.88.249) → operator.
+
+## Session 2026-09-07 21:10 UTC — CUTOVER DONE; orchestrator now on the new host
+
+CI is served from 195.201.88.249 (DNS repointed by the operator, no tailscale on the new box,
+opencode UI on 443 via traefik + basic auth, fail2ban on). `!testme` → Drone build 1326 on the
+new box. This session (on the old shared host) is the last one here: cc-ci units runtime-masked
+on the old host; a fresh orchestrator session is up on the new box (`ssh root@195.201.88.249`,
+`sudo -iu loops tmux attach -t cc-ci-orchestrator`). Details + the two incidents:
+`cc-ci-plan/plan-cc-ci-combined-host.md` (log) and README §2a/§7. Operator to merge
+notplants-nix `chore/drop-cc-ci`, delete the old CI server in a week, revoke the Hetzner token.

cc-ci-plan/plan-cc-ci-combined-host.md

@@ -78,11 +78,11 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
    new host does not double-process `!testme` or run a second weekly upgrade before cutover.
 5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
    /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
-6. [~] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
+6. [x] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
    drone, one direct `cc-ci-run` on custom-html-tiny).
-7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
+7. [x] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
    drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
-8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
+8. [~] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
    state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
 9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
 10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
@@ -104,3 +104,20 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
   reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator
   #19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped).
   Waiting on operator: new tailscale auth key; Gandi DNS cutover.
+- 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic
+  auth on 443 (traefik labels on the `opencode-ui` socat relay → nginx on docker_gwbridge:8090);
+  fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme)
+  → 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge +
+  drone scaled to 0, old-host timers masked, new bridge up; `!testme` on custom-html #7 = Drone
+  build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik
+  FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in
+  networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped.
+  `nixos-rebuild switch` = generation 4. The `switch` re-enabled cc-ci-orchestrator on the new
+  box: a fresh Claude orchestrator session runs there (tmux `cc-ci-orchestrator`, remote-control
+  label `cc-ci-orchestrator`) since 20:46 UTC; the old host's cc-ci units are runtime-masked and
+  the notplants-nix branch `chore/drop-cc-ci` (pushed, evaluates) removes them for good —
+  operator merges (that repo auto-deploys nightly).
+- Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the
+  Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host
+  key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during
+  the resolver mess).