diff --git a/cc-ci-plan/JOURNAL.md b/cc-ci-plan/JOURNAL.md index 10da278..13479b2 100644 --- a/cc-ci-plan/JOURNAL.md +++ b/cc-ci-plan/JOURNAL.md @@ -1061,3 +1061,13 @@ Plan + live log: `cc-ci-plan/plan-cc-ci-combined-host.md` (on the branch; copy h .sops/master-age.txt, .cc-ci-logs; nginx oc-* files (root:nginx 0640). - Open: tailscale auth key revoked (`invalid key: API key does not exist`) → operator issues a new one. DNS cutover at Gandi (ci, *.ci, ns-acme → 195.201.88.249) → operator. + +## Session 2026-09-07 21:10 UTC — CUTOVER DONE; orchestrator now on the new host + +CI is served from 195.201.88.249 (DNS repointed by the operator, no tailscale on the new box, +opencode UI on 443 via traefik + basic auth, fail2ban on). `!testme` → Drone build 1326 on the +new box. This session (on the old shared host) is the last one here: cc-ci units runtime-masked +on the old host; a fresh orchestrator session is up on the new box (`ssh root@195.201.88.249`, +`sudo -iu loops tmux attach -t cc-ci-orchestrator`). Details + the two incidents: +`cc-ci-plan/plan-cc-ci-combined-host.md` (log) and README §2a/§7. Operator to merge +notplants-nix `chore/drop-cc-ci`, delete the old CI server in a week, revoke the Hetzner token. diff --git a/cc-ci-plan/plan-cc-ci-combined-host.md b/cc-ci-plan/plan-cc-ci-combined-host.md index ee35ee2..fbb5051 100644 --- a/cc-ci-plan/plan-cc-ci-combined-host.md +++ b/cc-ci-plan/plan-cc-ci-combined-host.md @@ -78,11 +78,11 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`. new host does not double-process `!testme` or run a second weekly upgrade before cutover. 5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs, /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy. -6. [~] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports, +6. [x] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports, drone, one direct `cc-ci-run` on custom-html-tiny). -7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge + +7. [x] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge + drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page. -8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent +8. [~] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci. 9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node. 10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven. @@ -104,3 +104,20 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`. reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator #19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped). Waiting on operator: new tailscale auth key; Gandi DNS cutover. +- 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic + auth on 443 (traefik labels on the `opencode-ui` socat relay → nginx on docker_gwbridge:8090); + fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme) + → 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge + + drone scaled to 0, old-host timers masked, new bridge up; `!testme` on custom-html #7 = Drone + build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik + FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in + networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped. + `nixos-rebuild switch` = generation 4. The `switch` re-enabled cc-ci-orchestrator on the new + box: a fresh Claude orchestrator session runs there (tmux `cc-ci-orchestrator`, remote-control + label `cc-ci-orchestrator`) since 20:46 UTC; the old host's cc-ci units are runtime-masked and + the notplants-nix branch `chore/drop-cc-ci` (pushed, evaluates) removes them for good — + operator merges (that repo auto-deploys nightly). +- Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the + Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host + key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during + the resolver mess).