recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

plan + journal: cutover done 2026-09-07, orchestrator moved, remaining items

Browse Source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 21:06:09 +00:00
co-authored by Claude Fable 5.1
parent bf84245d4b
commit 666509d53d
2 changed files with 30 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cc-ci-plan/JOURNAL.md
+10
View File
@@ -1061,3 +1061,13 @@ Plan + live log: `cc-ci-plan/plan-cc-ci-combined-host.md` (on the branch; copy h
.sops/master-age.txt, .cc-ci-logs; nginx oc-* files (root:nginx 0640).
- Open: tailscale auth key revoked (`invalid key: API key does not exist`) → operator issues a
new one. DNS cutover at Gandi (ci, *.ci, ns-acme → 195.201.88.249) → operator.
## Session 2026-09-07 21:10 UTC — CUTOVER DONE; orchestrator now on the new host
CI is served from 195.201.88.249 (DNS repointed by the operator, no tailscale on the new box,
opencode UI on 443 via traefik + basic auth, fail2ban on). `!testme` → Drone build 1326 on the
new box. This session (on the old shared host) is the last one here: cc-ci units runtime-masked
on the old host; a fresh orchestrator session is up on the new box (`ssh root@195.201.88.249`,
`sudo -iu loops tmux attach -t cc-ci-orchestrator`). Details + the two incidents:
`cc-ci-plan/plan-cc-ci-combined-host.md` (log) and README §2a/§7. Operator to merge
notplants-nix `chore/drop-cc-ci`, delete the old CI server in a week, revoke the Hetzner token.
cc-ci-plan/plan-cc-ci-combined-host.md
+20 -3
View File
@@ -78,11 +78,11 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
new host does not double-process `!testme` or run a second weekly upgrade before cutover.
5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
/root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
6. [~] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
6. [x] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
drone, one direct `cc-ci-run` on custom-html-tiny).
7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
7. [x] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
8. [~] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
@@ -104,3 +104,20 @@ notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator
#19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped).
Waiting on operator: new tailscale auth key; Gandi DNS cutover.
- 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic
auth on 443 (traefik labels on the `opencode-ui` socat relay → nginx on docker_gwbridge:8090);
fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme)
→ 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge +
drone scaled to 0, old-host timers masked, new bridge up; `!testme` on custom-html #7 = Drone
build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik
FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in
networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped.
`nixos-rebuild switch` = generation 4. The `switch` re-enabled cc-ci-orchestrator on the new
box: a fresh Claude orchestrator session runs there (tmux `cc-ci-orchestrator`, remote-control
label `cc-ci-orchestrator`) since 20:46 UTC; the old host's cc-ci units are runtime-masked and
the notplants-nix branch `chore/drop-cc-ci` (pushed, evaluates) removes them for good —
operator merges (that repo auto-deploys nightly).
- Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the
Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host
key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during
the resolver mess).