173 lines
11 KiB
Markdown
173 lines
11 KiB
Markdown
# Plan: migrate cc-ci domains from `*.commoninternet.net` to `ci.autonomic.zone`
|
||
|
||
**Date:** 2026-09-21 · **Status:** DRAFT — blocked only on operator DNS access (Gandi zone `autonomic.zone`)
|
||
**Principle:** same host, same IP (195.201.88.249), same services — only names change. Dual-run new
|
||
alongside old; old names retire last; rollback at any point is "change nothing further" because the
|
||
old DNS zone keeps answering until Phase 4.
|
||
|
||
---
|
||
|
||
## 1. Inventory — what answers on what today
|
||
|
||
| Name | Served by | Router/config source |
|
||
|---|---|---|
|
||
| `ci.commoninternet.net` | results dashboard + bridge `/hook` | `cc-ci/nix/modules/dashboard.nix`, `bridge.nix` (traefik Host rules) |
|
||
| `report.ci.commoninternet.net` | weekly report static site | `cc-ci/nix/modules/reports.nix` |
|
||
| `drone.ci.commoninternet.net` | Drone server | `cc-ci/nix/modules/drone.nix` (abra app; env `DOMAIN`, `DRONE_SERVER_HOST`); runner RPC: `drone-runner.nix` |
|
||
| `oc.ci.commoninternet.net` | opencode UI (traefik → host nginx basic auth) | `cc-ci-orch/nix/modules/orchestrator-host.nix` (`opencodeUiHost`), set in `nix/hosts/cc-ci/configuration.nix` |
|
||
| `traefik.ci.commoninternet.net` | traefik stack identity + deploy-proxy health gate | `cc-ci/nix/modules/proxy.nix` |
|
||
| `warm-*.ci.commoninternet.net` | warm canonical recipe stacks | `warm-keycloak.nix`, warm-bluesky-pds (caddy labels) |
|
||
| `<tag>-<6hex>.ci.commoninternet.net` | per-run recipe test deploys | `cc-ci/runner/harness/naming.py` `app_domain()` — **resolves via public DNS** (containers use 1.1.1.1/8.8.8.8) |
|
||
| `backups.ci.commoninternet.net` | backup-bot-two — stack name only, no web route | `backupbot.nix` |
|
||
| `acme.commoninternet.net` (NS `ns-acme`) | local acme-dns (systemd, port 53) | `cc-ci/nix/modules/acme-dns.nix` |
|
||
|
||
**Cert chain today:** nixos `security.acme` (lego) issues `ci.commoninternet.net` + `*.ci.commoninternet.net`
|
||
via DNS-01 through the local acme-dns; Gandi delegates `_acme-challenge.ci.commoninternet.net` CNAME →
|
||
`9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net`. `postRun` fires
|
||
`cc-ci-acme-traefik-handoff.service` → installs cert into `/var/lib/ci-certs/live` → restarts
|
||
`deploy-proxy.service`, whose reconciler re-derives the `traefik_ci_commoninternet_net_ssl_{cert,key}_v<hash>`
|
||
swarm secrets (traefik runs co-op-cloud wildcard/no-ACME mode off those secrets).
|
||
**DNS:** both zones on Gandi. IPv4-only (no AAAA records anywhere; none needed).
|
||
|
||
**External references:** Gitea OAuth app for Drone (redirect URI → drone.ci…); per-repo webhooks →
|
||
`/hook` — **only `recipe-maintainers/ghost` and `recipe-maintainers/discourse` have any** (everything
|
||
else is bridge-driven via the Drone API); ~60 doc references across cc-ci-orch (AGENTS.md, README,
|
||
plan.md, skills).
|
||
|
||
## 2. Target naming
|
||
|
||
| Old | New |
|
||
|---|---|
|
||
| ci.commoninternet.net | ci.autonomic.zone |
|
||
| report.ci.commoninternet.net | report.ci.autonomic.zone |
|
||
| drone.ci.commoninternet.net | drone.ci.autonomic.zone |
|
||
| oc.ci.commoninternet.net | oc.ci.autonomic.zone |
|
||
| traefik.ci.commoninternet.net | traefik.ci.autonomic.zone |
|
||
| *.ci.commoninternet.net (warm-\*, per-run apps, backups identity) | *.ci.autonomic.zone |
|
||
| acme.commoninternet.net | **stays** through Phases 1–2 (infra, not a front door); optional Phase 3 |
|
||
|
||
Swarm **stack names** (`traefik_ci_commoninternet_net`, `ccci-*`, warm stacks) are identifiers, not
|
||
domains — keep them where possible to avoid churn in keep-lists/prune filters, EXCEPT abra-managed
|
||
apps whose name **is** the domain (drone, backup-bot, per-run apps).
|
||
|
||
## 3. DNS records to set (zone `autonomic.zone`, Gandi — operator)
|
||
|
||
| # | Name | Type | Value | TTL | Purpose |
|
||
|---|---|---|---|---|---|
|
||
| 1 | `ci.autonomic.zone` | A | `195.201.88.249` | 300 | apex front door (wildcard does NOT cover the apex) |
|
||
| 2 | `*.ci.autonomic.zone` | A | `195.201.88.249` | 300 | report/drone/oc/traefik/warm-\*/`<tag>-<6hex>` run apps |
|
||
| 3 | `_acme-challenge.ci.autonomic.zone` | CNAME | `9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` | 300 | DNS-01 for the new wildcard — **reuses the existing restricted acme-dns account** (`disable_registration=true`; the lego *account* authenticates, not the FQDN, so one acme-dns subdomain serves both zones) |
|
||
|
||
Also: lower TTL to 300 on the old `ci`/`*.ci` records a few days ahead (fast rollback later).
|
||
Phase 3 (optional) adds: `acme.ci.autonomic.zone` + `ns-acme.ci.autonomic.zone` A records.
|
||
|
||
**DNS verification (before touching the host):**
|
||
```bash
|
||
dig +short ci.autonomic.zone A @ns-187-a.gandi.net # 195.201.88.249
|
||
dig +short report.ci.autonomic.zone @1.1.1.1 # 195.201.88.249 (via wildcard)
|
||
dig +short drone.ci.autonomic.zone @8.8.8.8 # 195.201.88.249
|
||
dig +short _acme-challenge.ci.autonomic.zone CNAME # 9d3f0428-….acme.commoninternet.net.
|
||
```
|
||
|
||
## 4. Phase 1a — certificate (issue BEFORE any routing change)
|
||
|
||
1. `cc-ci/nix/modules/acme-dns.nix`: add a second `security.acme.certs."ci.autonomic.zone"` entry
|
||
(domain + `*.ci.autonomic.zone` SAN, `dnsProvider = "acmedns"`, `dnsResolver = "1.1.1.1:53"`).
|
||
Split the lego env per cert — today `ACME_DNS_STORAGE_PATH` is baked to
|
||
`/var/lib/acme/ci.commoninternet.net/…`; generate one env file per cert so each keeps its own
|
||
acme-dns account file.
|
||
2. Extend `cc-ci-acme-traefik-handoff` to stage the new cert alongside the old
|
||
(`/var/lib/ci-certs/live` holds both, e.g. `fullchain.pem` = new + `fullchain-old.pem`).
|
||
3. Extend the **deploy-proxy reconciler** (`proxy.nix`) for **dual-cert SNI**: new wildcard pair as
|
||
`ssl_cert/ssl_key` (default store) + old pair as a second secret pair wired through a small
|
||
file-provider `[[tls.certificates]]` so `*.ci.commoninternet.net` keeps serving during the bake
|
||
window. This is the one fiddly bit — validate against the **Let's Encrypt staging** endpoint
|
||
first (the module already has the staging-marker pattern).
|
||
4. `nixos-rebuild test` → confirm both certs issued and served (`openssl s_client -servername` for a
|
||
name in each zone, check SANs) → `switch`.
|
||
|
||
## 5. Phase 1b — routing + service config (one stack at a time, health-gated)
|
||
|
||
During the bake window every router rule carries **both** hostnames
|
||
(`Host(`ci.autonomic.zone`) || Host(`ci.commoninternet.net`)`); old names drop out only in Phase 4.
|
||
|
||
- `dashboard.nix` + `bridge.nix` (share the `ci` host: dashboard `/` + bridge `/hook`), `reports.nix`,
|
||
and the traefik health-gate name in `proxy.nix`.
|
||
- `bridge.py` / `dashboard.py` code defaults for `DRONE_URL` / `DASH_URL`.
|
||
- `recipe-report.py` — footer, publish URL text and its verification curl → `report.ci.autonomic.zone`
|
||
(regenerate the index after, so the home page stops linking the old name).
|
||
- Drone **abra app rename** (name = domain, so this is a new stack):
|
||
1. Export enabled-repo list from the old Drone API.
|
||
2. Deploy new abra app `drone.ci.autonomic.zone` (same module; secrets `rpc_secret`,
|
||
`client_secret` re-insert from `/run/secrets` as today). **Fresh DB** — Drone data volume is
|
||
stack-name-prefixed; repos re-sync on Gitea login, build history is lost (accepted; export the
|
||
old repo list and re-enable via API).
|
||
3. Point `drone-runner.nix` `DRONE_RPC_HOST` at the new name; restart runner.
|
||
4. Re-run `scripts/bootstrap-drone-oauth.sh` with `DRONE=https://drone.ci.autonomic.zone`
|
||
(updates the Gitea OAuth redirect URI; script already uses `--resolve` to hit loopback).
|
||
5. Update the two webhooks (ghost, discourse) to `https://drone.ci.autonomic.zone/hook?secret=<same>`
|
||
— read the secret off the existing hook config before deleting it.
|
||
6. `abra app undeploy` + remove the old drone app once green.
|
||
- `configuration.nix`: `opencodeUiHost = "oc.ci.autonomic.zone"`; extend the `networking.hosts`
|
||
self-pin list with the new names (keep old pins during bake).
|
||
- Gitea `GITEA_SERVER` stays `git.autonomic.zone` — unchanged.
|
||
|
||
## 6. Phase 2 — harness / warm / per-run naming (the `_ci_commoninternet_net` blast radius)
|
||
|
||
- `cc-ci/runner/harness/naming.py` `app_domain()` → `.ci.autonomic.zone` (single source of truth).
|
||
- Regexes/patterns that match the derived names: `docker-prune.nix` run-app filter,
|
||
`tests/concurrency/concutil.py` (`DOMAIN` const + `RUN_APP_RE`), any separate `RUN_APP_RE` in
|
||
`lifecycle.py` / `warm.py` / `warm_reconcile.py` / `canonical.py` / `generic.py` / `http.py`,
|
||
`nightly-sweep` keep-list patterns, `tests/gitea/recipe_meta.py` comment.
|
||
- Redeploy warm stacks under new names (abra `warm-<recipe>.ci.autonomic.zone`), reconcile, drain old.
|
||
- `backupbot.nix` `DOMAIN` (stack-name-only change).
|
||
- Verify: full deterministic test suite + one live `!testme` run end-to-end.
|
||
|
||
## 7. Phase 3 (optional) — move acme-dns off commoninternet.net
|
||
|
||
Only if `commoninternet.net` is being dropped entirely: new `domain`/`nsname` in acme-dns config
|
||
(`acme.ci.autonomic.zone` / `ns-acme.ci.autonomic.zone` + Gandi records), briefly re-enable
|
||
registration or hand-migrate the account DB, re-point `_acme-challenge` CNAMEs, re-issue both certs.
|
||
|
||
## 8. Phase 4 — retirement (after ≥7 clean days)
|
||
|
||
Remove old names from the dual router rules → remove `security.acme.certs."ci.commoninternet.net"`
|
||
+ its env file + old cert secret pair → drop old `networking.hosts` pins → delete old Gandi records
|
||
(`ci`, `*.ci`, `_acme-challenge.ci`) → restore TTLs to 3600. Keep `acme.commoninternet.net` unless
|
||
Phase 3 was done.
|
||
|
||
## 9. Verification checklist
|
||
|
||
- **DNS:** the dig table in §3 against a Gandi NS and a public resolver.
|
||
- **Cert:** `openssl s_client -connect 195.201.88.249:443 -servername <name>` for one name per zone —
|
||
SANs correct for both, `ssl_verify_result=0`.
|
||
- **Front doors:** `ci` → 200 · `report.ci` → 200 · `drone.ci` → 200/303 · `oc.ci` → **401**
|
||
(basic auth intact) · `traefik.ci/api/version` → 200.
|
||
- **SSO:** Drone login through Gitea on the new redirect URI.
|
||
- **Webhook:** empty commit to `recipe-maintainers/ghost` main → build fires on the new host.
|
||
- **Bridge/!testme:** `!testme` on a scratch PR → build runs, `results.json` written, PR comment
|
||
contains **new** domain links.
|
||
- **Report:** publish one report page and verify it at `https://report.ci.autonomic.zone`.
|
||
- **Host:** `systemctl --failed` empty; `deploy-proxy` reconcile logs clean.
|
||
|
||
## 10. Risks / gotchas
|
||
|
||
- **Dual-cert SNI in the traefik reconciler** is the highest-risk change — stage-first, verify SNI
|
||
selection for both zones before switching.
|
||
- **Drone DB** does not follow the abra rename — accept fresh build history; export/re-apply the
|
||
enabled-repo list; re-insert secrets from `/run/secrets` (module already does).
|
||
- **Per-run recipe domains resolve via public DNS** — the `*.ci.autonomic.zone` wildcard A record is
|
||
load-bearing for every test deploy; never merge Phase 2 before it propagates.
|
||
- Old PR comments/badges link old names — they keep working until Phase 4 removes the old records.
|
||
- ~60 doc references (AGENTS.md, README, plan.md, skills, status skill) — one cosmetic doc PR at the
|
||
end so the docs don't lie.
|
||
- No domain-bearing secret values were found (webhook secrets are per-repo and copied as-is).
|
||
|
||
## 11. Sequencing
|
||
|
||
1. **Operator:** TTLs down + 3 Gandi records (§3) — 5 minutes.
|
||
2. **Agent:** cert PR (§4) → `test` → verify dual SNI → `switch`.
|
||
3. **Agent:** routing/env PRs (§5) stack-by-stack, health-gated; OAuth + webhooks + drone rename.
|
||
4. **Agent:** end-to-end verification (§9) + report republish under the new name.
|
||
5. **Bake ≥7 days** → Phase 2 PR → full suite → bake → Phase 4 retirement.
|