recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
67226efe7284a0805a910714b6a1cba20fa549f6
cc-ci-orchestrator/cc-ci-plan/plan-cc-ci-digitalocean-terraform.md
T

7.8 KiB
Raw Blame History

Plan — cc-ci on DigitalOcean: terraform/ + nixos-infect + Nix provisioning

Status: PROPOSED. Add a terraform/ folder to the cc-ci product repo (recipe-maintainers/cc-ci) that provisions the cc-ci server on DigitalOcean (8 GB droplet), converts it to NixOS via nixos-infect, then applies the existing cc-ci flake config — making the CI server reproducible-from-scratch on real cloud hosting. Owner: a cc-ci-repo infra task (implementable by the Builder/Adversary loops as an infra unit, or by the assistant). This file: /srv/cc-ci/cc-ci-plan/plan-cc-ci-digitalocean-terraform.md.

0. Why

cc-ci currently runs as the Incus VM cc-nix-test on b1 — a small, shared 4-core host (the contention we kept hitting). A dedicated DO 8 GB droplet gives standard, reliable hosting with a public IP, fully reproducible via Terraform + the existing cc-ci NixOS flake. "Spin up cc-ci from nothing" becomes a terraform apply instead of hand-driven Incus API calls.

1. What already exists — build ON this, don't reinvent

  • cc-ci is a flake-based NixOS system: flake.nix → nixosConfigurations.cc-ci (pinned nixpkgs 24.11) → nix/hosts/cc-ci/{configuration.nix, hardware.nix} + nix/modules/* (proxy/traefik, drone, drone-runner, bridge, dashboard, backupbot, swarm, abra, harness, warm-keycloak, secrets).
  • From-scratch install is already VERIFIED (D8, docs/install.md): a blank NixOS host + the two repos (cc-ci cloned --recursive so the cc-ci-secrets submodule at secrets/ comes too) + the one bootstrap age key at /var/lib/sops-nix/key.txt → a single nixos-rebuild switch converges the whole server (0 failed units; serialized reconcile oneshots proxy→drone→bridge→dashboard→ backupbot). The wildcard TLS cert + all secrets are sops-encrypted in cc-ci-secrets (not out-of-band).
  • So "provision via Nix in the expected way" = that exact D8 flow: clone --recursive + bootstrap age key + nixos-rebuild switch --flake .#<host>.
  • The current nix/hosts/cc-ci/hardware.nix is Incus-VM-specific — DO needs its own hardware/bootloader/networking, which nixos-infect generates.

2. terraform/ layout (in recipe-maintainers/cc-ci)

terraform/
  versions.tf      # terraform + digitalocean/digitalocean provider, pinned
  variables.tf     # do_token(sensitive), region, size, ssh_key_id, ts_auth_key(sensitive), hostname
  main.tf          # digitalocean_droplet + ssh key + user_data
  outputs.tf       # droplet ipv4, id
  user-data.sh     # cloud-init stage-1: run nixos-infect (pinned)
  README.md        # apply instructions + operator inputs
  .gitignore       # *.tfstate*, *.auto.tfvars, .terraform/  (NEVER commit secrets/state)
  • Droplet: digitalocean_droplet — size s-4vcpu-8gb (8 GB RAM / 4 vCPU), region (e.g. nyc3), base image ubuntu-24-04-x64 (nixos-infect supports it), ssh_keys=[var.ssh_key_id], user_data=file("user-data.sh"), a stable name + tag. Optional: monitoring=true.
  • Secrets (DO token, TS key) are sensitive vars via TF_VAR_* env or a gitignored *.auto.tfvars; terraform.tfstate is gitignored (it can hold secrets). Mirrors cc-ci's no-secrets-in-git rule.

3. Stage 1 — nixos-infect (base Ubuntu → NixOS)

user-data.sh runs on first boot of the droplet:

#!/usr/bin/env bash
set -euo pipefail
export NIX_CHANNEL=nixos-24.11
curl -fsSL https://raw.githubusercontent.com/elitak/nixos-infect/<PINNED_SHA>/nixos-infect | bash -x

nixos-infect converts the droplet to NixOS in place, generates /etc/nixos/{configuration.nix, hardware-configuration.nix, networking.nix} (DO-correct: bootloader on the DO disk, public-IP networking via the DO metadata), and reboots into NixOS. Pin the nixos-infect revision — do not curl | bash master blind. After this, the droplet is bare NixOS on DO, ssh-able as root.

4. Stage 2 — provision via Nix (bare NixOS → converged cc-ci) — "the expected way"

  1. Capture DO hardware into the flake. Take the hardware-configuration.nix + networking.nix nixos-infect generated and add them as a flake host. Cleaner: a new host nix/hosts/cc-ci-do/ that imports the shared nix/modules/* + the DO hardware, with nixosConfigurations.cc-ci-do in flake.nix (keeps the Incus cc-ci host buildable during transition). Make DO the canonical cc-ci after cutover.
  2. Run the D8 install flow on the droplet: clone recipe-maintainers/cc-ci --recursive (brings cc-ci-secrets), provision the bootstrap age key at /var/lib/sops-nix/key.txt, then nixos-rebuild switch --flake .#cc-ci-do. The reconcile oneshots converge the swarm.
  3. Where stage 2 runs (recommendation): v1 = documented operator step (Terraform provisions + infects; the age-key placement + nixos-rebuild is the manual step, exactly like docs/install.md — the age key is operator-provided anyway). Automate later via a Terraform remote-exec provisioner or a cloud-init second stage once the key-delivery story is settled.

5. Operator inputs (class-A1 — provide at apply, NEVER commit)

  • DO API token (TF_VAR_do_token).
  • DO SSH key (registered on DO; the operator/agent holds the private half to ssh + run stage 2).
  • TS_AUTH_KEY — tailnet join (cc-ci enables tailscale; the droplet joins the same tailnet so the orchestrator/loops reach it exactly as today, direct peer).
  • Bootstrap age key → /var/lib/sops-nix/key.txt on the droplet (decrypts cc-ci-secrets incl. the wildcard TLS cert). The single out-of-band secret per docs/install.md.

6. DNS / gateway — a simplification DO enables (open decision)

Today *.ci.commoninternet.net reaches the Incus VM (no public IP) via an external nginx TLS-passthrough gateway → MagicDNS. A DO droplet has a public IP, so point ci.commoninternet.net + the *.ci wildcard A record straight at the droplet and drop the gateway — Traefik terminates TLS directly. The pre-issued sops wildcard cert still works as-is; or, with a public IP, switch Traefik to ACME (Let's Encrypt) and retire the manual cert + renewal. v1: keep the sops cert (no behavior change); evaluate ACME-on-public-IP as a follow-up. Record in DECISIONS.md.

7. Open decisions (log in DECISIONS.md)

  • Replace vs. parallel: stand DO up in parallel, verify a full !testme + the D-gates green on it, then cut DNS over and retire the Incus cc-nix-test. Nothing stateful is lost — recipes redeploy, warm canonicals re-seed on first green runs.
  • Flake host: parallel cc-ci-do host until cutover, then make DO the canonical cc-ci.
  • Droplet size/region; ACME vs sops cert (§6); stage-2 automation (§4.3).

8. Definition of Done

  • terraform/ in the cc-ci repo; terraform apply (with operator inputs) creates an 8 GB DO droplet and nixos-infect converts it to NixOS.
  • Given the bootstrap age key, the droplet converges to a full cc-ci via nixos-rebuild switch --flake .#<host> (the D8 flow) — 0 failed units; traefik/drone/bridge/dashboard/backupbot up.
  • A real recipe !testme runs green on the DO cc-ci; the dashboard + *.ci.commoninternet.net reachable via the chosen DNS path.
  • terraform/README.md documents apply + operator inputs; no secrets/state committed.
  • Adversary-verifiable: from-scratch reproducibility (the D8 guarantee) holds on DO.

9. Guardrails

  • No secrets in git (DO token, TS key, age key, tfstate all out-of-band/gitignored) — cc-ci's rule.
  • Pin everything (provider, nixos-infect rev; nixpkgs already pinned) — reproducible, no drift.
  • Don't break the running Incus cc-ci until the DO one is verified green (parallel bring-up + cutover).
  • Real Nix provisioning (the flake), not hand-installed packages.
Reference in New Issue View Git Blame Copy Permalink