notplantsandClaude Fable 5.1
77f156318e
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
391 lines
26 KiB
Markdown
391 lines
26 KiB
Markdown
# cc-ci-orchestrator
|
||
|
||
The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
|
||
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
|
||
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
|
||
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
|
||
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.
|
||
|
||
| | where |
|
||
|---|---|
|
||
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
|
||
| The host contract those need (loops user, opencode CLI, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
|
||
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
|
||
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
|
||
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
|
||
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
|
||
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |
|
||
|
||
Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
|
||
commit them.
|
||
|
||
---
|
||
|
||
# Deploying a cc-ci host from scratch
|
||
|
||
This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
|
||
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
|
||
can repeat it. Read it once before starting; the order matters.
|
||
|
||
## 0. What you need in hand
|
||
|
||
- A **Hetzner Cloud** project you can create servers in (console login or an API token).
|
||
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
|
||
public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
|
||
login is disabled and fail2ban watches sshd, so a key is the only way in.
|
||
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
|
||
read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` account has
|
||
it; the host gets its own deploy key on that account, §4b/§4c — and the bot password, in
|
||
`.testenv`, is what registers it).
|
||
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
|
||
you are starting fresh you create them (each row says how).
|
||
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
|
||
|
||
## 1. Provision the server on Hetzner (Debian image)
|
||
|
||
In the Hetzner Cloud console (or with `hcloud server create`):
|
||
|
||
| setting | value | why |
|
||
|---|---|---|
|
||
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
|
||
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
|
||
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
|
||
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
|
||
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
|
||
| Name | `cc-ci` | becomes the hostname |
|
||
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
|
||
|
||
Check you can log in: `ssh root@<ip> hostname`.
|
||
|
||
## 2. Convert Debian → NixOS with nixos-infect
|
||
|
||
[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
|
||
reboots. Run it detached so the SSH session dropping does not kill it:
|
||
|
||
```bash
|
||
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
|
||
#!/usr/bin/env bash
|
||
set -euo pipefail
|
||
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
|
||
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
|
||
export NIX_CHANNEL="nixos-26.05" # must match the nixpkgs channel in flake.nix
|
||
export PROVIDER="hetznercloud" # GRUB + Hetzner networking
|
||
export NIXOS_IMPORT="" # the real config comes from the flake in §5
|
||
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
|
||
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
|
||
export NO_SWAP=true
|
||
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
|
||
EOF
|
||
chmod +x /root/infect.sh
|
||
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
|
||
```
|
||
|
||
It downloads Nix, builds a NixOS system (5–10 min; follow with
|
||
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
|
||
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.
|
||
|
||
### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)
|
||
|
||
All three bit on the first attempt; the script above and §3 already include the fixes, this is
|
||
so you recognise them if they come back in another form.
|
||
|
||
1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
|
||
The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
|
||
Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
|
||
2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
|
||
(`dbus`, `systemd-logind`, `sshd`, networking …) with
|
||
`Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
|
||
nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
|
||
to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
|
||
lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
|
||
NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
|
||
(`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
|
||
move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
|
||
`etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
|
||
`root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
|
||
`disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
|
||
3. **Boots, units fine, but no network.** The generated `networking.nix` has
|
||
`defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
|
||
route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
|
||
`nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
|
||
mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
|
||
so turn it off for that one build:
|
||
```bash
|
||
for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
|
||
mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
|
||
chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
|
||
export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
|
||
export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
|
||
ln -sfn /nix/var/nix/profiles/system /run/current-system
|
||
nixos-rebuild boot --option sandbox false'
|
||
```
|
||
The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
|
||
from rescue mode) is what told these apart.
|
||
|
||
> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
|
||
> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
|
||
> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
|
||
> wraps these (token in `/srv/cc-ci/.hcloud-token`).
|
||
|
||
## 3. Capture the machine-specific config into this repo
|
||
|
||
nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
|
||
first two matter; the flake replaces `configuration.nix`.
|
||
|
||
```bash
|
||
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
|
||
scp root@<ip>:/etc/nixos/networking.nix nix/hosts/cc-ci/networking.nix
|
||
```
|
||
|
||
Then in `nix/hosts/cc-ci/`:
|
||
|
||
- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
|
||
`/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
|
||
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
|
||
`networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
|
||
the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
|
||
(as on the 2026-09 box) can stay.
|
||
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
|
||
is the release you installed (never change it later).
|
||
- `ssh-keys`: the root keys.
|
||
|
||
Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
|
||
|
||
## 4. Stage the workspace and secrets on the new host
|
||
|
||
Everything in this section is **outside git**. Do it as root over SSH, in this order.
|
||
|
||
### 4a. No tailscale
|
||
|
||
The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
|
||
public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
|
||
443 — traefik routes `oc.ci.commoninternet.net` to an nginx vhost on the docker bridge that
|
||
enforces basic auth, via traefik labels on a tiny swarm relay stack (`opencode-ui`) that
|
||
`opencode-ui-route.service` deploys. (Not via traefik's file *directory* provider: enabling
|
||
that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert
|
||
and the `security` middleware, and every front door goes down — it did, for two minutes, on
|
||
2026-09-07.) fail2ban guards sshd and that login
|
||
(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
|
||
traffic is docker-forwarded, not host INPUT).
|
||
|
||
### 4b. The CI server's checkout and its one out-of-band secret
|
||
|
||
```bash
|
||
# the bot deploy key for the private cc-ci-secrets submodule — generated ON this host by loops
|
||
# (§4c) and registered on the autonomic-bot Gitea account; root only points at it:
|
||
install -d -m700 /root/.ssh
|
||
cat > /root/.ssh/config <<'SSHCFG'
|
||
Host git.autonomic.zone
|
||
Port 2222
|
||
User git
|
||
IdentityFile /secrets/files/autonomic-bot-cc-ci-ed25519
|
||
IdentitiesOnly yes
|
||
SSHCFG
|
||
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
|
||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
|
||
# the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
|
||
# the master (recovery) key, temporarily. Once the host is a recipient (below): its own
|
||
# ssh-host-key-derived identity, and the master key leaves the box.
|
||
install -d -m700 /var/lib/sops-nix
|
||
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
|
||
```
|
||
|
||
`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH
|
||
host keys. The master key is enough for the FIRST deploy. Then make the new host a recipient so
|
||
the master key can leave the box again (it belongs off-box, with the operator):
|
||
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in
|
||
cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push,
|
||
bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then
|
||
replace `/var/lib/sops-nix/key.txt` with the host-derived identity
|
||
(`ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key`), rebuild, and delete the master key
|
||
from the host.
|
||
|
||
### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
|
||
run §5 once first if this is a fresh host, then come back)
|
||
|
||
```bash
|
||
sudo -iu loops
|
||
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
|
||
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci # every script and unit says /srv/cc-ci
|
||
cd /srv/cc-ci-orch
|
||
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci # Builder clone
|
||
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv # Adversary clone
|
||
mkdir -p .cc-ci-logs .sops
|
||
```
|
||
|
||
Secrets live in **`/secrets/files/`** (owned by loops, mode 700 on the directory), the same
|
||
convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one
|
||
`ls /secrets/files` is the complete inventory. **Only cc-ci's secrets go there** — nothing from
|
||
other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …):
|
||
secrets do not get spread around. `/secrets/README.txt` lists each file and its purpose.
|
||
|
||
| runtime path → `/secrets/files/…` | what | source |
|
||
|---|---|---|
|
||
| `/srv/cc-ci/.testenv` → `cc-ci.testenv` | `GITEA_PASSWORD` (autonomic-bot: PR/API calls), `DOCKERHUB_USERNAME/TOKEN` (harness image pulls). Nothing else: no tailscale key, no third-party model API keys (opencode's own auth is `opencode-auth.json`; add `TINFOIL_API_KEY` back only if `LOOP_MODEL` moves to a tinfoil model). | old host's copy minus the tailscale line; fresh: create each credential |
|
||
| `/srv/cc-ci/upgrader.env` (not a secret, lives in the checkout) | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run | old host, or copy the example in `AGENTS.md` |
|
||
| `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops |
|
||
| `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops |
|
||
| `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret) | opencode provider auth + config — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
|
||
| `/etc/nginx/oc-htpasswd` (root:nginx; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
|
||
|
||
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
|
||
`127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.
|
||
|
||
## 5. Build and activate
|
||
|
||
From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
|
||
|
||
```bash
|
||
# as root, detached (the activation restarts sshd; a dropped session must not kill it).
|
||
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
|
||
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
|
||
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
|
||
# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
|
||
# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
|
||
git config --global --add safe.directory '*'
|
||
systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
|
||
bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
|
||
journalctl -fu ccci-rebuild # ~10 min the first time (image pulls + two OCI image builds)
|
||
```
|
||
|
||
`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
|
||
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
|
||
(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
|
||
restarted tailscale under the session; this one is plain public ssh, but detached is still the
|
||
safe habit.)
|
||
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
|
||
|
||
The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
|
||
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
|
||
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
|
||
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
|
||
|
||
```bash
|
||
systemctl is-system-running # running — or list-units --failed and read journalctl -u <unit>
|
||
docker service ls # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
|
||
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
|
||
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
|
||
systemctl list-timers 'cc-ci-*' nightly-sweep
|
||
sudo -iu loops tmux ls # cc-ci-orchestrator (+ loops sessions if a phase is active)
|
||
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
|
||
# /var/lib/acme is restored or a cert has been issued):
|
||
curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
|
||
curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
|
||
curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
|
||
dig +short @<ip> ns-acme.commoninternet.net # acme-dns answering on the public 53
|
||
```
|
||
|
||
The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
|
||
→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
|
||
counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to
|
||
/var/log/nginx/error.log).
|
||
|
||
When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
|
||
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
|
||
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
|
||
the new box does not process PR comments or start a second weekly run while the old host is live:
|
||
|
||
```bash
|
||
docker service scale ccci-bridge_app=0
|
||
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
|
||
```
|
||
|
||
## 6. Migrating: restore state from the previous host
|
||
|
||
Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
|
||
stopped on the new host while its directory is copied:
|
||
|
||
| path | holds | notes |
|
||
|---|---|---|
|
||
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
|
||
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
|
||
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
|
||
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
|
||
| `/var/lib/acme-dns` | the acme-dns zone DB | |
|
||
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
|
||
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
|
||
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
|
||
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
|
||
|
||
## 7. Cutover and verification
|
||
|
||
1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
|
||
the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
|
||
are. Wait for propagation (`dig +short ci.commoninternet.net`) — the records carry a 3 h TTL,
|
||
and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that
|
||
long. The new host must not list such a resolver: with a mixed set, every other lookup of its
|
||
own names went to the old server on 2026-09-07 (runner/bridge 404s). `networking.nix` drops
|
||
the Hetzner resolvers and `configuration.nix` pins the host's own names.
|
||
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
|
||
(acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
|
||
(dashboard, valid cert), same for `report.ci` and `drone.ci`.
|
||
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
|
||
`systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
|
||
orchestrator. New host: `docker service scale ccci-bridge_app=1`;
|
||
`systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
|
||
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
|
||
open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
|
||
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
|
||
(or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
|
||
claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
|
||
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
|
||
old servers were on the tailnet; the new one is not).
|
||
|
||
## 8. Day 2
|
||
|
||
- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
|
||
`nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
|
||
this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
|
||
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
|
||
`nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
|
||
so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a
|
||
lock written by a newer Nix elsewhere once failed here with `NAR hash mismatch in input
|
||
git+https://…cc-ci.git…&shallow=1`; re-running `nix flake update cc-ci` on the host fetched the
|
||
input properly and the rebuild went through.
|
||
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
|
||
the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
|
||
`switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
|
||
`/hetzner-server-recovery`).
|
||
|
||
---
|
||
|
||
# Operating the orchestrator
|
||
|
||
The cc-ci orchestrator on this host is an **opencode agent** (`cc-ci-plan/agents.toml`:
|
||
backend `opencode`, model `opencode/glm-5.2`), launched on boot by `cc-ci-orchestrator.service`
|
||
via `cc-ci-plan/agents.py up orchestrator` into tmux session `cc-ci-orchestrator`, attached to
|
||
the shared opencode web server. Steer it from the web UI, **https://oc.ci.commoninternet.net**
|
||
(basic auth, user `oc`), where the weekly upgrader's and report's sessions show up too.
|
||
There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and
|
||
operate the CI from there over ssh (`Host cc-ci` in that box's loops ssh config → this host).
|
||
|
||
```bash
|
||
# on the host
|
||
sudo -iu loops tmux attach -t cc-ci-orchestrator # the agent's terminal
|
||
systemctl restart cc-ci-orchestrator # relaunch it (fresh opencode session)
|
||
sudo -iu loops tmux ls # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run
|
||
```
|
||
|
||
## Weekly upgrade run
|
||
|
||
`cc-ci-upgrade-all.timer` (Thu 22:00 America/New_York) starts `launch-upgrader.py`, which runs
|
||
`/upgrade-all` as an opencode agent (`upgrader.env`: `LOOP_MODEL`/`REPORT_MODEL`) in tmux
|
||
`cc-ci-upgrader` with a usage-limit watchdog; the hourly `cc-ci-upgrade-supervisor.timer` drives a
|
||
stalled run to completion. Start one by hand with `systemctl start cc-ci-upgrade-all`; follow
|
||
`/srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log`; it ends with `UPGRADE RUN COMPLETE`, a summary in
|
||
`.cc-ci-logs/upgrades/upgrade-all-<date>.md`, and a report published to
|
||
`https://report.ci.commoninternet.net/week-<date>.html`.
|
||
|
||
## Kick off / supervise the loops
|
||
|
||
```bash
|
||
cd /srv/cc-ci/cc-ci-plan
|
||
./launch.sh start # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
|
||
./launch.sh status # session + DONE state
|
||
./launch.sh logs builder|adversary|watchdog
|
||
./launch.sh stop
|
||
```
|
||
|
||
Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
|
||
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.
|