cc-ci-orchestrator/README.md

# cc-ci-orchestrator

The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.

| | where |
|---|---|
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
| The host contract those need (loops user, opencode CLI, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |

Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
commit them.

---

# Deploying a cc-ci host from scratch

This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
can repeat it. Read it once before starting; the order matters.

## 0. What you need in hand

- A **Hetzner Cloud** project you can create servers in (console login or an API token).
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
  login is disabled and fail2ban watches sshd, so a key is the only way in.
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
  read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` account has
  it; the host gets its own deploy key on that account, §4b/§4c — and the bot password, in
  `.testenv`, is what registers it).
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
  you are starting fresh you create them (each row says how).
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.

## 1. Provision the server on Hetzner (Debian image)

In the Hetzner Cloud console (or with `hcloud server create`):

| setting | value | why |
|---|---|---|
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
| Name | `cc-ci` | becomes the hostname |
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |

Check you can log in: `ssh root@<ip> hostname`.

## 2. Convert Debian → NixOS with nixos-infect

[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
reboots. Run it detached so the SSH session dropping does not kill it:

```bash
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
export NIXOS_IMPORT=""             # the real config comes from the flake in §5
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
export NO_SWAP=true
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
EOF
chmod +x /root/infect.sh
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
```

It downloads Nix, builds a NixOS system (5–10 min; follow with
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.

### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

All three bit on the first attempt; the script above and §3 already include the fixes, this is
so you recognise them if they come back in another form.

1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
   The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
   Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
   (`dbus`, `systemd-logind`, `sshd`, networking …) with
   `Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
   nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
   to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
   lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
   NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
   (`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
   move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
   `etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
   `root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
   `disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
3. **Boots, units fine, but no network.** The generated `networking.nix` has
   `defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
   route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
   `nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
   mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
   so turn it off for that one build:
   ```bash
   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
     ln -sfn /nix/var/nix/profiles/system /run/current-system
     nixos-rebuild boot --option sandbox false'
   ```
   The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
   from rescue mode) is what told these apart.

> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
> wraps these (token in `/srv/cc-ci/.hcloud-token`).

## 3. Capture the machine-specific config into this repo

nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
first two matter; the flake replaces `configuration.nix`.

```bash
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix
```

Then in `nix/hosts/cc-ci/`:

- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
  `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
  the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
  (as on the 2026-09 box) can stay.
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
  is the release you installed (never change it later).
- `ssh-keys`: the root keys.

Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.

## 4. Stage the workspace and secrets on the new host

Everything in this section is **outside git**. Do it as root over SSH, in this order.

### 4a. No tailscale

The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
443 — traefik routes `oc.ci.commoninternet.net` to an nginx vhost on the docker bridge that
enforces basic auth, via traefik labels on a tiny swarm relay stack (`ccci-opencode-ui`, on the weekly sweep's keep-list) that
`opencode-ui-route.service` deploys. (Not via traefik's file *directory* provider: enabling
that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert
and the `security` middleware, and every front door goes down — it did, for two minutes, on
2026-09-07.) fail2ban guards sshd and that login
(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
traffic is docker-forwarded, not host INPUT).

### 4b. The CI server's checkout and its one out-of-band secret

```bash
# the bot deploy key for the private cc-ci-secrets submodule — generated ON this host by loops
# (§4c) and registered on the autonomic-bot Gitea account; root only points at it:
install -d -m700 /root/.ssh
cat > /root/.ssh/config <<'SSHCFG'
Host git.autonomic.zone
    Port 2222
    User git
    IdentityFile /secrets/files/autonomic-bot-cc-ci-ed25519
    IdentitiesOnly yes
SSHCFG
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
# the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
# the master (recovery) key, temporarily. Once the host is a recipient (below): its own
# ssh-host-key-derived identity, and the master key leaves the box.
install -d -m700 /var/lib/sops-nix
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
```

`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH
host keys. The master key is enough for the FIRST deploy. Then make the new host a recipient so
the master key can leave the box again (it belongs off-box, with the operator):
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in
cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push,
bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then
replace `/var/lib/sops-nix/key.txt` with the host-derived identity
(`ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key`), rebuild, and delete the master key
from the host.

### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
run §5 once first if this is a fresh host, then come back)

```bash
sudo -iu loops
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
cd /srv/cc-ci-orch
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
mkdir -p .cc-ci-logs .sops
```

Secrets live in **`/secrets/files/`** (owned by loops, mode 700 on the directory), the same
convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one
`ls /secrets/files` is the complete inventory. **Only cc-ci's secrets go there** — nothing from
other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …):
secrets do not get spread around. `/secrets/README.txt` lists each file and its purpose.

| runtime path → `/secrets/files/…` | what | source |
|---|---|---|
| `/srv/cc-ci/.testenv` → `cc-ci.testenv` | `GITEA_PASSWORD` (autonomic-bot: PR/API calls), `DOCKERHUB_USERNAME/TOKEN` (harness image pulls). Nothing else: no tailscale key, no third-party model API keys (opencode's own auth is `opencode-auth.json`; add `TINFOIL_API_KEY` back only if `LOOP_MODEL` moves to a tinfoil model). | old host's copy minus the tailscale line; fresh: create each credential |
| `/srv/cc-ci/upgrader.env` (not a secret, lives in the checkout) | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run | old host, or copy the example in `AGENTS.md` |
| `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops |
| `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops |
| `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret) | opencode provider auth + config — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
| `/etc/nginx/oc-htpasswd` (root:nginx; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |

`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
`127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.

## 5. Build and activate

From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):

```bash
# as root, detached (the activation restarts sshd; a dropped session must not kill it).
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
git config --global --add safe.directory '*'
systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)
```

`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
restarted tailscale under the session; this one is plain public ssh, but detached is still the
safe habit.)
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.

The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:

```bash
systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
systemctl list-timers 'cc-ci-*' nightly-sweep
sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
# /var/lib/acme is restored or a cert has been issued):
curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
```

The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to
/var/log/nginx/error.log).

When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
the new box does not process PR comments or start a second weekly run while the old host is live:

```bash
docker service scale ccci-bridge_app=0
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
```

## 6. Migrating: restore state from the previous host

Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
stopped on the new host while its directory is copied:

| path | holds | notes |
|---|---|---|
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
| `/var/lib/acme-dns` | the acme-dns zone DB | |
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |

## 7. Cutover and verification

1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
   the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
   are. Wait for propagation (`dig +short ci.commoninternet.net`) — the records carry a 3 h TTL,
   and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that
   long. The new host must not list such a resolver: with a mixed set, every other lookup of its
   own names went to the old server on 2026-09-07 (runner/bridge 404s). `networking.nix` drops
   the Hetzner resolvers and `configuration.nix` pins the host's own names.
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
   (acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
   (dashboard, valid cert), same for `report.ci` and `drone.ci`.
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
   `systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
   orchestrator. New host: `docker service scale ccci-bridge_app=1`;
   `systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
   open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
   (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
   claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
   old servers were on the tailnet; the new one is not).

## 8. Day 2

- **The host updates itself weekly**: `cc-ci-auto-update.timer` (Tuesday 03:00 UTC,
  `nix/modules/auto-update.nix`) runs `nix flake update` on all inputs, builds, activates with
  `test`, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…),
  and only then commits the generation + bootloader and pushes the health-checked `flake.lock` to
  `main`; on any failure it re-activates the previous generation and restores the lock. It skips
  itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome
  in `/srv/cc-ci/.cc-ci-logs/auto-update-state` (one line) and `journalctl -u cc-ci-auto-update`;
  `/cc-ci-status` §5 reads it. `/cc-ci-orchestrator-update` runs the same unit by hand.
  The `cc-ci` input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
  `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
  so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a
  lock written by a newer Nix elsewhere once failed here with `NAR hash mismatch in input
  git+https://…cc-ci.git…&shallow=1`; re-running `nix flake update cc-ci` on the host fetched the
  input properly and the rebuild went through.
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
  the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
  `switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
  `/hetzner-server-recovery`).

---

# Operating the orchestrator

The cc-ci orchestrator on this host is an **opencode agent** (`cc-ci-plan/agents.toml`:
backend `opencode`, model `opencode/glm-5.2`), launched on boot by `cc-ci-orchestrator.service`
via `cc-ci-plan/agents.py up orchestrator` into tmux session `cc-ci-orchestrator`, attached to
the shared opencode web server. Steer it from the web UI, **https://oc.ci.commoninternet.net**
(basic auth, user `oc`), where the weekly upgrader's and report's sessions show up too.
There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and
operate the CI from there over ssh (`Host cc-ci` in that box's loops ssh config → this host).

```bash
# on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator     # the agent's terminal
systemctl restart cc-ci-orchestrator                 # relaunch it (fresh opencode session)
sudo -iu loops tmux ls                               # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run
```

## Weekly upgrade run

`cc-ci-upgrade-all.timer` (Thu 22:00 America/New_York) starts `launch-upgrader.py`, which runs
`/upgrade-all` as an opencode agent (`upgrader.env`: `LOOP_MODEL`/`REPORT_MODEL`) in tmux
`cc-ci-upgrader` with a usage-limit watchdog; the hourly `cc-ci-upgrade-supervisor.timer` drives a
stalled run to completion. Start one by hand with `systemctl start cc-ci-upgrade-all`; follow
`/srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log`; it ends with `UPGRADE RUN COMPLETE`, a summary in
`.cc-ci-logs/upgrades/upgrade-all-<date>.md`, and a report published to
`https://report.ci.commoninternet.net/week-<date>.html`.

## Kick off / supervise the loops

```bash
cd /srv/cc-ci/cc-ci-plan
./launch.sh start                       # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
./launch.sh status                      # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop
```

Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.