cc-ci-orchestrator

The cc-ci orchestrator: the agent loops that built the cc-ci Co-op Cloud recipe CI server, the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS host they run on. Since 2026-09 that host is the same Hetzner server as the CI server itself: one nixos-rebuild from this repo builds both, because this flake imports the CI server as a module from the cc-ci repo.

	where
Orchestrator loops, timers (weekly upgrader, hourly supervisor)	nix/modules/cc-ci.nix → nixosModules.cc-ci-orchestrator
The host contract those need (loops user, opencode CLI, opencode web UI)	nix/modules/orchestrator-host.nix → nixosModules.orchestrator-host
The CI server (swarm, traefik, drone, runner, !testme bridge, dashboard, reports, acme-dns)	cc-ci repo nix/modules/ → nixosModules.cc-ci-server (flake input cc-ci)
The machine: hardware, networking, root keys, firewall + fail2ban	nix/hosts/cc-ci/ → nixosConfigurations.cc-ci
Plans, launch tooling, loop prompts, journal	cc-ci-plan/ (see AGENTS.md for roles)
Skills the orchestrator runs (/upgrade-all, /recipe-upgrade, /cc-ci-status, …)	.claude/skills/, .opencode/skills/
How it used to be built (Pi → Incus VM → shared Hetzner box)	archive/

Secrets (.testenv, upgrader.env, .sops/, everything under /secrets) are gitignored — never commit them.

---

Deploying a cc-ci host from scratch

This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner server. It was last done on 2026-09-07 for 195.201.88.249 and is written so a person or an LLM can repeat it. Read it once before starting; the order matters.

0. What you need in hand

• A Hetzner Cloud project you can create servers in (console login or an API token).
• SSH keys: yours, and the orchestrator's own key so the automation can reach the box. The public keys that get root are tracked in nix/hosts/cc-ci/ssh-keys (one per line). Password login is disabled and fail2ban watches sshd, so a key is the only way in.
• Read access to recipe-maintainers/cc-ci, recipe-maintainers/cc-ci-orchestrator (both public read) and the private recipe-maintainers/cc-ci-secrets (the autonomic-bot account has it; the host gets its own deploy key on that account, §4b/§4c — and the bot password, in .testenv, is what registers it).
• The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if you are starting fresh you create them (each row says how).
• Control of the DNS zone (Gandi for commoninternet.net) for the cutover in §7.

1. Provision the server on Hetzner (Debian image)

In the Hetzner Cloud console (or with hcloud server create):

setting	value	why
Image	Debian 13 (any recent Debian/Ubuntu works with nixos-infect)	it is replaced by NixOS in §2
Type	x86, 8 GB RAM, 4 vCPU — e.g. cpx32 (dedicated AMD) or cx33. Never cax* (ARM): the flakes are x86_64-linux.	swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small
Disk	the type's default 150+ GB NVMe	docker layers alone are ~60 GB after a few weeks
Network	public IPv4 required; IPv6 optional (leave enabled or not, NixOS config ignores it)	cc-ci serves 80/443 and DNS on 53 publicly
SSH keys	add every key from nix/hosts/cc-ci/ssh-keys you want to log in with, at least the orchestrator's	nixos-infect carries /root/.ssh/authorized_keys over
Name	cc-ci	becomes the hostname
Firewall	if a Hetzner Cloud Firewall is attached it must allow 22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp in, and ICMP	the NixOS firewall is separate and is configured by the flake

Check you can log in: ssh root@<ip> hostname.

2. Convert Debian → NixOS with nixos-infect

nixos-infect installs NixOS over the running Debian and reboots. Run it detached so the SSH session dropping does not kill it:

ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
export NIXOS_IMPORT=""             # the real config comes from the flake in §5
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
export NO_SWAP=true
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
EOF
chmod +x /root/infect.sh
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'

It downloads Nix, builds a NixOS system (5–10 min; follow with ssh root@<ip> tail -f /var/log/nixos-infect.log), then reboots. The SSH host key changes: ssh-keygen -R <ip> and confirm ssh root@<ip> nixos-version prints a 26.05 version.

2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

All three bit on the first attempt; the script above and §3 already include the fixes, this is so you recognise them if they come back in another form.

1. swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument right at the start, script exits. The Debian 13 cloud image mounts /tmp as tmpfs and a swapfile cannot live there. Fix: NO_SWAP=true (in the script above). An 8 GB box does not need the temporary swap.
2. The box never comes back after the reboot: it boots NixOS, but nearly every unit fails (dbus, systemd-logind, sshd, networking …) with Could not start dynamically linked executable: /usr/bin/dbus-daemon in the journal. nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it to /old-root (/etc/NIXOS_LUSTRATE). With NixOS 26.05's systemd-based initrd that lustration did not happen, so Debian's /etc/systemd/system/*.service files shadowed the NixOS units and started Debian binaries. Fix, from Hetzner rescue mode (enable_rescue + reset in the API/console, ssh in, mount /dev/sda1 /mnt/root): move everything except nix, boot, swapfile, lost+found, var/log, var/empty, etc/nixos, etc/resolv.conf, etc/NIXOS, etc/machine-id, etc/ssh/ssh_host_*, root/.nix-*, root/.ssh into /mnt/root/old-root, delete etc/NIXOS_LUSTRATE, unmount, disable_rescue, reset. (/old-root, ~1 GB, can be deleted once the host is in service.)
3. Boots, units fine, but no network. The generated networking.nix has defaultGateway = "172.31.1.1"; — a bare string. Since NixOS 25.05 that yields no default route. Fix: defaultGateway = { address = "172.31.1.1"; interface = "eth0"; }; (this is what nix/hosts/cc-ci/networking.nix carries). To apply it from rescue mode, chroot into the mounted root and rebuild the boot entry — the nix sandbox cannot pivot_root inside a chroot, so turn it off for that one build:

   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
     ln -sfn /nix/var/nix/profiles/system /run/current-system
     nixos-rebuild boot --option sandbox false'
   

   The journalctl -D /mnt/root/var/log/journal -b 0 trick (reading the dead system's journal from rescue mode) is what told these apart.

Rescue mode without a console: POST /servers/<id>/actions/enable_rescue with your ssh key id, then …/actions/reset; afterwards disable_rescue and check rescue_enabled is false before the next reset, or it boots the rescue image again. scripts/recovery/hetzner.py wraps these (token in /srv/cc-ci/.hcloud-token).

3. Capture the machine-specific config into this repo

nixos-infect wrote /etc/nixos/{hardware-configuration,networking,configuration}.nix. Only the first two matter; the flake replaces configuration.nix.

scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix

Then in nix/hosts/cc-ci/:

• hardware.nix: keep as generated (GRUB EFI with efiInstallAsRemovable, /boot/efi by UUID, /dev/sda1 root). Do not copy another host's file — the UUIDs are per machine.
• networking.nix: keep the static IPv4 + Hetzner gateway 172.31.1.1. Make sure networking.defaultGateway has both address and interface = "eth0" (§2a item 3). If the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address (as on the 2026-09 box) can stay.
• configuration.nix: set cc-ci.publicIPv4 to the server's IPv4 and check system.stateVersion is the release you installed (never change it later).
• ssh-keys: the root keys.

Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.

4. Stage the workspace and secrets on the new host

Everything in this section is outside git. Do it as root over SSH, in this order.

4a. No tailscale

The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the public IP, the CI front doors are public via traefik, and the opencode UI is public on the same 443 — traefik routes oc.ci.commoninternet.net to an nginx vhost on the docker bridge that enforces basic auth, via traefik labels on a tiny swarm relay stack (ccci-opencode-ui, on the weekly sweep's keep-list) that opencode-ui-route.service deploys. (Not via traefik's file directory provider: enabling that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert and the security middleware, and every front door goes down — it did, for two minutes, on 2026-09-07.) fail2ban guards sshd and that login (nix/hosts/cc-ci/configuration.nix; the nginx jail bans in the DOCKER-USER chain because the traffic is docker-forwarded, not host INPUT).

4b. The CI server's checkout and its one out-of-band secret

# the bot deploy key for the private cc-ci-secrets submodule — generated ON this host by loops
# (§4c) and registered on the autonomic-bot Gitea account; root only points at it:
install -d -m700 /root/.ssh
cat > /root/.ssh/config <<'SSHCFG'
Host git.autonomic.zone
    Port 2222
    User git
    IdentityFile /secrets/files/autonomic-bot-cc-ci-ed25519
    IdentitiesOnly yes
SSHCFG
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
# the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
# the master (recovery) key, temporarily. Once the host is a recipient (below): its own
# ssh-host-key-derived identity, and the master key leaves the box.
install -d -m700 /var/lib/sops-nix
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt

/etc/cc-ci/secrets/secrets.yaml is encrypted to the master key and the previous hosts' SSH host keys. The master key is enough for the FIRST deploy. Then make the new host a recipient so the master key can leave the box again (it belongs off-box, with the operator): ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub → add it to .sops.yaml in cc-ci-secrets AND in cc-ci, sops updatekeys secrets.yaml (needs the master key, so do it where that lives), push, bump the submodule in cc-ci, git -C /etc/cc-ci pull --recurse-submodules on the host, then replace /var/lib/sops-nix/key.txt with the host-derived identity (ssh-to-age -private-key -i /etc/ssh/ssh_host_ed25519_key), rebuild, and delete the master key from the host.

4c. The orchestrator's workspace (as the loops user — it exists after the first rebuild, so

run §5 once first if this is a fresh host, then come back)

sudo -iu loops
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
cd /srv/cc-ci-orch
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
mkdir -p .cc-ci-logs .sops

Secrets live in /secrets/files/ (owned by loops, mode 700 on the directory), the same convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one ls /secrets/files is the complete inventory. Only cc-ci's secrets go there — nothing from other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …): secrets do not get spread around. /secrets/README.txt lists each file and its purpose.

runtime path → /secrets/files/…	what	source
/srv/cc-ci/.testenv → cc-ci.testenv	GITEA_PASSWORD (autonomic-bot: PR/API calls), DOCKERHUB_USERNAME/TOKEN (harness image pulls). Nothing else: no tailscale key, no third-party model API keys (opencode's own auth is opencode-auth.json; add TINFOIL_API_KEY back only if LOOP_MODEL moves to a tinfoil model).	old host's copy minus the tailscale line; fresh: create each credential
/srv/cc-ci/upgrader.env (not a secret, lives in the checkout)	LOOP_TIER, LOOP_MODEL, REPORT_MODEL, SUPERVISOR_MODEL for the weekly run. This host is on the OpenCode Go subscription (LOOP_TIER=go, opencode-go/… models); the ZEN tier is pay-as-you-go credit and its key is not on this host	old host, or copy the example in AGENTS.md
~loops/.ssh/cc-ci-local-ed25519 (+.pub) → same names	ssh cc-ci as root — to loopback on this host. Generated on the host, its pub in nix/hosts/cc-ci/ssh-keys	ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci as loops
~loops/.ssh/autonomic-bot-cc-ci-ed25519 (+.pub) → same names	pushes recipe branches / PRs as autonomic-bot; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. Generated on the host and registered on the bot's Gitea account (POST /api/v1/user/keys with the bot password)	ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host as loops
~loops/.local/share/opencode/auth.json → opencode-auth.json; ~loops/.config/opencode/opencode.jsonc (config, not secret — it must contain NO inline apiKey)	opencode provider auth + config. Exactly ONE provider, opencode-go — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host	old host; fresh: opencode auth login as loops
/etc/nginx/oc-htpasswd (root:nginx; the bcrypt line only — the plaintext stays with the operator, not on the host)	basic auth for the opencode UI (https://oc.ci.commoninternet.net, via traefik); nginx refuses to start without it, and its config check runs as the nginx user, so root:nginx 0640 (the nginx group exists after the first rebuild — fix ownership then and systemctl restart nginx)	old host (/secrets/files/oc-basic-auth.txt has the plaintext), or a new oc:<bcrypt> line via mkpasswd -m bcrypt

~loops/.ssh/config is written by the activation script on first rebuild (Host cc-ci → 127.0.0.1, git.autonomic.zone); it is not overwritten if present.

5. Build and activate

From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):

# as root, detached (the activation restarts sshd; a dropped session must not kill it).
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
git config --global --add safe.directory '*'
systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)

test first, always: it activates WITHOUT touching the bootloader, so if the activation breaks networking or sshd a reboot from the Hetzner console lands on the last known-good generation. (Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation restarted tailscale under the session; this one is plain public ssh, but detached is still the safe habit.) Later rebuilds are simply sudo nixos-rebuild test|switch --flake .#cc-ci from the checkout.

The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile oneshots (swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard → deploy-reports, deploy-backupbot, warm-keycloak). Verify:

systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
systemctl list-timers 'cc-ci-*' nightly-sweep
sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
# /var/lib/acme is restored or a cert has been issued):
curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53

The opencode UI: curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/ → 401 without credentials, 200 with -u oc:<password>; fail2ban-client status nginx-http-auth counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to /var/log/nginx/error.log).

When it is healthy: sudo nixos-rebuild switch --flake .#cc-ci (same config, now also the boot default). If you are migrating from another host, do §6 before letting it serve anything: right after the first activation scale the !testme bridge to 0 and mask the two orchestrator timers so the new box does not process PR comments or start a second weekly run while the old host is live:

docker service scale ccci-bridge_app=0
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer

6. Migrating: restore state from the previous host

Over ssh between the hosts (ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner', or rsync), with the matching service stopped on the new host while its directory is copied:

path	holds	notes
/var/lib/cc-ci-reports	the published weekly report pages (report.ci…)
/var/lib/cc-ci-runs	per-run artifacts the dashboard shows
/var/lib/ci-warm	warm-canonical state + alerts	recipe warm volumes are caches: not copied, rebuilt by the Sunday sweep / first use
/var/lib/acme	the Let's Encrypt cert + account and acme-dns-accounts.json — the account the permanent _acme-challenge CNAME points at	without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in acme-dns.nix)
/var/lib/acme-dns	the acme-dns zone DB
/var/lib/ci-certs	the copy traefik is handed	then systemctl restart cc-ci-acme-traefik-handoff
/root/.abra	abra's per-app env files for the deployed stacks
Drone data volume /var/lib/docker/volumes/drone_ci_commoninternet_net_data	Drone's DB: the Gitea OAuth grant, repo activation, build history	docker service scale drone_ci_commoninternet_net_app=0 on the new host, copy, scale back to 1. Otherwise run scripts/bootstrap-drone-oauth.sh (cc-ci repo) with the bot password and re-activate repos
/srv/cc-ci-orch/.cc-ci-logs, /srv/cc-ci-orch/cc-ci-plan/upstream/, REBOOTS.md, JOURNAL.md	orchestrator history, the upgrader's per-recipe release-note registry	as loops; do the final sync after stopping the orchestrator on the old host

7. Cutover and verification

1. DNS (operator, Gandi zone commoninternet.net): A records ci, *.ci and ns-acme → the new IPv4. acme NS ns-acme and _acme-challenge.ci CNAME <account>.acme… stay as they are. Wait for propagation (dig +short ci.commoninternet.net) — the records carry a 3 h TTL, and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that long. The new host must not list such a resolver: with a mixed set, every other lookup of its own names went to the old server on 2026-09-07 (runner/bridge 404s). networking.nix drops the Hetzner resolvers and configuration.nix pins the host's own names.
2. Check the new host answers on the new IP before DNS moves: dig @<new-ip> ns-acme.commoninternet.net (acme-dns), curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/ (dashboard, valid cert), same for report.ci and drone.ci.
3. Old host: docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0; systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer on the old orchestrator. New host: docker service scale ccci-bridge_app=1; systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start both.
4. End to end: post !testme on an open recipe PR and watch it turn green on the new Drone; open https://ci.commoninternet.net and https://report.ci.commoninternet.net.
5. The orchestrator: as loops on the new host cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator (or just systemctl restart cc-ci-orchestrator), attach with claude --resume or from claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the old servers were on the tailnet; the new one is not).

8. Day 2

• The host updates itself weekly: cc-ci-auto-update.timer (Tuesday 03:00 UTC, nix/modules/auto-update.nix) runs nix flake update on all inputs, builds, activates with test, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…), and only then commits the generation + bootloader and pushes the health-checked flake.lock to main; on any failure it re-activates the previous generation and restores the lock. It skips itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome in /srv/cc-ci/.cc-ci-logs/auto-update-state (one line) and journalctl -u cc-ci-auto-update; /cc-ci-status §5 reads it. /cc-ci-orchestrator-update runs the same unit by hand. The cc-ci input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
• Update only cc-ci's code (harness/tests/modules): merge in the cc-ci repo, then nix flake update cc-ci here and rebuild; also git -C /etc/cc-ci pull --recurse-submodules so the deployed checkout the sweep runs from matches. Run the lock update on this host: a lock written by a newer Nix elsewhere once failed here with NAR hash mismatch in input git+https://…cc-ci.git…&shallow=1; re-running nix flake update cc-ci on the host fetched the input properly and the rebuild went through.
• Something is down: systemctl --failed, journalctl -u deploy-<x>, docker service ps <svc>; the cc-ci repo's docs/runbook.md. Host unreachable: Hetzner console → reboot lands on the last switched generation; rescue mode + nixos-enter for anything worse (skill /hetzner-server-recovery).

---

Operating the orchestrator

The cc-ci orchestrator on this host is an opencode agent (cc-ci-plan/agents.toml: backend opencode, model opencode/glm-5.2), launched on boot by cc-ci-orchestrator.service via cc-ci-plan/agents.py up orchestrator into tmux session cc-ci-orchestrator, attached to the shared opencode web server. Steer it from the web UI, https://oc.ci.commoninternet.net (basic auth, user oc), where the weekly upgrader's and report's sessions show up too. There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and operate the CI from there over ssh (Host cc-ci in that box's loops ssh config → this host).

# on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator     # the agent's terminal
systemctl restart cc-ci-orchestrator                 # relaunch it (fresh opencode session)
sudo -iu loops tmux ls                               # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run

Weekly upgrade run

cc-ci-upgrade-all.timer (Thu 22:00 America/New_York) starts launch-upgrader.py, which runs /upgrade-all as an opencode agent (upgrader.env: LOOP_MODEL/REPORT_MODEL) in tmux cc-ci-upgrader with a usage-limit watchdog; the hourly cc-ci-upgrade-supervisor.timer drives a stalled run to completion. Start one by hand with systemctl start cc-ci-upgrade-all; follow /srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log; it ends with UPGRADE RUN COMPLETE, a summary in .cc-ci-logs/upgrades/upgrade-all-<date>.md, and a report published to https://report.ci.commoninternet.net/week-<date>.html.

Kick off / supervise the loops

cd /srv/cc-ci/cc-ci-plan
./launch.sh start                       # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
./launch.sh status                      # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop

Full supervision guide, credential map and history are in cc-ci-plan/kickoff.md, cc-ci-plan/plan.md §1.5 and cc-ci-plan/JOURNAL.md.