cc-ci-orchestrator/README.md

# cc-ci-orchestrator

The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.

| | where |
|---|---|
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |

Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
commit them.

---

# Deploying a cc-ci host from scratch

This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
can repeat it. Read it once before starting; the order matters.

## 0. What you need in hand

- A **Hetzner Cloud** project you can create servers in (console login or an API token).
- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
  login is disabled and fail2ban watches sshd, so a key is the only way in.
- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
  read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
  `autonomic-bot-gitea-ed25519`, has it).
- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
  you are starting fresh you create them (each row says how).
- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.

## 1. Provision the server on Hetzner (Debian image)

In the Hetzner Cloud console (or with `hcloud server create`):

| setting | value | why |
|---|---|---|
| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
| Name | `cc-ci` | becomes the hostname |
| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |

Check you can log in: `ssh root@<ip> hostname`.

## 2. Convert Debian → NixOS with nixos-infect

[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
reboots. Run it detached so the SSH session dropping does not kill it:

```bash
ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
#!/usr/bin/env bash
set -euo pipefail
# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
export NIXOS_IMPORT=""             # the real config comes from the flake in §5
# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
export NO_SWAP=true
curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
EOF
chmod +x /root/infect.sh
nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
```

It downloads Nix, builds a NixOS system (5–10 min; follow with
`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.

### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

All three bit on the first attempt; the script above and §3 already include the fixes, this is
so you recognise them if they come back in another form.

1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
   The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
   Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
   (`dbus`, `systemd-logind`, `sshd`, networking …) with
   `Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
   nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
   to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
   lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
   NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
   (`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
   move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
   `etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
   `root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
   `disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
3. **Boots, units fine, but no network.** The generated `networking.nix` has
   `defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
   route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
   `nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
   mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
   so turn it off for that one build:
   ```bash
   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
     ln -sfn /nix/var/nix/profiles/system /run/current-system
     nixos-rebuild boot --option sandbox false'
   ```
   The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
   from rescue mode) is what told these apart.

> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
> wraps these (token in `/srv/cc-ci/.hcloud-token`).

## 3. Capture the machine-specific config into this repo

nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
first two matter; the flake replaces `configuration.nix`.

```bash
scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix
```

Then in `nix/hosts/cc-ci/`:

- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
  `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
  the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
  (as on the 2026-09 box) can stay.
- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
  is the release you installed (never change it later).
- `ssh-keys`: the root keys.

Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.

## 4. Stage the workspace and secrets on the new host

Everything in this section is **outside git**. Do it as root over SSH, in this order.

### 4a. No tailscale

The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
443 — traefik routes `oc.ci.commoninternet.net` to an nginx vhost on the docker bridge that
enforces basic auth, via traefik labels on a tiny swarm relay stack (`opencode-ui`) that
`opencode-ui-route.service` deploys. (Not via traefik's file *directory* provider: enabling
that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert
and the `security` middleware, and every front door goes down — it did, for two minutes, on
2026-09-07.) fail2ban guards sshd and that login
(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
traffic is docker-forwarded, not host INPUT).

### 4b. The CI server's checkout and its one out-of-band secret

```bash
# root's deploy key for the private cc-ci-secrets submodule
install -d -m700 /root/.ssh
install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
cat > /root/.ssh/config <<'EOF'
Host git.autonomic.zone
    Port 2222
    User git
    IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
    IdentitiesOnly yes
EOF
# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
# the master (recovery) age key — the only sops recipient a fresh host can be
install -d -m700 /var/lib/sops-nix
install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
```

`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.

### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
run §5 once first if this is a fresh host, then come back)

```bash
sudo -iu loops
git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
cd /srv/cc-ci-orch
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
mkdir -p .cc-ci-logs .sops
```

Then the files below (`install -m600 -o loops -g users`):

| file | what | source |
|---|---|---|
| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |

`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.

## 5. Build and activate

From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):

```bash
# as root, detached (the activation restarts sshd; a dropped session must not kill it).
# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
# system has none — hence nix-shell), HOME=/root (so root's `git config --global
# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
git config --global --add safe.directory '*'
systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)
```

`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
restarted tailscale under the session; this one is plain public ssh, but detached is still the
safe habit.)
Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.

The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:

```bash
systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
fail2ban-client status sshd; fail2ban-client status nginx-http-auth
systemctl list-timers 'cc-ci-*' nightly-sweep
sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
# /var/lib/acme is restored or a cert has been issued):
curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
```

The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to
/var/log/nginx/error.log).

When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
default). **If you are migrating from another host, do §6 before letting it serve anything**: right
after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
the new box does not process PR comments or start a second weekly run while the old host is live:

```bash
docker service scale ccci-bridge_app=0
systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
```

## 6. Migrating: restore state from the previous host

Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
stopped on the new host while its directory is copied:

| path | holds | notes |
|---|---|---|
| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
| `/var/lib/acme-dns` | the acme-dns zone DB | |
| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
| `/root/.abra` | abra's per-app env files for the deployed stacks | |
| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |

## 7. Cutover and verification

1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
   the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
   are. Wait for propagation (`dig +short ci.commoninternet.net`) — the records carry a 3 h TTL,
   and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that
   long. The new host must not list such a resolver: with a mixed set, every other lookup of its
   own names went to the old server on 2026-09-07 (runner/bridge 404s). `networking.nix` drops
   the Hetzner resolvers and `configuration.nix` pins the host's own names.
2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
   (acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
   (dashboard, valid cert), same for `report.ci` and `drone.ci`.
3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
   `systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
   orchestrator. New host: `docker service scale ccci-bridge_app=1`;
   `systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
   open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
   (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
   claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
   old servers were on the tailnet; the new one is not).

## 8. Day 2

- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
  `nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
  this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
  `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
  so the deployed checkout the sweep runs from matches.
- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
  the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
  `switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
  `/hetzner-server-recovery`).

---

# Operating the orchestrator session

The steering session is a long-lived interactive Claude Code session under tmux with
`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
(backend + model in `cc-ci-plan/agents.toml`).

```bash
# attach on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator
# or resume the conversation by hand in a fresh tmux pane
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# already inside a live session and just want the web surface? /remote-control
```

`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
only the web display label. Don't pass `--fork-session` unless you mean to branch.

## Kick off / supervise the loops

```bash
cd /srv/cc-ci/cc-ci-plan
./launch.sh start                       # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
./launch.sh status                      # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop
```

Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.