orchestrator-host: symlink the opencode installer's binary into ~/.local/bin; README: nginx file perms, first-rebuild wrapper, pre-cutover checks

Found on the first activation of #cc-ci on 195.201.88.249: the opencode installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and nginx's pre-start config check runs as the nginx user, so the staged cert/htpasswd must be root:nginx 0640, not root:root 0600. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
nix/hosts/cc-ci: real hardware + networking from the box; document the infect fixes
2026-09-07 20:24:47 +00:00 · 2026-09-07 20:15:16 +00:00 · 2026-09-07 19:58:33 +00:00 · 2026-08-31 20:47:10 +00:00 · 2026-08-31 20:46:55 +00:00 · 2026-08-31 20:31:59 +00:00
52 changed files with 1333 additions and 239 deletions
@@ -18,7 +18,7 @@ RECIPE="${1:?usage: testme-on-pr.sh <recipe> <pr-index>}"
 PRIDX="${2:?usage: testme-on-pr.sh <recipe> <pr-index>}"
 TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
 set -a; . "$TESTENV"; set +a
-: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
+: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"
 NS="${GITEA_NAMESPACE:-recipe-maintainers}"
 API="https://${GITEA_URL}/api/v1"; AUTH=(-u "${GITEA_USERNAME}:${GITEA_PASSWORD}")
 POST="${POST:-1}"; MAX_WAIT="${MAX_WAIT:-480}"; INTERVAL="${INTERVAL:-30}"
@@ -13,8 +13,8 @@ a health gate, not as silent drift.

 > **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
 > the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
-> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
-> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
+> tailnet `cc-ci`, public `195.201.88.249` — the SAME host as the cc-ci CI server since 2026-09), flake checkout **`/srv/cc-ci-orch`** (repo
+> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci`** (which now also rebuilds the CI server half, from the cc-ci repo flake input). The **cc-ci
 > CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
 > machine — that's `/cc-ci-server-update`, NOT this skill.

@@ -72,7 +72,7 @@ deploy).
 ### 3. Build (catch errors before any activation)

 ```
-cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
+cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci 2>&1 | tail -15
 readlink -f result
 ```
 Build failure → fix on the branch (option renames etc.) before going further. Never activate a
@@ -85,7 +85,7 @@ activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route o
 reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
 `hetzner-server-recovery`) — lands back on the last-known-good generation.
 ```
-cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
+cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci \
  > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
 # after it settles (poll; tailscaled/sshd may blip):
 readlink /run/current-system            # should be the new store path
@@ -100,7 +100,7 @@ switch.
 ### 5. Switch (make permanent — only after 4 is healthy)

 ```
-cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
+cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci 2>&1 | tail -10
 ```
 (If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
 detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
@@ -127,7 +127,7 @@ git add flake.lock                                # flake.nix too if the channel
 git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))

 nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
-Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
+Deployed to the cc-ci host (.#cc-ci): build + test + switch + health gate green."
 git push -u origin HEAD
 ```
 Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
@@ -30,15 +30,18 @@ the orchestrator watches from outside.

 Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
 to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
-watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
-operator reconnects to it (that's why the startup notification matters). The orchestrator now runs on
-a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.30`, public
-`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
-`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
-`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
-`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
+watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
+`cc-ci-orchestrator.service` (`agents.py up orchestrator`) — the operator reconnects to it (that's
+why the startup notification matters). Since 2026-09 the orchestrator runs on the **same Hetzner
+host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
+`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
+repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
+The full provisioning + deploy guide is `README.md`; the move is recorded in
+`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
+shared with notplants) are in `archive/`. Rebuild this host with
+`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
 `nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
-before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
+before the `switch`** (general policy for nix deploys to this host: `test`
 leaves the bootloader and system profile untouched, so a reboot always recovers to the
 last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
 `.cc-ci-logs/server-update-2026-08-03.md`).
@@ -1,47 +1,344 @@
 # cc-ci-orchestrator

-Orchestrator workspace for building the **cc-ci** Co-op Cloud recipe CI server. The plan, launch
-tooling, and loop prompts live in [`cc-ci-plan/`](cc-ci-plan/); see [`AGENTS.md`](AGENTS.md) for the
-roles and operating model. Secrets (`.testenv`) are gitignored — never commit them.
+The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
+the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
+host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
+one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
+module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.

-## Run the orchestrator in tmux (survives disconnects + closing your laptop)
+| | where |
+|---|---|
+| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
+| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
+| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
+| The machine: hardware, networking, tailscale, root keys | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
+| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
+| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
+| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |

-Keep this supervising session alive on the host with tmux, and use `--remote-control` so you can
-watch/steer it from **claude.ai/code** (or the mobile app).
+Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
+commit them.
+
+---
+
+# Deploying a cc-ci host from scratch
+
+This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
+server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
+can repeat it. Read it once before starting; the order matters.
+
+## 0. What you need in hand
+
+- A **Hetzner Cloud** project you can create servers in (console login or an API token).
+- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
+  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line).
+- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
+  read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` deploy key,
+  `autonomic-bot-gitea-ed25519`, has it).
+- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
+  you are starting fresh you create them (each row says how).
+- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
+
+## 1. Provision the server on Hetzner (Debian image)
+
+In the Hetzner Cloud console (or with `hcloud server create`):
+
+| setting | value | why |
+|---|---|---|
+| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
+| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
+| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
+| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
+| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
+| Name | `cc-ci` | becomes the hostname |
+| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
+
+Check you can log in: `ssh root@<ip> hostname`.
+
+## 2. Convert Debian → NixOS with nixos-infect
+
+[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
+reboots. Run it detached so the SSH session dropping does not kill it:

 ```bash
-# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
-#      /exit        (inside Claude)   or Ctrl-D
-
-# 1. Start a detachable tmux session on this host
-tmux new -s orchestrator
-
-# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
-claude --resume autonomous-orchestrator \
-  --remote-control "autonomous-orchestrator" \
-  --dangerously-skip-permissions
-#    - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
-#    - Or resume by the stable session id (more deterministic in a fresh pane):
-#        claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
-#          --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
-
-# 3. Detach — the process keeps running:  press Ctrl-b, then d
+ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
+#!/usr/bin/env bash
+set -euo pipefail
+# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
+INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
+export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
+export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
+export NIXOS_IMPORT=""             # the real config comes from the flake in §5
+# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
+# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
+export NO_SWAP=true
+curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
+EOF
+chmod +x /root/infect.sh
+nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
 ```

-**Reconnect later**
- On this host: `tmux attach -t orchestrator`
- From anywhere: **claude.ai/code** → the `autonomous-orchestrator` session
+It downloads Nix, builds a NixOS system (5–10 min; follow with
+`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
+`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.

-**Why it survives:** tmux keeps the `claude` process alive across SSH disconnects and your laptop
-closing; remote-control runs *outbound* from this host to Anthropic, so it stays connected
-regardless of the viewer. After a host reboot, re-run steps 1–2.
+### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

-> Two different "names": `--resume <name|id>` selects the *conversation* to restore (shown in the
-> `/resume` picker); the `--remote-control "<name>"` value is only the web display label and resumes
-> nothing. Resuming reuses the same session id each time (stays `34a8…`) — don't pass
-> `--fork-session` unless you intend to branch a new conversation.
->
-> Already inside a live session and just want the web surface? Run `/remote-control` — no exit/resume.
+All three bit on the first attempt; the script above and §3 already include the fixes, this is
+so you recognise them if they come back in another form.
+
+1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
+   The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
+   Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
+2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
+   (`dbus`, `systemd-logind`, `sshd`, networking …) with
+   `Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
+   nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
+   to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
+   lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
+   NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
+   (`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
+   move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
+   `etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
+   `root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
+   `disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
+3. **Boots, units fine, but no network.** The generated `networking.nix` has
+   `defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
+   route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
+   `nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
+   mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
+   so turn it off for that one build:
+   ```bash
+   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
+   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
+   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
+     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
+     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
+     ln -sfn /nix/var/nix/profiles/system /run/current-system
+     nixos-rebuild boot --option sandbox false'
+   ```
+   The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
+   from rescue mode) is what told these apart.
+
+> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
+> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
+> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
+> wraps these (token in `/srv/cc-ci/.hcloud-token`).
+
+## 3. Capture the machine-specific config into this repo
+
+nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
+first two matter; the flake replaces `configuration.nix`.
+
+```bash
+scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
+scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix
+```
+
+Then in `nix/hosts/cc-ci/`:
+
+- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
+  `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
+- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
+  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
+  the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
+  (as on the 2026-09 box) can stay.
+- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
+  is the release you installed (never change it later).
+- `ssh-keys`: the root keys.
+
+Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
+
+## 4. Stage the workspace and secrets on the new host
+
+Everything in this section is **outside git**. Do it as root over SSH, in this order.
+
+### 4a. Tailscale
+
+```bash
+# a reusable (or fresh) tailnet auth key from the tailscale admin console
+install -m600 /dev/stdin /etc/ts-auth-key <<<'tskey-auth-…'
+```
+
+### 4b. The CI server's checkout and its one out-of-band secret
+
+```bash
+# root's deploy key for the private cc-ci-secrets submodule
+install -d -m700 /root/.ssh
+install -m600 <autonomic-bot-gitea-ed25519> /root/.ssh/autonomic-bot-gitea-ed25519
+cat > /root/.ssh/config <<'EOF'
+Host git.autonomic.zone
+    Port 2222
+    User git
+    IdentityFile /root/.ssh/autonomic-bot-gitea-ed25519
+    IdentitiesOnly yes
+EOF
+# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
+git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
+# the master (recovery) age key — the only sops recipient a fresh host can be
+install -d -m700 /var/lib/sops-nix
+install -m600 <master-age.txt> /var/lib/sops-nix/key.txt
+```
+
+`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *old* host's SSH host
+key. That is enough to deploy. Afterwards (optional, tidier) add the new host as a recipient:
+`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub`, add it to `secrets/.sops.yaml` in cc-ci-secrets,
+`sops updatekeys secrets.yaml`, push, `git -C /etc/cc-ci submodule update --remote`.
+
+### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
+run §5 once first if this is a fresh host, then come back)
+
+```bash
+sudo -iu loops
+git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
+sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
+cd /srv/cc-ci-orch
+git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
+git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
+mkdir -p .cc-ci-logs .sops
+```
+
+Then the files below (`install -m600 -o loops -g users`):
+
+| file | what | source |
+|---|---|---|
+| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
+| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
+| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
+| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
+| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
+| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
+| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
+| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
+| `/etc/nginx/oc-selfsigned.{crt,key}`, `/etc/nginx/oc-htpasswd` | the tailnet-only opencode UI; **nginx refuses to start without them**, and its config check runs as the `nginx` user, so: `root:nginx`, crt `0644`, key + htpasswd `0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host, or generate (commands in `nix/modules/orchestrator-host.nix`) |
+
+`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
+`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
+
+## 5. Build and activate
+
+From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
+
+```bash
+# as root, detached (the activation restarts sshd/tailscale; a dropped session must not kill it).
+# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
+# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
+# system has none — hence nix-shell), HOME=/root (so root's `git config --global
+# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
+# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
+git config --global --add safe.directory '*'
+systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
+  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
+journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)
+```
+
+`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
+networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
+Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
+
+The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
+and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
+oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
+deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
+
+```bash
+systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
+tailscale status | head -3
+docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
+systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns
+systemctl list-timers 'cc-ci-*' nightly-sweep
+sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
+# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
+# /var/lib/acme is restored or a cert has been issued):
+curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
+curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
+curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
+dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
+```
+
+Seen on 2026-09-07: `tailscaled-autoconnect` failed with `invalid key: API key does not exist` —
+the reusable auth key had been revoked. Generate a fresh one in the tailscale admin console, put
+it in `/etc/ts-auth-key`, `systemctl restart tailscaled-autoconnect`. Nothing else depends on it
+during the install; the box is reachable on its public IP throughout.
+
+When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
+default). **If you are migrating from another host, do §6 before letting it serve anything**: right
+after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
+the new box does not process PR comments or start a second weekly run while the old host is live:
+
+```bash
+docker service scale ccci-bridge_app=0
+systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
+```
+
+## 6. Migrating: restore state from the previous host
+
+Over tailscale (`rsync -aHAX --numeric-ids root@<old>:<path> <path>`), with the matching service
+stopped on the new host while its directory is copied:
+
+| path | holds | notes |
+|---|---|---|
+| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
+| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
+| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
+| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
+| `/var/lib/acme-dns` | the acme-dns zone DB | |
+| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
+| `/root/.abra` | abra's per-app env files for the deployed stacks | |
+| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
+| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
+
+## 7. Cutover and verification
+
+1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
+   the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
+   are. Wait for propagation (`dig +short ci.commoninternet.net`).
+2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
+   (acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
+   (dashboard, valid cert), same for `report.ci` and `drone.ci`.
+3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
+   `systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
+   orchestrator. New host: `docker service scale ccci-bridge_app=1`;
+   `systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
+4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
+   open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
+5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
+   (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
+   claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
+6. Keep the old host as a cold standby for a week, then delete it and its tailnet node.
+
+## 8. Day 2
+
+- **Update the host** (nixpkgs bump for both halves): `/cc-ci-orchestrator-update`, which is
+  `nix flake update` → `nixos-rebuild test` → verify → `switch` → PR. The `cc-ci` input follows
+  this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
+- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
+  `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
+  so the deployed checkout the sweep runs from matches.
+- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
+  the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
+  `switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
+  `/hetzner-server-recovery`).
+
+---
+
+# Operating the orchestrator session
+
+The steering session is a long-lived interactive Claude Code session under tmux with
+`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
+`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
+(backend + model in `cc-ci-plan/agents.toml`).
+
+```bash
+# attach on the host
+sudo -iu loops tmux attach -t cc-ci-orchestrator
+# or resume the conversation by hand in a fresh tmux pane
+claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
+# already inside a live session and just want the web surface? /remote-control
+```
+
+`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
+only the web display label. Don't pass `--fork-session` unless you mean to branch.

 ## Kick off / supervise the loops

@@ -53,5 +350,5 @@ cd /srv/cc-ci/cc-ci-plan
 ./launch.sh stop
 ```

-Full supervision guide, credential map, and the Incus VM fallback are in
-[`cc-ci-plan/kickoff.md`](cc-ci-plan/kickoff.md) and [`cc-ci-plan/plan.md`](cc-ci-plan/plan.md) §1.5.
+Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
+`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.
@@ -0,0 +1,22 @@
+# archive/ — how cc-ci and its orchestrator were built and moved, before the combined host
+
+Historical record only. Nothing in here is deployed or evaluated. It was moved out of the live
+tree on 2026-09-07 when the CI server and the orchestrator were consolidated onto one Hetzner
+host (`nixosConfigurations.cc-ci` in `../flake.nix`; deploy guide in `../README.md`; the plan
+that did it is `../cc-ci-plan/plan-cc-ci-combined-host.md`).
+
+| path | what it was |
+|---|---|
+| `nix/configuration-incus-vm.nix` | Channel-based NixOS config of the first orchestrator VM on b1 (Incus, 2 GB). Ran the loops as root; hard-coded the dead Incus cc-ci IP. Replaced by the Hetzner host 2026-05-31. |
+| `nix/README.md` | The README for that Incus VM config. |
+| `nix/cc-ci-orchestrator-hetzner/` | The orchestrator's own Hetzner `cpx22` host (`168.119.126.100`, tailnet `cc-ci-orchestrator-1`), 2026-05-31 → 2026-09. From 2026-08-20 the live copy of this config was `notplants-nix`'s `notplants-orchestrator` host (the box became a shared agent host for several projects); this one had drifted and still carried lichen/project-orchestrator units. Superseded by `../nix/hosts/cc-ci` + `../nix/modules/orchestrator-host.nix`. |
+| `nix/atproto-likes.nix` | A notplants (not cc-ci) service that lived on the shared box; kept by notplants-nix. |
+| `terraform/` | OpenTofu for the `cpx22` orchestrator server (Debian 12 → nixos-infect at `nixos-24.11`). The combined host was provisioned by hand instead; the README documents that path. Note its `user-data.sh` would fail on the Debian 13 image (nixos-infect's temp swapfile on a tmpfs `/tmp`) — see the README's `NO_SWAP=true` note. |
+| `plans/plan-orchestrator-migration.md` | Pi → Incus VM move of the orchestrator (2026-05). |
+| `plans/plan-orchestrator-hetzner-migration.md` | Incus VM → Hetzner `cpx22` move of the orchestrator (2026-05-31). Has the reboot-resilience design (`cc-ci-loops.service`). |
+| `plans/plan-migrate-cc-ci-to-hetzner.md`, `plans/plan-cc-ci-hetzner-migration.md`, `plans/plan-cc-ci-hetzner-terraform.md` | The CI server's own move from the `cc-nix-test` Incus VM to Hetzner `cpx32` (`91.98.47.73`, 2026-05-31), and the terraform that provisioned it (lives in the cc-ci repo). |
+| `plans/plan-repo-consolidation.md` | The earlier repo layout consolidation. |
+
+The cc-ci server's own history (machine-docs, decisions, the clean-room rebuild that proved
+"two repos + one age key + one `nixos-rebuild switch`") is in the cc-ci repo under
+`machine-docs/` and `docs/`.
@@ -14,7 +14,11 @@
  # with ACME, and opens 80/443 (previously only 22 was public). Canonical source of
  # the module is the project repo; ../atproto-likes.nix is a copy kept in this tree
  # because pure evaluation cannot import a path outside the flake.
-  imports = [ ../../atproto-likes.nix ];
+  # cc-ci.nix holds everything that exists FOR cc-ci (the loops, the cc-ci orchestrator session,
+  # the weekly recipe-upgrade run and its hourly supervisor). Split out 2026-08-20 so this host —
+  # a general agent box that also serves unrelated projects — can be reasoned about separately
+  # from cc-ci. Also exported as `nixosModules.cc-ci` from this repo's flake for other hosts.
+  imports = [ ../../atproto-likes.nix ../../modules/cc-ci.nix ];

  services.openssh = {
    enable = true;
@@ -127,7 +131,8 @@ SSHCFG

  # opencode web server — one shared instance; agent sessions attach to it for web visibility.
  # Serves the web UI at http://oc.commoninternet.net (via nginx below, tailscale-only).
-  # Provider creds are read from /srv/cc-ci/.testenv at startup.
+  # Public provider configuration is tracked in the cc-ci repo; credentials
+  # remain in /srv/cc-ci/.testenv.
  systemd.services.opencode-web = {
    description = "opencode web server for cc-ci agents";
    wantedBy = [ "multi-user.target" ];
@@ -137,7 +142,10 @@ SSHCFG
      Type = "simple";
      User = "loops"; Group = "users";
      WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
-      EnvironmentFile = "/srv/cc-ci/.testenv";
+      EnvironmentFile = [
+        "-/srv/cc-ci/cc-ci/.env.public"
+        "/srv/cc-ci/.testenv"
+      ];
      ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
      ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
      Restart = "on-failure";
@@ -225,72 +233,7 @@ SSHCFG
    };
  };

-  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
-  systemd.services.cc-ci-loops = {
-    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
-    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
-      # default (control-group) systemd kills every leftover process in the cgroup when the unit
-      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
-      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
-      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
-      KillMode = "process";
-      Type = "oneshot"; RemainAfterExit = true;
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci/cc-ci";
-      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
-      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
-    };
-    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
-    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
-    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
-    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
-      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
-      export PATH="/home/loops/.local/bin:$PATH"
-      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
-        echo "workspace not staged yet — skipping loop start"
-    '';
-  };

-  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
-  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
-  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
-  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
-  # the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
-  # attaches to the shared opencode web server (opencode-web.service below) and is reachable for
-  # Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
-  # watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
-  # supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
-  # the same reboot-resilience the other two orchestrators already have.
-  systemd.services.cc-ci-orchestrator = {
-    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
-    wantedBy = [ "multi-user.target" ];
-    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
-      # must not tear down the (shared) tmux server and every agent session with it.
-      KillMode = "process";
-      Type = "oneshot"; RemainAfterExit = true;
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci-orch";
-    };
-    environment = { HOME = "/home/loops"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      proj="/srv/cc-ci-orch"
-      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
-        >> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
-      cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
-    '';
-  };

  # p-lichen-orchestrator supervisor — the SEPARATE lichen.page testing/hardening orchestrator
  # (distinct from cc-ci-loops above). Reboot-resilience: on boot, resume the orchestrator's Remote
@@ -356,76 +299,7 @@ SSHCFG
    '';
  };

-  # Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
-  # verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
-  # phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
-  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
-  systemd.services.cc-ci-upgrade-all = {
-    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci";
-      # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
-      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults
-      # (current behavior). To run the weekly job on e.g. opencode-go/glm-5.2, drop a file with
-      #   LOOP_BACKEND=opencode
-      #   LOOP_MODEL=opencode-go/glm-5.2
-      # No rebuild needed to switch — the env file is read at each timer fire. Holds no secrets
-      # (the opencode-go API key lives in ~/.local/share/opencode/auth.json, mode 600).
-      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
-    };
-    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
-    '';
-  };

-  systemd.timers.cc-ci-upgrade-all = {
-    description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
-    wantedBy = [ "timers.target" ];
-    timerConfig = {
-      # 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
-      OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
-      Persistent = true;  # if the box was down at the scheduled time, run once on next boot
-    };
-  };

-  # Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
-  # log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
-  # disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
-  # CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
-  # (zero model tokens). Only when a run has stalled/died before completing does it launch a
-  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
-  systemd.services.cc-ci-upgrade-supervisor = {
-    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
-    after = [ "network-online.target" "tailscaled.service" ];
-    wants = [ "network-online.target" ];
-    serviceConfig = {
-      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
-      User = "loops"; Group = "users";
-      WorkingDirectory = "/srv/cc-ci";
-      # Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
-      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
-    };
-    environment = { HOME = "/home/loops"; };
-    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
-    script = ''
-      export PATH="/home/loops/.local/bin:$PATH"
-      python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
-    '';
-  };

-  systemd.timers.cc-ci-upgrade-supervisor = {
-    description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
-    wantedBy = [ "timers.target" ];
-    timerConfig = {
-      OnCalendar = "*-*-* *:07:00";  # every hour at :07 (offset from the weekly :00 fire)
-      Persistent = false;            # a missed hourly check is moot — the next hour re-checks
-    };
-  };
 }
@@ -867,6 +867,129 @@ session cc-ci-orchestrator-stale can be killed; recipe-mirrors org still private
  (/srv/cc-ci-orch/cc-ci), and task-tool subagents inherit their parent session's directory. The
  config now lives in the cc-ci repo at that path. VERIFIED end-to-end with the launcher's exact
  invocation: parent=glm-5.2, subagent=deepseek-v4-pro read back from the session DB.
-  LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
-  modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
-  real launcher) and landed in a different project.
+   LESSON: `opencode debug config` proves resolution, NOT binding — only a live subagent's recorded
+   modelID proves binding. First attempt was a false pass because the probe passed --dir (unlike the
+   real launcher) and landed in a different project.
+
+## Session 2026-08-15 19:25 UTC — opencode glm-5.2
+
+**Left off:** Recovered the stalled 2026-08-14 weekly /upgrade-all run. Killed a supervisor that had
+been relaunching hourly for ~40h (balance exhausted), then started a FRESH scoped upgrader. Run is now
+progressing (surveying the 9 remaining recipes). Watching it through to completion.
+
+**What happened (the stall):**
+- The 2026-08-14 /upgrade-all run (session ses_00200382fffeYIGl2sc3mO9JId) stalled at 03:18 Aug 14
+  mid-`lasuite-drive` with `Error: Insufficient balance` (opencode zen workspace balance ran out). It
+  had already done bluesky-pds, ghost, gitea, hedgedoc (PRs) + immich, lasuite-docs (SKIPPED up-to-date)
+  alphabetically; lasuite-drive had a plan + partial PR #6 but no RESULT/verify.
+- The supervisor cron (glm-5.2, opencode-go tier) relaunched an hourly one-shot supervisor ~40 times
+  to "drive it to completion", but each was also balance-walled (and later, just spinning). The run sat
+  INCOMPLETE + not progressing for 40h. No weekly summary, no report published for week of Aug 14.
+
+**What I did this session:**
+- Diagnosed: the opencode zen endpoint is NOW healthy (direct probe `say OK` → HTTP 200 in 1.35s —
+  balance is restored). But resuming the ORIGINAL giant session is impossible: it's 2.58M tokens
+  (267K input + 2.3M cache) and `opencode run -s … --continue` sits idle on `do_epoll_wait` with zero
+  I/O — the inference endpoint silently drops the oversized request (matches the supervisor's
+  `socket connection was closed unexpectedly` errors). A fresh small `opencode run` works fine. So the
+  giant session is unresumable; a fresh start is the only path.
+- Killed the stuck supervisor (tmux `cc-ci-supervisor`, proc 377329).
+- `UPGRADER_ARGS="lasuite-drive lasuite-meet mailu matrix-synapse mattermost-lts mumble n8n plausible
+  wordpress --sequential" python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py fresh` — this killed the
+  stuck resume, archived the old giant session (`archive-cc-ci-upgrader — 2026-08-14`), reclaimed 10GB
+  stale images on cc-ci (disk 29%), and started a FRESH small session
+  `ses_ff920cf39ffeoogwXHTajp94cr` (zen/glm-5.2) scoped to the 9 recipes not yet done this week
+  (positions 13-21 alphabetically; positions 1-12 were already surveyed — 6 PRs + 6 up-to-date). A
+  fresh watchdog is watching the new session. The skill is idempotent (reuses existing PRs incl.
+  lasuite-drive #6, never duplicates), so scoping is safe.
+- Confirmed the fresh run is progressing: pane shows it surveying the 9 recipes (verified all present
+  in abra + all `weekly` tier; currently probing plausible/wordpress tags). Proc alive, log growing.
+
+**Phase / loop state:**
+- Build/adversary loops: STOPPED (whole sequence completed 2026-08-01; phase ghost DONE).
+- Weekly upgrader: RUNNING (fresh session ses_ff920cf39, scoped 9 recipes, --sequential, watchdog up).
+- cc-ci server: healthy (disk 29%, runner active).
+
+**Open items for next session:**
+- **Monitor the fresh upgrader to completion.** It will survey the 9 recipes, /recipe-upgrade the
+  upgradeable ones (subagents, !testme verify, open/extend PRs — NEVER merge), write the weekly summary
+  to `/srv/cc-ci/.cc-ci-logs/upgrades/`, then `launch-report.py fresh` (the upgrade-all skill does this
+  itself per SKILL.md §5), print `UPGRADE RUN COMPLETE`, and go idle. If it stalls on a usage limit,
+  the watchdog auto-resumes the SAME (small) session — that works now.
+- **Do NOT try to resume the archived giant session ses_00200382** — it's unresumable (endpoint drops
+  the 2.58M-token request). It's archived; leave it.
+- After the run completes + report publishes, operator review queue = this week's recipe PRs.
+- The supervisor cron (hourly at XX:07) should now leave the run alone once it's progressing; if a
+  supervisor fires while the run is mid-flight, its guardrails say to hand back to the resumed run, not
+  double-write. No action needed unless it interferes.
+
+**Notes:**
+- Root cause of the 40h silence was the same BUG 1 from 2026-08-10 (supervisor progress gate) partly:
+  the supervisor kept firing because the run never reached "progressing". Now that balance is restored
+  and a fresh small session is running, the gate should see progress and stand down.
+- Lesson: when a weekly run dies mid-flight on a giant context, do NOT resume the original session —
+  start fresh and scope to the remaining recipes. The /upgrade-all skill is idempotent so this is safe.
+
+## Session 2026-08-31 17:17 UTC — restricted acme-dns renewal staged
+
+**What happened:** Implemented and deployed the cc-ci restricted ACME DNS-01 design in cc-ci PRs
+#24–#27 (all merged). The initial `sqlite3` backend setting was detected during staging bootstrap
+and corrected to acme-dns 2.x `sqlite` before any account registration succeeded.
+
+**Live state:**
+- `acme-dns` is authoritative for `acme.commoninternet.net` on `91.98.47.73`, with public UDP/TCP
+  53 verified directly; it serves the apex NS and `ns-acme.commoninternet.net A 91.98.47.73`.
+- Its HTTP update API is bound solely to `127.0.0.1:8080`; registration is disabled (local POST
+  returns 403). The one restricted Lego account is stored mode `0600` under `/var/lib/acme/`, outside
+  Git and the Nix store. No Gandi credentials were used or stored.
+- Nix ACME is in Let's Encrypt staging mode for `ci.commoninternet.net` and its wildcard; its renewal
+  timer is enabled. Traefik handoff remains production-gated so staging cannot replace the live cert.
+- Every server deployment used `nixos-rebuild test`, reachability/service verification, then
+  `nixos-rebuild switch` on `#cc-ci-hetzner`.
+
+**Manual Gandi handoff (only blocker):** Add the parent-zone A+NS delegation and permanent CNAME:
+`ns-acme.commoninternet.net. A 91.98.47.73`,
+`acme.commoninternet.net. NS ns-acme.commoninternet.net.`, and
+`_acme-challenge.ci.commoninternet.net. CNAME
+9d3f0428-eb97-4b14-bb07-da2f0f6c2073.acme.commoninternet.net.` Once propagated, resume with a
+staging issuance, then production cutover/handoff verification. Do not obtain or add a Gandi token.
+
+## Session 2026-08-31 19:05 UTC — ACME production issuance complete
+
+**Completed:** The operator added the narrow Gandi delegation and permanent CNAME. Staging then
+validated both `ci.commoninternet.net` and `*.ci.commoninternet.net`; production issuance validated
+the same SAN pair and produced a Let's Encrypt certificate valid through 2026-11-29. cc-ci PRs #28–#30
+were merged to promote production, make Traefik roll certificate-secret changes even when recipe
+upgrades are held, and remove the retired SOPS wildcard-file injection.
+
+**Live cc-ci state:**
+- `/var/lib/acme/ci.commoninternet.net` is the sole source of the production cert/key; the runtime
+  pair matches it, is root-only for the key, and Traefik now references matching versioned Swarm
+  secrets. Reboot-safe `nixos-rebuild switch` completed at cc-ci `7147d77` after the required test
+  activations and health checks.
+- acme-dns remains authoritative only for the delegated subzone, its update API is localhost-only,
+  registration is disabled, and the restricted-account JSON is mode `0600`. The Nix ACME renewal
+  timer is active. No Gandi credential was added.
+
+**External blocker, not cc-ci-owned:** Direct TLS to `91.98.47.73:443` now presents the new valid
+certificate. The operator-managed public gateway at `143.244.213.108` currently accepts TCP then
+closes TLS before presenting a certificate, so its passthrough to cc-ci needs repair/verification.
+The cc-ci plan explicitly marks the gateway as operator infrastructure; do not change the DNS
+delegation or add a Gandi token to work around it.
+
+## Session 2026-08-31 20:45 UTC — weekly DeepSeek Flash run and report complete
+
+**Completed:** Started the weekly upgrader explicitly on `opencode/deepseek-v4-flash`; its parent
+and every recipe subagent were confirmed with that persisted model. All nine eligible recipe PRs
+completed `!testme` successfully. The public report was then restarted before publication when the
+first handoff was found to use the old GLM model; the replacement report session was confirmed
+`opencode/deepseek-v4-flash` and published successfully at
+`https://report.ci.commoninternet.net/week-2026-08-31.html` (HTTP 200).
+
+**Follow-up fixes:** Orchestrator PR #17 (`1b75d98`) made `testme-on-pr.sh` apply the documented
+public Gitea-host default. cc-ci PR #31 (`769fd29`) adds that public hostname to `.env.public`.
+Both commits were scanned clean and contain no coauthor trailers. No recipe PR was merged.
+
+**Security note:** A subagent briefly enabled shell tracing while debugging the verifier, exposing
+runtime credentials in its private agent trace. No values were committed or put in this journal,
+but rotate the affected `/srv/cc-ci/.testenv` credentials as a precaution.
@@ -1114,19 +1114,24 @@ def markdown(rep: dict) -> str:


 def _gitea_auth(url: str) -> dict:
-    """Basic auth for the private mirror, from /srv/cc-ci/.testenv.
+    """Basic auth for the private mirror, using public config plus .testenv.

    Sent as a HEADER, never embedded in the URL: in-URL credentials leak into shell history, process
    lists and error messages, and urllib mis-parses a password containing a colon."""
    host = re.sub(r"^https?://", "", url).split("/")[0]
    env = {}
-    try:
-        for ln in open(os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")):
+    for path in (
+        "/srv/cc-ci/cc-ci/.env.public",
+        os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv"),
+    ):
+        try:
+            lines = open(path)
+        except OSError:
+            continue
+        for ln in lines:
            if "=" in ln and not ln.strip().startswith("#"):
                k, v = ln.strip().split("=", 1)
                env[k] = v.strip().strip("\"'")
-    except OSError:
-        return {}
    if host != env.get("GITEA_URL", "git.autonomic.zone"):
        return {}
    u, pw = env.get("GITEA_USERNAME"), env.get("GITEA_PASSWORD")
@@ -34,7 +34,7 @@ supports_resume = false
 prompt_delivery = "ping"       # send after the TUI connects
 connect_delay   = 12
 submit_key      = "C-m"
-preamble        = "set -a; . /srv/cc-ci/.testenv; set +a"
+preamble        = "set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a"
 stall_idle      = 900
 active_re = "esc interrupt|thinking|inferring|running tool|tool call|preparing patch|reading|searching"
 limit_re  = "spend limit|usage limit|limit reached|reached your .*limit|out of (credits|tokens)"
@@ -112,7 +112,7 @@ def start(mode="resume"):
        if not Path(OPENCODE_BIN).exists():
            die(f"opencode not found at {OPENCODE_BIN}")
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {WORKDIR}"
        )
        log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
@@ -154,7 +154,7 @@ def start(mode="resume"):
                f"OPENCODE_CONFIG_CONTENT={shlex.quote(json.dumps({'model': LOOP_MODEL}))} "
            )
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"{model_env}NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {shlex.quote(WORKDIR)}"
        )
        log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
@@ -109,7 +109,7 @@ def start(mode, date):
        cwd = "/srv/cc-ci-orch/cc-ci"
        share_flag = "--share" if OPENCODE_SHARE else ""
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
            f"--title '{SESSION}' --dir {cwd} \"$(cat '{kf}')\""
        )
@@ -107,7 +107,7 @@ def spawn_supervisor(sid, reason):
    kf = Path(LOG_DIR) / f".kickoff-{SUP_SESSION}.txt"
    kf.write_text(build_kickoff(sid, reason))
    share = "--share" if OPENCODE_SHARE else ""
-    cmd = (f"set -a; . {WORKDIR}/.testenv; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
+    cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run --model '{MODEL}' {share} "
           f"--attach '{OPENCODE_SERVER}' --title '{SUP_SESSION}' --dir {WORKDIR} \"$(cat '{kf}')\"")
    _sh(["tmux", "new-session", "-d", "-s", SUP_SESSION, "-c", WORKDIR, cmd])
    _sh(["tmux", "pipe-pane", "-o", "-t", SUP_SESSION, f"cat >> '{LOG_DIR}/{SUP_SESSION}.log'"])
@@ -216,7 +216,7 @@ def start(mode="use-or-create"):
        # so they must come AFTER `run` (a global `opencode --model X run` is ignored).
        share_flag = "--share" if OPENCODE_SHARE else ""
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
            f"--title '{SESSION}' \"$(cat '{kf}')\""
        )
@@ -521,7 +521,7 @@ def resume(reason="manual"):
            "write+push the weekly summary, then `python3 /srv/cc-ci/cc-ci-plan/launch-report.py fresh`, print "
            "'" + DONE_MARKER + "', and go idle.")
    share = "--share" if OPENCODE_SHARE else ""
-    cmd = (f"set -a; . /srv/cc-ci/.testenv; set +a; {OPENCODE_BIN} run -s {sid} --continue "
+    cmd = (f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; {OPENCODE_BIN} run -s {sid} --continue "
           f"--model '{MODEL}' {share} --attach '{OPENCODE_SERVER}' --dir '{WORKDIR}' \"$(cat '{kf}')\"")
    subprocess.run(["tmux", "new-session", "-d", "-s", SESSION, "-c", WORKDIR, cmd])
    subprocess.run(["tmux", "pipe-pane", "-o", "-t", SESSION, f"cat >> '{LOG_FILE}'"])
@@ -323,7 +323,7 @@ def start_agent(role, session, workdir):
        # absolute repo paths for builder/adversary work.
        session_cwd = "/srv/cc-ci-orch/cc-ci"
        cmd = (
-            f"set -a; . /srv/cc-ci/.testenv; set +a; "
+            f"set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a; "
            f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {session_cwd}"
        )
        log(f"starting {session} (backend=opencode, phase={pid}, model={model or 'default'})")
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Load cc-ci runtime configuration. Source this file; do not execute it.
+#
+# Callers enable `set -a` so both files become exported environment variables.
+# Public configuration is loaded first, then the secret file. This preserves a
+# local secret override only where an emergency migration explicitly needs it.
+
+if [ -r /srv/cc-ci/cc-ci/.env.public ]; then
+  . /srv/cc-ci/cc-ci/.env.public
+fi
+
+if [ ! -r /srv/cc-ci/.testenv ]; then
+  echo "cc-ci secret environment is missing: /srv/cc-ci/.testenv" >&2
+  return 1
+fi
+
+. /srv/cc-ci/.testenv
@@ -71,7 +71,7 @@ log "writing report $REPORT ..."
  if [ -f "$SUMMARY" ]; then cat "$SUMMARY"; else echo "(no dated summary at $SUMMARY — /upgrade-all may have stalled on the usage limit or errored; check the cc-ci-upgrader session + $LOG)"; fi
  echo
  echo "## Open PRs per recipe (post-run)"
-  set -a; . /srv/cc-ci/.testenv 2>/dev/null; set +a
+  set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
  G="https://${GITEA_USERNAME}:${GITEA_PASSWORD}@${GITEA_URL}/api/v1"
  for r in $(curl -fsS "$G/orgs/recipe-maintainers/repos?limit=100" 2>/dev/null | python3 -c "import sys,json;print('\n'.join(sorted(x['name'] for x in json.load(sys.stdin))))" 2>/dev/null); do
    case "$r" in cc-ci|cc-ci-orchestrator|cc-ci-secrets|archived-*) continue;; esac
@@ -0,0 +1,102 @@
+# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator
+
+**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
+cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
+cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
+(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
+`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
+orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
+Last step (separate, after everything works on the current names): move both to `autonomic.zone`
+subdomains.
+
+## Facts (2026-09-07)
+
+| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
+|---|---|---|---|
+| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
+| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
+| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
+| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
+| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |
+
+Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
+`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
+(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
+`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
+`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
+volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
+rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
+oneshots recreate them from sops.
+
+Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
+key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
+`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
+`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
+htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.
+
+## Design
+
+**cc-ci repo** (`feat/nixos-module-export`):
+- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
+  host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
+  `environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
+  tailscale, no root keys, no stateVersion — the host supplies those.
+- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
+  `sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
+  path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
+- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.
+
+**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
+- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
+- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
+  `nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
+  orchestrator session and the weekly/hourly timers.
+- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
+  nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
+  (on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
+- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
+  both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
+  `--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
+- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
+- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
+- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
+  `nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.
+
+**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
+four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
+notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
+
+## Steps
+
+1. [ ] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
+   `hardware-configuration.nix` + `networking.nix`.
+2. [ ] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
+3. [ ] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
+   `#cc-ci` evaluates; PR.
+4. [ ] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
+   Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
+   new host does not double-process `!testme` or run a second weekly upgrade before cutover.
+5. [ ] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
+   /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
+6. [ ] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
+   drone, one direct `cc-ci-run` on custom-html-tiny).
+7. [ ] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
+   drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
+8. [ ] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
+   state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
+9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
+10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
+
+## Log
+
+- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
+  `notplants-orchestrator-ed25519`.
+- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
+  PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
+  nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
+  Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
+  Hetzner console (no API token for that project on this host).
+- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
+  `#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
+  `#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
+  hardware/networking copied from the old CI server — to be replaced by the infect output.
@@ -83,9 +83,10 @@ repo, a commit, a log, or the dashboard** (§9) — reference locations only.
 | What | Where | How to use |
 |---|---|---|
 | **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; `Host cc-ci` in `~/.ssh/config` (HostName `100.90.116.4`, no ProxyCommand) | Just run `ssh cc-ci` (logs in as **root**). The orchestrator VM is a direct tailnet peer — direct route, no proxy. Pubkey already in cc-ci's `/root/.ssh/authorized_keys`. |
-| **Gitea bot account** | `/srv/cc-ci/.testenv` → `GITEA_USERNAME` (`autonomic-bot`), `GITEA_PASSWORD`, `GITEA_URL` (`git.autonomic.zone`) | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
+| **Gitea bot account** | `/srv/cc-ci/cc-ci/.env.public` → `GITEA_USERNAME` (`autonomic-bot`); `/srv/cc-ci/.testenv` → `GITEA_PASSWORD`; `GITEA_URL` defaults to `git.autonomic.zone` | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |

-Load them in a shell with: `set -a; . /srv/cc-ci/.testenv; set +a` (don't echo the values).
+Load runtime configuration in a shell with: `set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a`
+(don't echo secret values). The helper loads tracked public configuration first and then `.testenv`.

 ### The Tailscale connection (how `ssh cc-ci` works)

@@ -40,19 +40,22 @@ from datetime import datetime, timezone
 HERE = os.path.dirname(os.path.abspath(__file__))
 USED_RECIPES = os.path.join(HERE, "used-recipes.md")
 TESTENV = os.environ.get("CCCI_TESTENV", "/srv/cc-ci/.testenv")
+PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
 NS = "recipe-maintainers"


 def _env() -> dict:
    e = {}
-    try:
-        for ln in open(TESTENV):
+    for path in (PUBLIC_ENV, TESTENV):
+        try:
+            lines = open(path)
+        except OSError:
+            continue
+        for ln in lines:
            ln = ln.strip()
            if "=" in ln and not ln.startswith("#"):
                k, v = ln.split("=", 1)
                e[k] = v.strip().strip('"').strip("'")
-    except OSError:
-        pass
    return e


@@ -46,6 +46,7 @@ from datetime import datetime, timezone

 LOGDIR = "/srv/cc-ci/.cc-ci-logs"
 TESTENV = "/srv/cc-ci/.testenv"
+PUBLIC_ENV = "/srv/cc-ci/cc-ci/.env.public"
 INFRA = {"cc-ci", "cc-ci-orchestrator", "cc-ci-secrets"}
 HOST_REPORTS = "/var/lib/cc-ci-reports"
 # Both kinds live in ONE archive, distinguished by a suffix on a common title.
@@ -56,14 +57,16 @@ KINDS = {"week": ("The Recipe Report", "Week of {d} — full"),

 def _env():
    e = {}
-    try:
-        for ln in open(TESTENV):
+    for path in (PUBLIC_ENV, TESTENV):
+        try:
+            lines = open(path)
+        except FileNotFoundError:
+            continue
+        for ln in lines:
            ln = ln.strip()
            if "=" in ln and not ln.startswith("#"):
                k, v = ln.split("=", 1)
                e[k] = v.strip().strip('"').strip("'")
-    except FileNotFoundError:
-        pass
    return e


@@ -30,13 +30,12 @@ set -o errexit -o nounset -o pipefail
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ORCH="$(dirname "$HERE")"
 SSH="${SSH:-cc-ci}"
-TESTENV="${TESTENV:-/srv/cc-ci/.testenv}"
 RECONCILE="${RECONCILE:-$ORCH/.claude/skills/recipe-upgrade/open-recipe-pr.sh}"
 USED_RECIPES="${USED_RECIPES:-$HERE/used-recipes.md}"

 [ -f "$RECONCILE" ] || { echo "ERROR: reconcile helper not found: $RECONCILE" >&2; exit 1; }
-set -a; . "$TESTENV"; set +a
-: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:?}"
+set -a; . /srv/cc-ci/cc-ci-plan/load-env.sh; set +a
+: "${GITEA_USERNAME:?}"; : "${GITEA_PASSWORD:?}"; : "${GITEA_URL:=git.autonomic.zone}"

 if [ "${1:-}" = "--all" ]; then
  mapfile -t RECIPES < <(awk '!/^[[:space:]]*#/ && ($2=="weekly" || $2=="external") {print $1}' "$USED_RECIPES")
@@ -6,6 +6,14 @@
 | postgres | postgres                                  | https://github.com/postgres/postgres              | https://www.postgresql.org/docs/release/                          |

 ## Standing notes
+  - **2026-08-28 re-check** (endoflife.date/api/mattermost.json 2026-08-28; Docker Hub; GitHub
+    releases): **11.7.10** (released 2026-08-26, "various bug fixes", not prerelease) is the newest
+    11.7.x ESR patch — the ESR/LTS line has NOT moved, still **11.7** (EOL **2027-05-15**). This run
+    bumps open PR #2 **11.7.9 → 11.7.10** (ESR patch within the same line). 11.8/11.9/11.10 remain
+    innovation releases (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.12 expired.
+    postgres **15-alpine HELD** (DB-major 16/17/18 out of scope, plain-postgres recipe w/o
+    pgautoupgrade: operator dump/restore needed). git.autonomic.zone mirror main reconciled to
+    coopcloud (86e0d0b5 = 10.11.22, upstream still pins the EXPIRED 10.11 ESR).
 - mattermost-lts tracks the **11.7 ESR (Extended Support Release)** = Mattermost's LTS line. The "LTS"
  concept in Mattermost is **ESR**; there is NO "10.x LTS line". Release calendar (endoflife.date/mattermost):
  - **10.11** = previous ESR, security support **ends 2026-08-15** (~7 weeks) — migrated away 2026-06-19
@@ -76,6 +84,26 @@
  The previous inline-label approach (no restore hook) was a defect: raw PGDATA restore without a reload
  was a silent no-op. Fixed in PR #2 (restore fix cherry-picked from PR #1 ci/pg-restore).

+  - **2026-08-21 re-check** (endoflife.date/api/mattermost.json 2026-08-17; Mattermost docs; GitHub):
+    **11.7 ESR is the only current ESR/LTS** — latest patch **11.7.9** (2026-08-13, "Low to High
+    security fixes"), EOL 2027-05-15. **10.11 ESR ENDED 2026-08-15** — upstream main still pins
+    10.11.22 (now on an EXPIRED ESR). 11.10/11.9/11.8 are innovation (EOL 2026-11-15/10-15/09-15),
+    NOT LTS. The `-lts` recipe must track 11.7. This run **redirects open PR #2 away from the
+    innovation 11.10.0 onto 11.7.9** (the correct LTS/ESR; ESR→ESR 10.11→11.7 is "fully supported
+    and tested" per Mattermost docs — no staging hop). Refer to the 2026-08-15 report where PR #2
+    was on 11.10.0 (operator chose innovation line that week); operator should now decide the
+    `-lts` line. postgres 15-alpine still HELD (DB-major out of scope).
+
+- **2026-08-31 re-check** (endoflife.date/api/mattermost.json 2026-08-31; Docker Hub tag list;
+    GitHub release `v11.7.10`): **11.7 ESR line UNCHANGED** — latest patch still **11.7.10**
+    (released 2026-08-26, "various bug fixes"), EOL 2027-05-15. PR #2 already carries 11.7.9 →
+    11.7.10; the wildcard cert is RENEWED (valid to 2026-11-29), so this run just RE-VERIFIES PR #2
+    via `!testme` (the 2026-08-28 run's `!testme` was infra-blocked). 11.8.5 / 11.9.1 / 11.10.1
+    remain innovation (EOL 2026-09-15 / 10-15 / 11-15), NOT ESR — do NOT target; 10.11 ESR ended
+    2026-08-15 (upstream main still pins 10.11.22 = EXPIRED ESR → the 10→11 ESR move PR #2 carries
+    remains required; Mattermost docs: ESR→ESR is "fully supported and tested"). postgres 15-alpine
+    still HELD (DB-major out of scope, operator dump/pg_upgrade).
+
 ## NVD CPE fallback
 This project publishes nothing machine-readable we can reach — no GitHub advisory feed,
 no release-attributable changelog — so its CVE count was `?` (nothing measured). NVD is
@@ -65,3 +65,75 @@
 - 2.35.3 (2026-08-14, Pre-release): bugfixes (Google Ads v21→v25 API migration, MS Teams OAuth scope
  restore, workflow publication outbox abort deadline) + feature (skip update approval for workflows
  from same Instance AI session). Not deployed (2.35.2 was the survey target).
+- 2.36.0 (2026-08-18, Pre-release; 2.36.1 was briefly tagged then withdrawn — 2.36.2 compares
+  2.36.0...2.36.2): a large feature minor. Core/ai-builder/editor bugfixes + new features (Agent
+  sandboxes + writable workspaces, MCP folder/move/read-data-table tools, Schedule Trigger missed-
+  execution grace period + catch-up coalescing, Confluence Cloud OAuth2 + page get sub-tree, MiniMax
+  M3/H3, LangSmith session debug export, role-mapping/roles API endpoints, n8n User Auth GA, etc.).
+  No breaking compose/env/migration changes, no `N8N_*` env renames; the deprecation notices
+  (Array.merge→Array.mergeIntoObject in expressions, workflow tags API migration) are API-level only.
+  2.36.2 (2026-08-19, patch): 3 fixes (decorator-route request schema in /discover, end-user
+  credential resolution in node parameters, editor agent-preview alignment). 2.36.3 (2026-08-20,
+  patch): 4 core/editor fixes (task-runners only restart when slow, expression engine init on
+  expression commands, test-webhook isolate release after teardown, Edit Fields type selector styles).
+- 2026-08-21 run: planning 2.34.4 → 2.36.3 (newest tag abra lists). Whole 2.36.x line still
+  Pre-release on GitHub (2.33.5/Latest was superseded by 2.35.x line). Rolling upgrade safe, sqlite
+  mode, TypeORM migrations auto-run on boot. No operator action required.
+- 2.36.4 (2026-08-21, Pre-release): 3 core fixes (env normalization before schema parsing; chat and
+  MCP trigger auth-mode validation for end-user credentials; wrap trigger closeFunction in an
+  expression isolate).
+- 2.36.5 (2026-08-21, Pre-release): 1 core fix (raise AI Assistant model verification token limit).
+- 2.36.6 (2026-08-24): 1 editor fix (personalization survey / community registration modal on
+  Instance AI landing page).
+- 2.36.7 (2026-08-25, **current Stable / Latest badge**; `stable` tag points here): API fix (adapt
+  binary data endpoint), API cross-backport (require a JSON content type on decorator routes that
+  take a body), core fixes (reconcile license auto-renewal on startup leader takeover; route OAuth2
+  token requests through environment proxies), feature (remove offset query param from list workflow
+  history endpoint). This is the newest stable-line release; the 2.37.x line is still pre-release.
+- 2.37.0 (2026-08-25, Pre-release): a large feature minor. Agent Builder / AIA / core/editor
+  bugfixes + features (Slack Agent view, prompt caching for Anthropic, SharePoint node v2 = default,
+  form-trigger OAuth2 (added then reverted), role-mapping rule CRUD endpoints, execution
+  startedAfter/startedBefore filters, Agent/community-package hardening: serialize community
+  package installs + non-destructive downloads + cold-boot storm reduction, MCP tool schemas as JSON
+  Schema 2020-12, distroless n8n image variant). **Deprecation: the "Any workflow" caller policy**
+  (API/config-level — flag for operators using execution-caller policies). Two enforced API behavior
+  changes: **require JSON content type on decorator body routes** and **"adapt binary data endpoint"**
+  (HTTP clients sending a body to those routes must send `Content-Type: application/json`). No
+  breaking compose/env/migration changes, no `N8N_*` env renames.
+- 2.37.1 (2026-08-25, Pre-release): 1 core fix (route OAuth2 token requests through env proxies in
+  every process).
+- 2.37.3 (2026-08-27, Pre-release; 2.37.2 was withdrawn): 2 runner-image fixes (copy glibc into
+  distroless runners image as a matched pair; stage libatomic into the distroless runners image) —
+  relevant if using n8n's community/distroless runner image.
+- 2026-08-28 run: PR #7 extended 2.34.4 → **2.37.3** (newest tag abra lists; 2.37.2 withdrawn, that
+  tag does not exist). 2.36.7 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub,
+  consistent with the tracking-the-newest-tag precedent (2.34.2/2.35.2/2.36.3 were pre-release when
+  taken). Rolling upgrade safe (sqlite, TypeORM auto-migrate on boot). Operator flag: the two 2.37.0
+  API behavior changes (JSON content-type on decorator body routes; binary-data endpoint adapt) only
+  affect HTTP API callers. Recommended release: `-y` (feature minor).
+  NOTE: PR #7's `!testme` on this date was BLOCKED — `*.ci.commoninternet.net` wildcard cert expired
+  (bridge: `CERTIFICATE_VERIFY_FAILED certificate has expired`). UNVERIFIED; operator renewed the cert
+  (now valid to 2026-11-29). The 2.37.3 work must be re-verified by a fresh `!testme`.
+- 2.34.5 (2026-08-12, patch): 1 core fix (apply TLS options per hop through a proxy).
+- 2.34.6 (2026-08-14, patch): 3 fixes (core `continueErrorOutput` `details` field; Google Ads sunset
+  v21→v25 API migration; MS Teams Restore `Group.ReadWrite.All` OAuth2 scope — matches the 2.35.3
+  in-progress backports).
+- 2.35.4/2.35.5/2.35.6/2.35.7 (2026-08-19..21, patches): API schema for decorator routes in /discover;
+  Google Ads v25 view metric rename; task-runners not restarted when only slow; expression engine
+  init on expression commands; test-webhook isolate release after teardown; env normalization before
+  schema parsing; end-user credential resolution; trigger closeFunction isolate; AI Assistant token
+  limit raise. All patch bugfixes, no breaking changes.
+- 2.36.8 (2026-08-28, patch): 1 core fix (domain-restricted credential usable in its own node) —
+  cross-backport of the 2.37.4 fix into the stable line.
+- 2.36.9 (2026-08-31, patch; **now the Stable/Latest badge** — `stable` tag): 1 core fix (apply proxy
+  environment variables consistently across packages and processes, cross-backport of 2.37.1/2.37.3).
+- 2.37.4 (2026-08-28, Pre-release): 1 core fix (allow a domain-restricted credential to work in its
+  own node).
+- 2.37.5 (2026-08-31): **withdrawn** — no plain `2.37.5` tag on Docker Hub (only partial
+  `2.37.5-amd64/-arm64/-pc` artifact tags, no manifest). Skip entirely, like 2.37.2/2.36.1.
+- 2.37.6 (2026-08-31, Pre-release; **newest 2.37.x tag**): 1 core fix (thread execution id through
+  dynamic-credential storage).
+- 2026-08-31 run: PR #7 extended 2.34.4 → **2.37.6** (newest tag abra lists = 2.37.6/2.37.4/2.37.3/…;
+  2.37.5 withdrawn). 2.36.9 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub
+  (consistent precedent). Re-verified 2.37.3→2.37.6 (pure core bugfixes), no breaking changes beyond
+  the already-flagged 2.37.0 API behavior pair. Rolling upgrade safe. Recommended release: `-y`.
@@ -4,6 +4,7 @@
 |---------|-------|-------------|----------------------|
 | app | wordpress | https://github.com/WordPress/WordPress | https://wordpress.org/news/category/releases/ (image: https://hub.docker.com/_/wordpress) |
 | db | mariadb | https://github.com/MariaDB/server | https://mariadb.com/kb/en/release-notes/ (image: https://hub.docker.com/_/mariadb) |
+| ftp (optional overlay) | atmoz/sftp | https://github.com/atmoz/sftp | https://github.com/atmoz/sftp/releases (image: https://hub.docker.com/r/atmoz/sftp) |

 ## Standing notes
 - Enrolled 2026-08-03 (operator request). Recipe = coopcloud `wordpress`; mirror
@@ -22,3 +23,7 @@
  image). mariadb major bumps (e.g. 12.x→13.x) need the usual dump/restore caution if the
  recipe ever pins a non-`mariadb`-auto-upgrading setup; the official image handles minor
  bumps in place.
+- `atmoz/sftp` is pinned as the floating `alpine` tag (updated 2026-07-28, current as of
+  2026-08-21) — not a semver abra can evaluate, so it never contributes an upgrade candidate;
+  verified current by direct Docker Hub tag check. Only bump if a real newer supported tag
+  exists and the app needs it.
@@ -31,7 +31,7 @@ matrix-synapse    weekly
 mattermost-lts    weekly
 mumble            weekly
 n8n               weekly
-plausible         weekly
+plausible         external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 uptime-kuma       external   # maintained elsewhere — used/tested by cc-ci but NOT weekly-upgraded
 wordpress         weekly
 ```
@@ -1,5 +1,28 @@
 {
  "nodes": {
+    "cc-ci": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs"
+        ],
+        "sops-nix": [
+          "sops-nix"
+        ]
+      },
+      "locked": {
+        "lastModified": 1788812004,
+        "narHash": "sha256-Vc7RSeqFHCwlVRhnEEjavuJoyIID+RQdJSygHNA8s8Y=",
+        "ref": "refs/heads/main",
+        "rev": "f6dbfa368995f4d45de09f4052631fd433c87d5b",
+        "revCount": 1533,
+        "type": "git",
+        "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
+      }
+    },
    "nixpkgs": {
      "locked": {
        "lastModified": 1785734586,
@@ -18,6 +41,7 @@
    },
    "root": {
      "inputs": {
+        "cc-ci": "cc-ci",
        "nixpkgs": "nixpkgs",
        "sops-nix": "sops-nix"
      }
@@ -1,32 +1,52 @@
 {
-  description = "cc-ci-orchestrator — NixOS host for the cc-ci loops runtime (Builder/Adversary/Watchdog)";
+  description = "cc-ci-orchestrator — the cc-ci orchestrator (loops, steering session, weekly upgrader) and the NixOS host it shares with the cc-ci CI server";

  inputs = {
-    # Follow the current stable release channel (operator 2026-08-01), was a hard rev pin at
-    # nixpkgs 24.11 (50ab7937, 2025-06-30) kept "the same as the cc-ci server". This host runs
-    # agents/tmux/nginx/docker, not recipe CI, so it does not need to match that server — and a
-    # frozen rev only accrues unpatched CVEs. `nix flake update` now actually moves.
+    # Stable release channel (operator 2026-08-01). `nix flake update` moves it; the cc-ci input
+    # below FOLLOWS it, so one nixpkgs builds the whole combined host and CVEs get patched once.
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

-    # sops-nix follows nixpkgs below, so it no longer needs its own matching pin.
    sops-nix.url = "github:Mic92/sops-nix";
    sops-nix.inputs.nixpkgs.follows = "nixpkgs";
+
+    # The cc-ci CI server, as a NixOS module (`nixosModules.cc-ci-server`). HTTPS, anonymous read:
+    # nix evaluates every input for every output, so the input must be fetchable without
+    # credentials. The private secrets submodule is deliberately NOT fetched through this input —
+    # the host reads the deployed --recursive checkout's secrets.yaml at activation instead
+    # (`cc-ci.sopsFile`). Both `follows` are REQUIRED: without them cc-ci's own nixpkgs/sops-nix
+    # pins would produce a second sops-nix module tree and a second nixpkgs in one system.
+    cc-ci.url = "git+https://git.autonomic.zone/recipe-maintainers/cc-ci.git";
+    cc-ci.inputs.nixpkgs.follows = "nixpkgs";
+    cc-ci.inputs.sops-nix.follows = "sops-nix";
  };

-  outputs = { nixpkgs, sops-nix, ... }:
+  outputs = { self, nixpkgs, sops-nix, cc-ci, ... }:
    let
      system = "x86_64-linux";
    in
    {
-      # Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
-      # Provision with terraform/ then run Stage 2 per terraform/README.md.
-      nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {
-        inherit system;
-        modules = [
-          sops-nix.nixosModules.sops
-          ./nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix
-          ./nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix
-        ];
+      nixosModules = {
+        # The orchestrator itself: loops supervisor, steering session, weekly/hourly timers.
+        cc-ci-orchestrator = ./nix/modules/cc-ci.nix;
+        # The host contract those units assume: loops user, claude/opencode CLIs, opencode web
+        # server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config.
+        orchestrator-host = ./nix/modules/orchestrator-host.nix;
+        # Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09).
+        cc-ci = ./nix/modules/cc-ci.nix;
+      };
+
+      nixosConfigurations = {
+        # THE live host: cc-ci CI server + cc-ci orchestrator on one Hetzner cpx32-class box
+        # (195.201.88.249, since 2026-09). README.md is the deploy guide.
+        cc-ci = nixpkgs.lib.nixosSystem {
+          inherit system;
+          modules = [
+            cc-ci.nixosModules.cc-ci-server
+            self.nixosModules.cc-ci-orchestrator
+            self.nixosModules.orchestrator-host
+            ./nix/hosts/cc-ci/configuration.nix
+          ];
+        };
      };
    };
 }
@@ -10,9 +10,10 @@ metadata:
 The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx22** as of
 2026-05-31, replacing the Incus VM (100.116.55.106).

- Hetzner server **134487234**, public **168.119.126.100**, tailnet **cc-ci-orchestrator-1** @
-  **100.84.190.30**. Flake host **cc-ci-orchestrator-hetzner**.
- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch`
+- Since 2026-09-07: ONE Hetzner host for CI server + orchestrator, public **195.201.88.249**,
+  tailnet **cc-ci**, flake host **`.#cc-ci`** (this repo). Before: orchestrator on Hetzner
+  134487234 (168.119.126.100 / 100.84.190.30, `cc-ci-orchestrator-hetzner`), shared with notplants.
+- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch`
  (`/srv/cc-ci` is a symlink to it). The Bash tool runs as user **loops** (uid 1000, passwordless
  sudo) — plain `nixos-rebuild switch` fails on the profile symlink; use `sudo`.
 - Reboot-resilience: `cc-ci-loops.service` is **enabled** (wantedBy multi-user.target); ExecStartPre
@@ -23,4 +24,4 @@ The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx
  identity unknown". Set per-repo to match prior commits: `autonomic-bot
  <autonomic-bot@git.autonomic.zone>`.

-Full record: `cc-ci-plan/plan-orchestrator-hetzner-migration.md`.
+Full record: `archive/plans/plan-orchestrator-hetzner-migration.md`.
@@ -0,0 +1,68 @@
+# cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
+#
+# This file is only what is physical or identity about the machine: hardware, networking, the
+# tailscale node, root SSH keys, swap, stateVersion. Everything functional comes from modules:
+#   cc-ci.nixosModules.cc-ci-server        recipe-maintainers/cc-ci — swarm, traefik, drone,
+#                                          runner, bridge, dashboard, reports, acme-dns, harness
+#   self.nixosModules.cc-ci-orchestrator   nix/modules/cc-ci.nix — loops, orchestrator, timers
+#   self.nixosModules.orchestrator-host    nix/modules/orchestrator-host.nix — loops user, CLIs
+# See README.md for provisioning (Hetzner Debian → nixos-infect → this flake) and staging.
+{ lib, pkgs, ... }:
+{
+  imports = [
+    ./hardware.nix
+    ./networking.nix
+  ];
+
+  networking.hostName = "cc-ci";
+
+  # ---- cc-ci server identity --------------------------------------------------------------
+  # Public address: acme-dns binds to it and publishes it as the `ns-acme` glue record; the
+  # Gandi A records for ci / *.ci / ns-acme .commoninternet.net point here.
+  cc-ci.publicIPv4 = "195.201.88.249";
+  # cc-ci is a plain flake input here (no private submodule), so the sops file is the one in
+  # the deployed --recursive checkout the weekly sweep runs from (README "Stage the workspace").
+  cc-ci.sopsFile = "/etc/cc-ci/secrets/secrets.yaml";
+
+  # ---- orchestrator identity --------------------------------------------------------------
+  # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
+  cc-ci-orchestrator.ciSshHost = "127.0.0.1";
+
+  # ---- tailscale — auth key staged out of band at /etc/ts-auth-key -----------------------
+  services.tailscale = {
+    enable = true;
+    authKeyFile = "/etc/ts-auth-key";
+    extraUpFlags = [ "--hostname=cc-ci" ];
+  };
+
+  # ---- ssh ----------------------------------------------------------------------------------
+  services.openssh = {
+    enable = true;
+    settings.PermitRootLogin = "yes";
+  };
+  # Root keys: PUBLIC keys, tracked deliberately in ./ssh-keys (one per line, blank lines ok).
+  users.users.root.openssh.authorizedKeys.keys =
+    builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+  # The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
+  users.users.loops.openssh.authorizedKeys.keys =
+    builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+
+  # ---- firewall -------------------------------------------------------------------------------
+  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module. The tailscale
+  # interface is trusted, which is what makes the opencode UI on 8443 tailnet-only.
+  networking.firewall = {
+    enable = true;
+    trustedInterfaces = [ "tailscale0" ];
+    allowedTCPPorts = [ 22 ];
+  };
+  networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
+
+  # ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------
+  swapDevices = [ { device = "/swapfile"; size = 8192; } ];
+
+  # ssh client for root (the orchestrator's `ssh cc-ci` goes through the loops user's own config).
+  environment.systemPackages = [ pkgs.openssh ];
+
+  # Fresh NixOS 26.05 install (nixos-infect, 2026-09-07). Never change this on an existing host.
+  system.stateVersion = "26.05";
+}
@@ -0,0 +1,19 @@
+# Generated by nixos-infect on this machine (2026-09-07), captured verbatim per README §3.
+# The ESP UUID is specific to THIS server; a new server gets a new file.
+{ modulesPath, ... }:
+{
+  imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
+  boot.loader = {
+    efi.efiSysMountPoint = "/boot/efi";
+    grub = {
+      efiSupport = true;
+      efiInstallAsRemovable = true;
+      device = "nodev";
+    };
+  };
+  fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/E079-7D41"; fsType = "vfat"; };
+  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
+  boot.initrd.kernelModules = [ "nvme" ];
+  fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
+  
+}
@@ -0,0 +1,38 @@
+# Generated by nixos-infect on this machine (2026-09-07), captured per README §3, with ONE edit:
+# `defaultGateway` as an attrset WITH `interface = "eth0"`. The generated bare-string form leaves
+# NixOS ≥25.05 without a default route (the host boots and is unreachable) — see README §2.
+{ lib, ... }: {
+  # This file was populated at runtime with the networking
+  # details gathered from the active system.
+  networking = {
+    nameservers = [ "2a01:4ff:ff00::add:2"
+ "2a01:4ff:ff00::add:1"
+ "185.12.64.2"
+ ];
+    defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };
+    defaultGateway6 = {
+      address = "fe80::1";
+      interface = "eth0";
+    };
+    dhcpcd.enable = false;
+    usePredictableInterfaceNames = lib.mkForce false;
+    interfaces = {
+      eth0 = {
+        ipv4.addresses = [
+          { address="195.201.88.249"; prefixLength=32; }
+        ];
+        ipv6.addresses = [
+          { address="2a01:4f8:1c1c:a9b::1"; prefixLength=64; }
+{ address="fe80::2ff8:e3ea:bbb8:aa39"; prefixLength=64; }
+        ];
+        ipv4.routes = [ { address = "172.31.1.1"; prefixLength = 32; } ];
+        ipv6.routes = [ { address = "fe80::1"; prefixLength = 128; } ];
+      };
+      
+    };
+  };
+  services.udev.extraRules = ''
+    ATTR{address}=="92:00:09:d5:ec:0d", NAME="eth0"
+    
+  '';
+}
@@ -0,0 +1,10 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZGp/DQTFuD1GvsyTzCVBUTmoWqcb5T+Z7zZo5nYLXO
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhgo41nt8/L+Cr0PKd8jQK45mw/A+h041j6LQ8JWZisEVaQOzr6s9rxPL8VT5ML4P3/4bMblzdDiXWlJxymcb+yk5S5TnVrMavzHEDhWHwEvTRMe6xNTmsU6cmmhRw7PJqqQ+0GTlQalu3I4jkC0kTF7kuPwduUOgUuSpJqxvDTwYiXoyVnOQHAIygh+BmQvYUz0PBfQgIhgcbYmGZ++T0DnMzdGFzW2UB/iy5mymnpmbaZCgLy0w8AoDE+0YLtUc4gwTXc183nvqO1i7LQr+3jBYkv5ZthCCc52vXFHDSw9xZ5ohsOrBvoi5foRbqinmU5/t0aTK7SSrat7xXm/odIOyS+S7PJyeEcsXN6d5zdxbabAy5vLfodEaKGZd4rqQeDCxOTPAS/BlrBV/EV714n4E+fSOAllAuMBO4IibJM/gLJrh2Dql3co50QW9HEDeSC7iqp2lxRBDxvUs3rIEzy7o4HSN8chqBUK1bbBY6B17fuNHIpBAw4akRVVvPnVM= trav@trav480sweet
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6jrKj7iZUNRLBTZG0vZM1D/BXtARhhB4+GrvpyuqmPb9iw2ifT9YqRUwgyGrOW9U6nIAR9yFnfp9+FkyhEKWByqEBbe/zYKlGLRGjfsIdDdW29QQ3hvmqNyboCkXLxZGat93poYhnoomqicmGD/xST4s0OUhcK9E494lUmenlD9dcMZW1aKpJ+9O4Dq6A7nk2z1e4KFcZdrZDI2Hgg+gfEdsKZQqd/R3Mls/eVKpzhfv3Y8BiNoHssUChVf8IGESqTOBOR7Dk7FsU5Z2ZcnQ1coxY7VlBn4fPjTWmz/Ac0jLqgcpCLpNyQzFPDVMYZKYrPVoqBeKVhN5YnfwR5OVP8YsakT/obLwC43sx/esXfjhVGcsRoGpiLOfazzNw/eC8s6FlS8cesOubEM37a7F25z4UEG3d487oM7EjQ39gBCCj/KRgUimCKMWsm6yIas4OSctBWEAo/NhZp0gwulSRxleW6eJCNNwzOmWjdzYIVWoVP0EIeM95Tq8PVUN7gpc= aadil@t480
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMyHSi12R0+HCVBz7+d9fyOBnoJi8Nsj5D7vQ9UQO8a5
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQFuqUB2qNZSDNjDsjjhVA/WnnQNVAMmsUscW6OgMDN
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS notplants-orchestrator
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
@@ -0,0 +1,157 @@
+# cc-ci.nix — the cc-ci ORCHESTRATOR: the Builder/Adversary loops supervisor, the operator's
+# steering session, and the weekly-upgrade + hourly-supervisor timers. Nothing else.
+#
+# Exported from this repo's flake as `nixosModules.cc-ci-orchestrator` (and, for the host that
+# used to import it under the old name, `nixosModules.cc-ci`). Split out of the shared agent
+# host config on 2026-08-20; since 2026-09 it runs on the same Hetzner host as the CI server
+# itself (`#cc-ci` in flake.nix), next to recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
+#
+# All of it assumes the cc-ci workspaces exist on the host:
+#   /srv/cc-ci        the loops workspace (+ .cc-ci-logs, upgrader.env) — a symlink to
+#   /srv/cc-ci-orch   this repo (the orchestrator's own working dir), with cc-ci/ checked out
+# and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
+# those are host concerns, provided by nix/modules/orchestrator-host.nix, not by this module.
+{ config, pkgs, lib, ... }:
+{
+  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
+  systemd.services.cc-ci-loops = {
+    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
+    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
+      # default (control-group) systemd kills every leftover process in the cgroup when the unit
+      # stops — and since one tmux server hosts every agent session on this host, a rebuild that
+      # merely touched this unit wiped all of them (operator 2026-08-01). Only the (already
+      # exited) main process is killed now; `systemctl stop` therefore does NOT tear down agents.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci/cc-ci";
+      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
+      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
+    };
+    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
+    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
+    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
+    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
+      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
+      export PATH="/home/loops/.local/bin:$PATH"
+      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
+        echo "workspace not staged yet — skipping loop start"
+    '';
+  };
+
+  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
+  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
+  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
+  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
+  # (backend/model chosen there — Claude Code under Remote Control since 2026-09-07; before that
+  # opencode/glm-5.2 attached to the shared opencode web server, opencode-web.service in
+  # orchestrator-host.nix, which the upgrader still uses). The harness watchdog (started by
+  # `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
+  # be killed just for idling). Added 2026-08-03 for reboot-resilience.
+  systemd.services.cc-ci-orchestrator = {
+    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
+      # must not tear down the (shared) tmux server and every agent session with it.
+      KillMode = "process";
+      Type = "oneshot"; RemainAfterExit = true;
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci-orch";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      proj="/srv/cc-ci-orch"
+      echo "$(cat /proc/sys/kernel/random/boot_id) boot $(date -u +%FT%TZ) — cc-ci-orchestrator up" \
+        >> "$proj/cc-ci-plan/.ao-boot.log" 2>/dev/null || true
+      cd "$proj" && python3 cc-ci-plan/agents.py up orchestrator || echo "cc-ci orchestrator agents.py up failed"
+    '';
+  };
+
+  # Weekly recipe upgrade — runs /upgrade-all over every enrolled recipe (opens recipe PRs
+  # verified by !testme, never merges). Replaces the boot-fragile busybox-crond-in-tmux from
+  # phase 5 §4 with a reboot-safe systemd timer. The service is timer-triggered only (NOT
+  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
+  systemd.services.cc-ci-upgrade-all = {
+    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
+    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci";
+      # Optional per-run overrides for backend/model (LOOP_BACKEND, LOOP_MODEL, OPENCODE_SHARE,
+      # UPGRADER_ARGS, …). The leading "-" makes it optional: absent file → claude/sonnet defaults.
+      # Current config (as of 2026-08-16): the upgrader + report run on tinfoil/deepseek-v4-pro
+      # (LOOP_MODEL + REPORT_MODEL in the env file); the hourly SUPERVISOR stays on glm-5.2
+      # (SUPERVISOR_MODEL defaults to opencode-go/glm-5.2 in launch-supervisor.py, NOT overridden
+      # here). Subagents bind deepseek via the cc-ci repo's opencode config. LOOP_TIER=zen is kept
+      # so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to
+      # the zen endpoint, which returns 200 (not 429) → resume immediately (correct: tinfoil has no
+      # rolling usage limit to wait out). No rebuild needed to switch — the env file is read at each
+      # timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
+      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
+    };
+    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      python3 /srv/cc-ci/cc-ci-plan/launch-upgrader.py start >> /srv/cc-ci/.cc-ci-logs/upgrader-cron.log 2>&1
+    '';
+  };
+
+  systemd.timers.cc-ci-upgrade-all = {
+    description = "Weekly trigger for cc-ci-upgrade-all (Thursdays 22:00 America/New_York — Boston 10pm)";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      # 10pm Thursday Boston time — DST-aware (EDT→02:00 UTC, EST→03:00 UTC) via the tz in OnCalendar.
+      OnCalendar = "Thu *-*-* 22:00:00 America/New_York";
+      Persistent = true;  # if the box was down at the scheduled time, run once on next boot
+    };
+  };
+
+  # Hourly SUPERVISOR — a glm-5.2 orchestrator wake-up that keeps the weekly run on track. The
+  # log-idle/429 watchdog only handles opencode-go usage-limit stalls; it does NOT cover a host
+  # disk-full crash (which killed the 2026-07-03 run) or any other environmental wedge. This is a
+  # CHEAP deterministic gate: if the weekly run is complete or actively progressing it does NOTHING
+  # (zero model tokens). Only when a run has stalled/died before completing does it launch a
+  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
+  systemd.services.cc-ci-upgrade-supervisor = {
+    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
+    after = [ "network-online.target" "tailscaled.service" ];
+    wants = [ "network-online.target" ];
+    serviceConfig = {
+      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
+      User = "loops"; Group = "users";
+      WorkingDirectory = "/srv/cc-ci";
+      # Shares the weekly run's optional override file (e.g. SUPERVISOR_MODEL=…); "-" = optional.
+      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
+    };
+    environment = { HOME = "/home/loops"; };
+    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
+    script = ''
+      export PATH="/home/loops/.local/bin:$PATH"
+      python3 /srv/cc-ci/cc-ci-plan/launch-supervisor.py check >> /srv/cc-ci/.cc-ci-logs/supervisor-cron.log 2>&1
+    '';
+  };
+
+  systemd.timers.cc-ci-upgrade-supervisor = {
+    description = "Hourly trigger for cc-ci-upgrade-supervisor (weekly-run health check + drive)";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      OnCalendar = "*-*-* *:07:00";  # every hour at :07 (offset from the weekly :00 fire)
+      Persistent = false;            # a missed hourly check is moot — the next hour re-checks
+    };
+  };
+}
@@ -0,0 +1,203 @@
+# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
+# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
+# the standalone claude/opencode CLIs, the shared opencode web server and its tailnet-only UI,
+# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
+#
+# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
+# `nixosModules.cc-ci-orchestrator`; the combined CI-server + orchestrator host (`#cc-ci`) also
+# imports recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
+#
+# History: until 2026-09 this lived (twice, drifting) in nix/hosts/cc-ci-orchestrator-hetzner/
+# configuration.nix here and in notplants-nix's hosts/notplants-orchestrator/configuration.nix,
+# the shared agent box that also ran lichen + project-orchestrator. The cc-ci half moved to its
+# own host; this file is that half.
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.cc-ci-orchestrator;
+in
+{
+  options.cc-ci-orchestrator = {
+    ciSshHost = lib.mkOption {
+      type = lib.types.str;
+      default = "127.0.0.1";
+      example = "100.95.31.88";
+      description = ''
+        Where `ssh cc-ci` (used by every skill and script that drives the CI server) connects to,
+        as root with ~loops/.ssh/cc-ci-root-ed25519. On the combined host the CI server IS this
+        machine, so the default is loopback; a standalone orchestrator points it at the CI
+        server's tailnet address.
+      '';
+    };
+
+    opencodeUiPort = lib.mkOption {
+      type = lib.types.port;
+      default = 8443;
+      description = ''
+        TLS port of the nginx front door for the opencode web UI. Not 443: on the combined host
+        Traefik (docker swarm) owns 80/443. The port is not opened in the firewall, so it is
+        reachable only over the trusted tailscale interface.
+      '';
+    };
+
+    opencodeUiHost = lib.mkOption {
+      type = lib.types.str;
+      default = "oc.commoninternet.net";
+      description = "nginx server_name for the opencode web UI (self-signed, basic auth).";
+    };
+  };
+
+  config = {
+    # ---- the loops user -------------------------------------------------------------------
+    # claude sessions run as non-root (--dangerously-skip-permissions is refused for root).
+    users.users.loops = {
+      isNormalUser = true;
+      uid = 1000;                 # fixed: workspace files are rsynced between hosts by uid
+      home = "/home/loops";
+      shell = pkgs.bash;
+      extraGroups = [ "wheel" "docker" ];
+    };
+    security.sudo.wheelNeedsPassword = false;
+    security.sudo.extraRules = [{
+      users = [ "loops" ];
+      commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
+    }];
+
+    # /home/loops/.local/bin holds the standalone claude + opencode binaries; it must be first on
+    # every PATH (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
+    environment.variables.PATH = lib.mkForce
+      "/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
+
+    # ---- nix-ld: the standalone Claude Code / opencode CLIs are foreign dynamic ELF binaries ---
+    programs.nix-ld.enable = true;
+    programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
+
+    # ---- the toolbox every agent on this box gets ----------------------------------------
+    # Bar for adding something: an agent doing ordinary work would otherwise waste a turn
+    # discovering it is absent.
+    environment.systemPackages = with pkgs; [
+      git tmux python3 jq curl cacert
+      gnused gawk coreutils gnugrep findutils util-linux nettools openssh
+      age sops ssh-to-age
+      wget gnutar gzip unzip zip xz
+      ripgrep fd tree file less which
+      procps psmisc htop lsof strace ncdu
+      dnsutils socat netcat-gnu iproute2 iputils
+      openssl gnumake gcc pkg-config
+      yq-go diffutils patch rsync bubblewrap
+    ];
+
+    # ---- ssh config for the loops user: `ssh cc-ci` = the CI server (root) -----------------
+    # Written only if absent so a manual customisation survives rebuilds.
+    system.activationScripts.loopsSshConfig = ''
+      mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
+      if [ ! -f /home/loops/.ssh/config ]; then
+        cat > /home/loops/.ssh/config <<'SSHCFG'
+Host cc-ci
+    HostName ${cfg.ciSshHost}
+    User root
+    IdentityFile /home/loops/.ssh/cc-ci-root-ed25519
+    IdentitiesOnly yes
+    StrictHostKeyChecking accept-new
+    ServerAliveInterval 30
+
+Host git.autonomic.zone
+    HostName git.autonomic.zone
+    Port 2222
+    User git
+    IdentityFile /home/loops/.ssh/autonomic-bot-gitea-ed25519
+    IdentitiesOnly yes
+
+Host tangled.org
+    IdentityFile /home/loops/.ssh/tangled-ed25519
+    IdentitiesOnly yes
+SSHCFG
+        chmod 600 /home/loops/.ssh/config
+        chown loops:users /home/loops/.ssh/config
+      fi
+    '';
+
+    # ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
+    systemd.services.claude-install = {
+      description = "Install Claude Code CLI for loops user (idempotent)";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
+      environment = { HOME = "/home/loops"; };
+      path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
+      script = ''
+        if [ ! -x "$HOME/.local/bin/claude" ]; then
+          echo "installing Claude Code CLI for loops user..."
+          curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
+        fi
+      '';
+    };
+
+    systemd.services.opencode-install = {
+      description = "Install opencode CLI for loops user (idempotent)";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
+      environment = { HOME = "/home/loops"; };
+      path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
+      script = ''
+        if [ ! -x "$HOME/.local/bin/opencode" ]; then
+          echo "installing opencode CLI for loops user..."
+          curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation"
+          # The installer puts the binary in ~/.opencode/bin; every unit here expects ~/.local/bin.
+          if [ -x "$HOME/.opencode/bin/opencode" ]; then
+            mkdir -p "$HOME/.local/bin" && ln -sfn "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode"
+          fi
+        fi
+      '';
+    };
+
+    # ---- opencode web server: one shared instance the opencode-backed agents attach to -------
+    # Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
+    systemd.services.opencode-web = {
+      description = "opencode web server for cc-ci agents";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" "tailscaled.service" "opencode-install.service" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = {
+        Type = "simple";
+        User = "loops"; Group = "users";
+        WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
+        EnvironmentFile = [ "-/srv/cc-ci/cc-ci/.env.public" "/srv/cc-ci/.testenv" ];
+        ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
+        ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
+        Restart = "on-failure";
+        RestartSec = "5s";
+      };
+      environment = {
+        HOME = "/home/loops";
+        PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
+      };
+      path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
+    };
+
+    # ---- tailnet-only nginx front door for the opencode UI -------------------------------
+    # Self-signed cert + basic auth, both created out of band (a store path would be world
+    # readable) — see README "Secrets to stage". nginx FAILS TO START if they are missing.
+    #   /etc/nginx/oc-selfsigned.crt  root:nginx 0644
+    #   /etc/nginx/oc-selfsigned.key  root:nginx 0640
+    #   /etc/nginx/oc-htpasswd        root:nginx 0640   (`oc:<bcrypt>`; plaintext in /secrets)
+    services.nginx = {
+      enable = true;
+      recommendedProxySettings = true;
+      virtualHosts.${cfg.opencodeUiHost} = {
+        listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiPort; ssl = true; } ];
+        # onlySSL flags the vhost as SSL so the module renders ssl_certificate for the listener.
+        onlySSL = true;
+        sslCertificate = "/etc/nginx/oc-selfsigned.crt";
+        sslCertificateKey = "/etc/nginx/oc-selfsigned.key";
+        basicAuthFile = "/etc/nginx/oc-htpasswd";
+        locations."/" = {
+          proxyPass = "http://127.0.0.1:4096";
+          proxyWebsockets = true;
+        };
+      };
+    };
+  };
+}
Author	SHA1	Message	Date
notplantsandClaude Fable 5.1	4bc483326b	orchestrator-host: symlink the opencode installer's binary into ~/.local/bin; README: nginx file perms, first-rebuild wrapper, pre-cutover checks Found on the first activation of #cc-ci on 195.201.88.249: the opencode installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and nginx's pre-start config check runs as the nginx user, so the staged cert/htpasswd must be root:nginx 0640, not root:root 0600. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:24:47 +00:00
notplantsandClaude Fable 5.1	f105808bed	nix/hosts/cc-ci: real hardware + networking from the box; document the infect fixes hardware.nix / networking.nix are now the nixos-infect output of 195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that matters: defaultGateway as an attrset with interface = "eth0". README §2a records what it actually took to get NixOS running on the Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's unit files shadowed every NixOS service), and the gateway fix applied via a chroot rebuild with the nix sandbox off. flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned to 1000 so workspace rsyncs by uid line up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:15:16 +00:00
notplantsandClaude Fable 5.1	31af820079	nix: one Hetzner host for the CI server + the orchestrator (#cc-ci) The orchestrator's flake now builds the machine it shares with the cc-ci CI server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server (new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix — the host contract those units always assumed (loops user, claude/opencode CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns 80/443, nix-ld, tool set, `ssh cc-ci` → loopback). nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old server's layout so the flake evaluates; they get replaced by the nixos-infect output of 195.201.88.249. README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake → staging → data restore → cutover). archive/ holds the retired Incus/Hetzner orchestrator host configs, the old terraform and the migration plans; references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working plan for the move. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 19:58:33 +00:00
autonomic-bot	5bebd13e49	docs: record weekly upgrade report Completed DeepSeek Flash run and published report verified.	2026-08-31 20:47:10 +00:00
autonomic-bot	82d3127c2a	docs: record weekly upgrade report	2026-08-31 20:46:55 +00:00
autonomic-bot	d824800f8a	upstream(n8n): 2026-08-31 release-notes (2.34.5/2.34.6, 2.35.4-7, 2.36.8/2.36.9, 2.37.4, 2.37.5 withdrawn, 2.37.6)	2026-08-31 20:31:59 +00:00
autonomic-bot	e7f861ec9e	upstream(mattermost-lts): 2026-08-31 re-check note (ESR still 11.7.10, PR #2 re-verify)	2026-08-31 20:20:09 +00:00
autonomic-bot	1b75d986b2	fix: default testme Gitea URL Verified with bash -n; public default only.	2026-08-31 19:29:59 +00:00
autonomic-bot	e72fbdb79c	fix: default testme Gitea URL	2026-08-31 19:29:31 +00:00
autonomic-bot	f6c6b39b18	docs: record acme-dns production issuance	2026-08-31 19:04:54 +00:00
autonomic-bot	02ff90777f	docs: record acme-dns production issuance	2026-08-31 19:04:41 +00:00
autonomic-bot	012212498d	docs: record acme-dns handoff	2026-08-31 17:17:09 +00:00
autonomic-bot	521f6e6934	docs: record acme-dns handoff	2026-08-31 17:16:51 +00:00
autonomic-bot	07bc0a15d5	Merge pull request 'config: load public cc-ci runtime environment' (#14 ) from chore/public-runtime-config into main	2026-08-31 16:50:25 +00:00
autonomic-bot	00a4ee6752	config: load public cc-ci runtime environment	2026-08-31 16:49:15 +00:00
autonomic-bot	9c619abd6c	upstream(n8n): release-notes sources 2.36.4..2.37.3	2026-08-28 04:15:13 +00:00
autonomic-bot	980530f505	upstream(mattermost-lts): 2026-08-28 re-check — 11.7.10 newest ESR patch	2026-08-28 04:00:02 +00:00
autonomic-bot	3047ee7724	config: GITEA_URL is public config, not a credential The Gitea host was stored in the credentials env file, so a literal secret scanner treated it as a secret and refused two READMEs in another repo that link to git.autonomic.zone. It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py already default it to git.autonomic.zone, and it is this repo's own remote host. It is a bare hostname with no scheme, userinfo or path. Removed from the env file and defaulted here the same way the Python already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard -- those are real credentials and must still fail loudly if unset.	2026-08-26 17:46:58 +00:00
autonomic-bot	c472aa3642	upstream(wordpress): release-notes sources (sftp/atmoz + standing note)	2026-08-21 03:45:57 +00:00
autonomic-bot	7252f7ee35	upstream(mattermost-lts): 2026-08-21 re-check — 11.7.9 ESR is the only current LTS; PR #2 redirected to ESR	2026-08-21 03:34:36 +00:00
autonomic-bot	75386fc23b	upstream(n8n): 2.36.x release-notes coverage (2.36.0 feature minor, 2.36.2/2.36.3 patches)	2026-08-21 03:28:47 +00:00
autonomic-botandClaude Opus 4.8	19abebb312	nix: split the cc-ci-specific config into its own module + flake output The orchestrator host is a general agent box that also serves several unrelated projects (atproto-likes, the lichen and project orchestrators, the shared opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci orchestrator session, the weekly recipe-upgrade run and its hourly supervisor — now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so the two can evolve and be reviewed independently and another host can consume just the cc-ci half. No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the split: the unit set is identical to the running system, and all six cc-ci units are byte-identical modulo nix store hashes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet	2026-08-20 16:29:02 +00:00
autonomic-bot	565e021185	Merge pull request 'config: drop plausible from weekly upgrade set (-> external)' (#13 ) from config/drop-plausible-weekly into main	2026-08-17 16:01:38 +00:00
autonomic-bot	929faeb055	config: drop plausible from weekly upgrade set (-> external) Move plausible from weekly to external in used-recipes.md. cc-ci still deploys/tests it (the recipe + tests stay enrolled, !testme bridge unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all run — maintained elsewhere, like uptime-kuma. The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's run stays open for the operator; no future weekly PRs will be opened for it.	2026-08-17 16:01:26 +00:00
autonomic-bot	a29b993160	Merge pull request 'config: switch upgrader + report to deepseek, keep supervisor on glm' (#12 ) from config/deepseek-upgrader-model into main	2026-08-16 02:29:20 +00:00
autonomic-bot	a0d6fc9417	config: switch upgrader + report to deepseek, keep supervisor on glm The weekly /upgrade-all parent session and the /recipe-report session now run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env). The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default in launch-supervisor.py, not overridden). Subagents already bind deepseek via the cc-ci repo's opencode config (fix from 2026-08-10, verified this week: all 16 subagents across both waves ran deepseek-v4-pro). LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit probe sends the deepseek model name to the zen endpoint, which returns 200 (not 429) → resume immediately — correct, since tinfoil has no rolling usage limit to wait out. Verified the probe behaviour with a direct curl. Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance' (opencode zen workspace balance exhausted), then sat unfinished for 40h while the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use API key) has no rolling balance limit, so this can't recur. Also documents the session recovery in JOURNAL.md (the stalled run was completed via a fresh scoped upgrader — the original 2.58M-token session was unresumable: the inference endpoint silently drops the oversized request).	2026-08-16 02:28:38 +00:00
autonomic-bot	ab5d11ea2a	Merge pull request 'advisory-scan: NVD by CPE, so mattermost and mumble stop scanning as '?'' (#11 ) from feat/nvd-cpe-source into main	2026-08-11 22:17:13 +00:00