README + journal: the /secrets layout, and the weekly run's flash models

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
/secrets is the authoritative location for every secret, incl. the ssh host keys
2026-09-08 17:30:18 +00:00 · 2026-09-08 17:23:28 +00:00 · 2026-09-08 17:07:23 +00:00 · 2026-09-08 16:53:57 +00:00 · 2026-09-08 16:53:20 +00:00 · 2026-09-08 16:50:20 +00:00
50 changed files with 1769 additions and 587 deletions
@@ -1,6 +1,6 @@
 ---
 name: cc-ci-orchestrator-update
-description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
+description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
 ---

 # cc-ci-orchestrator-update (thin wrapper)
@@ -1,6 +1,6 @@
 ---
 name: cc-ci-server-update
-description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
+description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
 ---

 # cc-ci-server-update (thin wrapper)
@@ -1,6 +1,6 @@
 ---
 name: cc-ci-update
-description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
+description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
 ---

 # cc-ci-update (thin wrapper)
@@ -9,14 +9,16 @@
 # SAFE BY ALLOWLIST. It removes ONLY things NOT on the keep-list, so it can never take down infra or
 # the warm canonicals. The keep-list (leading name prefix) is:
 #   - traefik, drone, backups            : Swarm + CI infra
-#   - ccci-bridge / -dashboard / -reports: the cc-ci control plane
+#   - ccci-bridge / -dashboard / -reports / -opencode-ui: the cc-ci control plane (opencode-ui is
+#     the traefik route to the orchestrator's opencode web UI, deployed by the host's nix config —
+#     the 2026-09-07 run reaped it as an "orphan" before it was on this list)
 #   - warm-*                             : warm canonicals (idle persistent deps reused across runs;
 #                                          their retained volumes are spared too)
 # Everything else deployed on the Swarm is a per-run test stack and is fair game.
 set -uo pipefail
 export PATH=/run/current-system/sw/bin:$PATH

-KEEP_RE='^(traefik|drone|backups|ccci-(bridge|dashboard|reports)|warm-)'
+KEEP_RE='^(traefik|drone|backups|ccci-(bridge|dashboard|reports|opencode-ui)|warm-)'
 removed=0

 echo "== orphan sweep: scanning (keep-list: infra + warm-* canonicals) =="
@@ -32,3 +32,6 @@ __pycache__/
 # Local API tokens — never committed (advisory-scan / hetzner recovery)
 .github-token
 .hcloud-token
+
+# auto-update.nix keeps the pre-update lock here so a failed run can restore it (never committed)
+.flake.lock.auto-update-prev
@@ -1,164 +1,87 @@
 ---
 name: cc-ci-orchestrator-update
-description: Update the cc-ci ORCHESTRATOR host (the machine the orchestrator itself runs on) to the latest nixos-26.05 nixpkgs, then nixos-rebuild test + switch + verify, ship the bump as a verified PR on recipe-maintainers/cc-ci-orchestrator and merge it directly (the skill invocation IS the authorization). The orchestrator-host analogue of /cc-ci-server-update. Self-update caveats apply — the switch restarts services on the very host running this session; always `nixos-rebuild test` before `switch` so a reboot recovers to the last-known-good generation. The end-of-run report includes the merged PR link + a summary of what changed. Invoke as /cc-ci-orchestrator-update (also answers to /cc-ci-update-orchestrator).
+description: Update the cc-ci HOST — since 2026-09 the ONE Hetzner machine that runs both the cc-ci CI server and the orchestrator (built from this repo's `#cc-ci`, which imports cc-ci's server module) — to the latest nixos-26.05 nixpkgs, sops-nix and cc-ci main, then `nixos-rebuild test` → cc-ci health checks → `switch`, committing the health-checked flake.lock straight to main (the skill invocation IS the authorization; the operator reviews retrospectively). It drives the host's own weekly auto-update unit by hand, so a manual and an automatic update are the same code path. Refuses while CI is busy. `--cc-ci-only` moves only the cc-ci input (harness/tests/modules), for after a cc-ci merge. Answers to /cc-ci-orchestrator-update, /cc-ci-update-orchestrator and (since the hosts merged) /cc-ci-server-update.
 ---

 # cc-ci-orchestrator-update

-Manually update the **orchestrator host**'s NixOS dependencies (nixpkgs; sops-nix follows nixpkgs)
-to the current `nixos-26.05` channel tip, ship the bump via a verified PR + a live
-`nixos-rebuild test` → `switch`, and confirm the host is healthy afterward. This is the
-orchestrator-host analogue of `/cc-ci-server-update` — done deliberately, with a reviewable PR and
-a health gate, not as silent drift.
+Update **the cc-ci host** — one machine since 2026-09-07 (`cc-ci`, 195.201.88.249, no tailnet):
+the CI server (docker swarm, traefik, drone, runner, bridge, dashboard, reports, acme-dns) AND the
+orchestrator (loops, weekly upgrader, opencode agent) — to the current inputs, verified, committed.

-> **Two hosts, two flakes — don't confuse them.** This skill updates the **orchestrator** host:
-> the machine this session runs on (`cc-ci-orchestrator-1`, Hetzner cpx22 **server 134487234**,
-> tailnet `100.84.190.30`, public `168.119.126.100`), flake checkout **`/srv/cc-ci-orch`** (repo
-> `recipe-maintainers/cc-ci-orchestrator`), target **`.#cc-ci-orchestrator-hetzner`**. The **cc-ci
-> CI server** (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, target `.#cc-ci`) is a different
-> machine — that's `/cc-ci-server-update`, NOT this skill.
+> **One flake, one target.** The host is `nixosConfigurations.cc-ci` in
+> `recipe-maintainers/cc-ci-orchestrator` (`/srv/cc-ci-orch` on the host). The cc-ci repo's own
+> `flake.nix` inputs are IRRELEVANT to the live build — this flake makes the `cc-ci` input follow
+> its nixpkgs/sops-nix. Never run `nixos-rebuild` on the cc-ci repo's `#cc-ci` target on this host:
+> that is the standalone-server config and would replace the combined one (orchestrator half gone).

-> **This is a SELF-update.** The `switch` restarts units on the host running this very session.
-> tmux sessions and the orchestrator survive a switch (systemd restarts don't kill user tmux), but
-> `tailscaled`/`sshd` restarts can briefly drop Remote Control / SSH — run activation steps
-> detached (`setsid nohup … &`) and re-poll. A full **reboot** DOES end the session: the
-> auto-launch flow (REBOOTS.md + `cc-ci-loops.service`) restarts the loops, and the operator
-> reconnects the orchestrator — announce a planned reboot to the operator first.
-
-## What this skill does NOT do
-
- **Does NOT touch the cc-ci CI server** — that's `/cc-ci-server-update`.
- **Does NOT bump recipe images or cc-ci tests** — `/upgrade-all` / `/cc-ci-tests-update`.
- **Does NOT change the channel** (`nixos-26.05` branch ref in `flake.nix`) without operator
-  choice — a channel jump (e.g. → 26.11 when it releases) is a bigger change: ask first, read the
-  release notes, and expect module/option renames.
+The work is done by **`cc-ci-auto-update.service`** (`nix/modules/auto-update.nix`), the same unit
+the Tuesday 03:00 UTC timer fires. Running it by hand gives you the identical sequence —
+busy gate → `nix flake update` → build → `switch-to-configuration test` → 45 s settle → health
+check (sshd, 0 failed units, docker/acme-dns/fail2ban/nginx/opencode-web/runner active, every swarm
+service at its replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) →
+profile + bootloader → `flake.lock` committed as `auto-update` and pushed to `main` →
+`/etc/cc-ci` fast-forwarded — with a journal to read and a one-line state file.

 ## Preconditions

- You are on the orchestrator host; `/srv/cc-ci-orch` is the live checkout (git repo, remote
-  `recipe-maintainers/cc-ci-orchestrator`).
- Quiescent moment: no `/upgrade-all` in flight, no active build-loop phase, no cc-ci CI run that
-  would be disturbed if this host blips (`pgrep -af run_recipe_ci` empty on cc-ci is a bonus, but
-  the orchestrator host restarting mostly risks its own sessions, not the CI server).
- `git status` in `/srv/cc-ci-orch` — note pre-existing dirt; don't sweep unrelated changes into
-  the bump commit.
+- `ssh cc-ci` works from the notplants-orchestrator box (root on the host), or you are on the host.
+- **Not busy.** The unit checks itself (CI run, weekly upgrader, report, sweep, running Drone
+  builds → `result=skipped note=busy`, exit 0). Don't fight it: wait, or `tmux ls` as loops and
+  `docker service ls` to see what is running. Never kill a run to update a host.
+- Note the current generation so you can name the old one in the report:
+  `ssh cc-ci 'readlink /run/current-system; cat /srv/cc-ci/.cc-ci-logs/auto-update-state'`.

 ## Procedure

-### 1. Snapshot baseline
+```bash
+# 1. what will move
+ssh cc-ci 'cd /srv/cc-ci-orch && sudo -H -u loops nix flake update --dry-run 2>&1 | grep -v "^warning" | tail -8' # informational; the unit does the real update
+git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12

-```
-nixos-version; readlink /run/current-system; uname -r
-systemctl --failed --no-legend
-df -h / | tail -1
-cd /srv/cc-ci-orch && nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
-tmux ls
-```
-Record: current nixpkgs rev, generation, failed units (should be 0), disk, live tmux sessions.
+# 2. run the unit and follow it (10-25 min: nixpkgs bumps rebuild the two OCI images too)
+ssh cc-ci 'systemctl start --no-block cc-ci-auto-update; journalctl -fu cc-ci-auto-update'   # Ctrl-C when it prints "auto-update committed" / "REVERT" / "BUSY"
+# or, non-interactively:
+ssh cc-ci 'systemctl start cc-ci-auto-update; systemctl is-active cc-ci-auto-update; journalctl -u cc-ci-auto-update --no-pager -n 40'

-### 2. Branch + bump the lock
-
-Work directly in `/srv/cc-ci-orch` on a branch (it's the live checkout; the flake edit is inert
-until a rebuild):
-```
-cd /srv/cc-ci-orch
-git checkout -b chore/orchestrator-flake-update-$(date -u +%Y%m%d)
-nix flake update 2>&1 | tail -5      # nixpkgs tracks nixos-26.05; sops-nix follows nixpkgs
-nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
-```
-Note old→new revs. If the lock didn't move, report "already at channel tip" and stop (nothing to
-deploy).
-
-### 3. Build (catch errors before any activation)
-
-```
-cd /srv/cc-ci-orch && nixos-rebuild build --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -15
-readlink -f result
-```
-Build failure → fix on the branch (option renames etc.) before going further. Never activate a
-build that didn't build cleanly.
-
-### 4. Test-activate (MANDATORY before switch — general policy)
-
-`nixos-rebuild test` activates without touching the bootloader or system profile, so if the
-activation breaks the host (cf. the cc-ci server's 2026-08-03 no-default-route outage), a plain
-reboot — Hetzner API power-cycle on server **134487234** if SSH is gone (see
-`hetzner-server-recovery`) — lands back on the last-known-good generation.
-```
-cd /srv/cc-ci-orch && setsid nohup nixos-rebuild test --flake .#cc-ci-orchestrator-hetzner \
-  > /tmp/orchestrator-test-switch.log 2>&1 < /dev/null & echo launched
-# after it settles (poll; tailscaled/sshd may blip):
-readlink /run/current-system            # should be the new store path
-ip route show default                   # default route survived
-systemctl --failed --no-legend          # 0 failed
-tmux ls                                 # sessions survived
-tail -5 /tmp/orchestrator-test-switch.log
-```
-Regression here → you're still on the old boot default: diagnose, fix, or reboot to revert. Do NOT
-switch.
-
-### 5. Switch (make permanent — only after 4 is healthy)
-
-```
-cd /srv/cc-ci-orch && nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner 2>&1 | tail -10
-```
-(If it fails with "Unit nixos-rebuild-switch-to-configuration.service was already loaded", the
-detached test's transient unit is still running — wait or `systemctl stop` it, then retry.)
-
-### 6. Health gate (do not skip)
-
-```
-nixos-version; readlink /run/current-system
-systemctl --failed --no-legend
-tmux ls                                          # orchestrator + loop sessions intact
-tailscale status | head -3                       # tailnet up
-systemctl list-timers cc-ci-upgrade-all.timer --no-pager | head -3
-ssh cc-ci 'hostname' 2>&1                        # can still reach the CI server
-df -h / | tail -1
-```
-Optional (operator's call — it ends this session): a cold-boot proof reboot. If done, the
-REBOOTS.md auto-launch flow restarts the loops; the operator reconnects the orchestrator.
-
-### 7. Commit + PR + merge + report
-
-```
-cd /srv/cc-ci-orch
-git add flake.lock                                # flake.nix too if the channel ref changed
-git commit -m "flake: bump nixpkgs (nixos-26.05, $(date -u +%Y-%m-%d))
-
-nixpkgs: <old-rev[:8]> -> <new-rev[:8]> (nixos-26.05 tip)
-Deployed to cc-ci-orchestrator-hetzner: build + test + switch + health gate green."
-git push -u origin HEAD
-```
-Open the PR on `recipe-maintainers/cc-ci-orchestrator` (Gitea API with the `GITEA_*` creds from
-`/srv/cc-ci/.testenv`, same as `open-cc-ci-pr.sh` does for cc-ci — the helper is hardcoded to the
-cc-ci repo, so use the API directly or a copy with the repo swapped). PR body: old→new revs, what
-was verified. Then **merge it directly** — the PR is for visibility/historical record; the skill
-invocation is the authorization. On a rolled-back/failed run, leave the PR open with an
-explanatory comment.
-
-Back on the host, return the checkout to main so the live tree matches the merged state:
-```
-git checkout main && git pull
+# 3. read the outcome
+ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; readlink /run/current-system; readlink /nix/var/nix/profiles/system; systemctl --failed --no-legend; cd /srv/cc-ci-orch && git log --oneline -1 && git status --short'
 ```

-Write `/srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-<YYYY-MM-DD>.md`
-(`RESULT: SUCCESS/ROLLED-BACK/FAILED …`), append a line to `cc-ci-plan/JOURNAL.md`, and report to
-the operator with the **merged PR link + a summary of what changed**.
+`--cc-ci-only` (after merging something in the cc-ci repo that the host should run now):
+```bash
+ssh cc-ci 'sudo -H -u loops bash -lc "cd /srv/cc-ci-orch && nix flake update cc-ci && git -c user.name=auto-update -c user.email=auto-update@cc-ci commit -qm \"flake.lock: cc-ci -> main\" -- flake.lock && git push -q origin HEAD:main"'
+ssh cc-ci 'git -C /etc/cc-ci pull --ff-only --recurse-submodules'
+ssh cc-ci 'systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch bash -lc "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci && nixos-rebuild switch --flake /srv/cc-ci-orch#cc-ci"'
+# then the same step-3 checks, plus the §6 health commands from /cc-ci-status
+```
+
+## When it fails
+
+`result=failed note=<step>` means the unit already re-activated the previous generation and
+restored `flake.lock`; the host is as it was. Read `journalctl -u cc-ci-auto-update` for the
+`HEALTH FAIL:` / `REVERT:` line:
+- **build failed** — usually an upstream nixpkgs breakage or a cc-ci main that does not evaluate
+  against the new nixpkgs. Reproduce: `sudo -H -u loops nixos-rebuild build --flake /srv/cc-ci-orch#cc-ci`
+  in the checkout after `nix flake update`; fix in the right repo (PR, merge), re-run.
+- **swarm services not at their replica count** — one of the reconcile oneshots did not converge
+  in 10 min: `journalctl -u deploy-<proxy|drone|bridge|dashboard|reports>`, `docker service ps <svc>`.
+- **NAR hash mismatch** on the cc-ci input — the lock was made on another machine; `nix flake
+  update cc-ci` on the host and re-run (README day-2).
+- **push failed** — bookkeeping only (the generation is committed): push the lock by hand as loops.
+- A hung run: the unit has a 3 h timeout; `systemctl status cc-ci-auto-update`. Do NOT restart the
+  unit or the host mid-switch (see the `stopIfChanged` comment in the module).
+
+## Report (end of run)
+
+- old generation → new generation, the nixpkgs/sops-nix/cc-ci revs before → after, the lock commit
+  on `main`, the health check result, and anything the journal warned about.
+- Then `/cc-ci-status` §5–§6 should read `ok` — say if it does not.

 ## Guardrails

- **Build before test, test before switch** — never `switch` without a healthy `test` activation
-  (general policy for this host and the cc-ci server; `test` keeps the bootloader untouched so a
-  reboot always recovers).
- **Detach activations** (`setsid nohup … &`) — a dropped SSH/Remote-Control connection must not
-  kill a half-applied activation.
- **Self-update awareness:** announce before any reboot (it ends the operator's session); a
-  switch is fine but expect brief tailscaled/sshd blips.
- **Single-writer:** dedicated branch; don't sweep unrelated working-tree changes into the bump
-  commit; return the live checkout to `main` after the merge.
- **Stable channel only** (`nixos-26.05` until the operator chooses a jump).
- **PRs for visibility, merged directly** once verified; failed runs leave the PR open. Always
-  report merged-PR links + change summaries.
- **Rescue path:** Hetzner server **134487234** — same rescue-mode procedure as
-  `hetzner-server-recovery` (grub generations live in a submenu: `1>N` ids).
+- The unit is the only thing that `switch`es here; do not hand-roll `nixos-rebuild switch` unless
+  `--cc-ci-only` says so, and then `test` first.
+- Never `nix flake update` from the notplants box and push a lock for this host (NAR hashes differ
+  between Nix versions) — locks for the host are made on the host.
+- Nothing recipe-side: no PRs merged, no tests touched; `/cc-ci-tests-update` is separate.
@@ -1,298 +1,25 @@
 ---
 name: cc-ci-server-update
-description: Update the cc-ci CI server host to the latest pinned nixpkgs/sops-nix flake inputs, then rebuild + verify the live server. Bumps the cc-ci repo's flake inputs (nixpkgs + sops-nix, currently a hard rev pin — moves deliberately, not drift), commits the flake.lock change on a branch, opens a cc-ci PR, then deploys it to the live cc-ci host via nixos-rebuild switch (the operator's choice to invoke this skill IS the authorization — no mid-run check needed), and verifies the server is healthy after (no failed units, infra services up, endpoints 200, disk free). NEVER pushes upstream recipe main. Opens the cc-ci PR for visibility/historical record and merges it directly once the deploy verifies (the skill invocation IS the authorization); the end-of-run report includes the merged PR link + a summary of what changed. The manual trigger for what should ideally be an automatic periodic nixpkgs bump. Invoke as /cc-ci-server-update.
+description: (Delegates) Update the cc-ci CI server host to the latest nixpkgs — which, since 2026-09-07, is the SAME machine as the orchestrator, built from the cc-ci-orchestrator flake. This skill only forwards to /cc-ci-orchestrator-update; it no longer bumps or rebuilds the cc-ci repo's own flake, because that standalone `#cc-ci` target would replace the combined host config. Invoke as /cc-ci-server-update.
 ---

-# cc-ci-server-update
+# cc-ci-server-update → `/cc-ci-orchestrator-update`

-Manually update the **cc-ci CI server host**'s NixOS dependencies (nixpkgs + sops-nix) to their
-latest pinned revisions, ship the bump via a verified cc-ci PR + a live `nixos-rebuild switch`, and
-confirm the server is healthy afterward. This is the cc-ci-server analogue of a `nix flake update`
-+ rebuild — done deliberately, with a reviewable PR and a health gate, not as silent drift.
+There is no separate CI-server host any more. The CI server runs on the cc-ci host
+(195.201.88.249) together with the orchestrator, from **one** flake:
+`recipe-maintainers/cc-ci-orchestrator` `#cc-ci`, which imports the cc-ci repo's
+`nixosModules.cc-ci-server` and makes that input follow its own nixpkgs and sops-nix.

-> **Two hosts, two flakes — don't confuse them.** This skill updates the **cc-ci server** flake
-> (`ssh cc-ci`, repo `recipe-maintainers/cc-ci`, host target `.#cc-ci` = `nix/hosts/cc-ci-hetzner/`).
-> The **orchestrator** host (`/srv/cc-ci-orch`, flake host `.#cc-ci-orchestrator-hetzner`) is a
-> different machine with its own flake — it is NOT touched here. If you want to update the
-> orchestrator host, that's a separate `nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner`
-> from `/srv/cc-ci-orch` (the orchestrator flake already tracks `nixos-26.05` and is rebuilt by the
-> operator). This skill is only about the cc-ci **server**.
+**Run `/cc-ci-orchestrator-update`.** It updates nixpkgs, sops-nix AND the cc-ci input, tests,
+health-checks (the CI server's swarm, front doors, sops, acme-dns — the checks this skill used to
+do) and switches, via the host's own weekly auto-update unit.

-## What this skill does NOT do
+Do **not** do what the old version of this skill did:
+- do not bump `flake.lock` in the cc-ci repo expecting it to change the live host (it cannot — the
+  host follows the orchestrator flake's inputs; cc-ci's lock only matters for the standalone
+  `#cc-ci` target, which nothing deploys);
+- do not run `nixos-rebuild switch --flake /etc/cc-ci#cc-ci` or `...#cc-ci-hetzner` on the host —
+  that is the standalone-server configuration and would take the orchestrator half down.

- **Does NOT bump recipe image tags** — that's `/upgrade-all` / `/recipe-upgrade`.
- **Does NOT touch cc-ci tests / harness / runner** — that's `/cc-ci-tests-update` or
-  `/ci-dev-workflow`. This is a flake-input bump only (`flake.nix` input revs + `flake.lock`).
- **Does NOT update the orchestrator host** — only the cc-ci server.
- **Does merge the cc-ci PR itself** — the PR exists for visibility and historical record, and is
-  merged directly at the end of a verified run (the skill invocation is the authorization; the
-  operator reviews the merged PR afterwards via the link in the report).
-
-## Preconditions / access
-
- `ssh cc-ci` works (root, tailnet `100.95.31.88`). If it doesn't, fix access first (Tailscale ACL
-  / `cc-ci-tailscaled` proxy) — do NOT proceed.
- **The operator's choice to invoke this skill IS the authorization for the live rebuild** — there
-  is no mid-run operator check to wait for. A `nixos-rebuild switch` cycles the abra reconcile
-  oneshots (`deploy-proxy`, `warm-keycloak`, …), so prefer a **quiescent moment** (no `!testme` /
-  nightly sweep running) to avoid disrupting an in-flight CI run. Check first:
-  ```
-  ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"'   # any unhealthy task?
-  ssh cc-ci 'pgrep -af run_recipe_ci | grep -v grep'                                 # a CI run in flight?
-  ```
-  If either is non-empty, WAIT until the host is quiescent — do not rebuild mid-CI.
- `/srv/cc-ci/.testenv` has the `GITEA_*` creds (for opening the cc-ci PR).
- The **operator-held sops secrets** are on the cc-ci host at `/etc/cc-ci/secrets/secrets.yaml`
-  (NOT in git). The rebuild needs them — see the rebuild procedure below.
-
-## Procedure
-
-### 1. Snapshot current state (so you can compare + roll back)
-
-```
-ssh cc-ci 'hostname; uptime; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
-  deploy-bridge deploy-reports deploy-backupbot drone-runner-exec; systemctl --failed --no-legend; \
-  df -h / | tail -1; nixos-version; readlink /run/current-system'
-```
-Record: the current `nixos-version`, the current-system store path, the list of active infra
-services, and the disk usage. These are your "before" baseline. Any rebuild that leaves a unit
-failed or an endpoint down gets compared against this.
-
-Also record the **current flake input revs** (the "before" of what you're bumping):
-```
-ssh cc-ci 'cd /root/builder-clone && nix flake metadata --json 2>/dev/null | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\")\""' 2>&1 | head
-```
-
-### 2. Prepare a working clone of the cc-ci repo (single-writer)
-
-Work in a **separate clone on the orchestrator** — never push `main`, never touch the build loops'
-working clones. Use the existing `/home/loops/work/cc-ci-fix` clone if present and clean, or make a
-fresh one:
-```
-git clone ssh://git@git.autonomic.zone:2222/recipe-maintainers/cc-ci.git /home/loops/work/cc-ci-server-update
-cd /home/loops/work/cc-ci-server-update
-git fetch origin && git checkout main && git reset --hard origin/main
-git checkout -b chore/server-flake-update-$(date -u +%Y%m%d)
-```
-(If `/home/loops/work/cc-ci-fix` already exists and is on a stale branch, either reuse it on a new
-branch or make the fresh clone above — the key rule is the branch is dedicated + off `main`.)
-
-### 3. Bump the flake inputs (deliberate, not drift)
-
-The cc-ci server flake **intentionally hard-pins** `nixpkgs` to a specific revision (the comment in
-`flake.nix` says "Bump deliberately, not drift") and `sops-nix` to a commit compatible with that
-nixpkgs. So the bump is a **considered move to a newer nixpkgs + a compatible sops-nix**, not a
-blind `nix flake update --relock-all`.
-
-**(a) Decide the new nixpkgs.** The cc-ci server runs recipe CI; nixpkgs provides docker, abra's
-runtime, traefik, keycloak, drone, the harness python/pytest/playwright toolchain. Pick the newest
-**stable channel revision** that's compatible:
- Default: move to the latest revision of the **same channel** the orchestrator host tracks
-  (`nixos-26.05`), so both hosts stay on a shared nixpkgs and CVEs get patched. Look up the current
-  tip: `git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05` (the SHA after the ref).
- If a newer stable channel is out and the operator wants it, use that — but treat a **channel
-  jump** (e.g. 26.05 → 26.11) as a bigger change: scan the release notes for breaking changes
-  (docker, systemd, python, nginx module option renames) and surface them in the PR body. Default
-  to staying on the same channel unless the operator asks for the jump.
- **Never** jump to `nixos-unstable` for the cc-ci server — recipe CI reproducibility depends on a
-  stable nixpkgs. Unstable is for experiments, not production.
-
-**(b) Update `flake.nix`** with the new nixpkgs rev (replace the `rev = …` in the `nixpkgs.url`),
-and check whether `sops-nix` needs a matching bump:
- sops-nix `inputs.nixpkgs.follows = "nixpkgs"`, so it tracks nixpkgs automatically — but its own
-  rev pin may need moving if the pinned sops-nix commit is incompatible with the new nixpkgs (e.g.
-  the `buildGoModule` → `buildGo125Module` gotcha the current pin's comment calls out). After
-  editing `flake.nix`, run `nix flake lock --update-input nixpkgs --update-input sops-nix` (or
-  `nix flake update` if both follow) in the clone and let nix resolve a compatible sops-nix. If the
-  lock fails (sops-nix master needs a newer nixpkgs than you picked), pick a sops-nix commit that
-  still builds against your chosen nixpkgs (its `flake.nix` declares the nixpkgs it expects).
- **Keep the `sops-nix.inputs.nixpkgs.follows = "nixpkgs"` line** — that's what keeps them aligned.
- Update the comment in `flake.nix` to reflect the new pin date + reason (the existing comment
-  style: "Pinned to … (operator YYYY-MM-DD)" or "Follows nixos-26.05 channel").
-
-**(c) Re-lock + sanity-build locally first** (cheap, catches a broken input before touching the
-host):
-```
-cd /home/loops/work/cc-ci-server-update   # (or cc-ci-fix on the new branch)
-nix flake lock --update-input nixpkgs --update-input sops-nix   # rewrites flake.lock
-nix flake metadata --json | jq -r '.locks.nodes | to_entries[] | "\(.key) \(.value.locked.rev // "-")"'
-git diff flake.nix flake.lock | head -40
-```
-Verify the lock moved to the revs you intended. **Do NOT `nixos-rebuild build` from the
-orchestrator** — the orchestrator's nixpkgs is different and the cc-ci host target won't build here
-(meaningfully). The real build happens on the cc-ci host in step 5.
-
-### 4. Commit + open the cc-ci PR (merged in step 7 after the health gate)
-
-```
-git add flake.nix flake.lock
-git commit -m "flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))
-
-nixpkgs: <old-rev[:8]> → <new-rev[:8]> (nixos-<channel>)
-sops-nix: <old-rev[:8]> → <new-rev[:8]>
-<one-line reason — CVE patch / channel bump / sops-nix compat>"
-git push -u origin HEAD
-```
-Open the PR via the helper:
-```
-set -a; . /srv/cc-ci/.testenv; set +a
-TITLE="flake: bump nixpkgs + sops-nix ($(date -u +%Y-%m-%d))" \
-BODY_FILE=/tmp/cc-ci-server-update-pr-body.md \
-bash /srv/cc-ci-orch/.claude/skills/ci-test-review/open-cc-ci-pr.sh
-```
-(Write the PR body to `/tmp/cc-ci-server-update-pr-body.md` first: old→new revs, the channel, any
-breaking changes from the release notes, and the rebuild plan. The PR is opened for visibility and
-historical record; it gets merged directly in step 7 once the deploy passes the health gate.)
-
-Capture the `PR_URL`. **The cc-ci push-build lint gate will run on the branch** — if it's red on a
-flake-lock-only change, it's a pre-existing lint drift (not yours); note it in the PR body and
-proceed (the rebuild doesn't depend on the lint gate passing).
-
-### 5. Deploy to the live cc-ci host (no-CI window)
-
-This is the host-rebuild procedure from memory `cc-ci-host-rebuild-procedure` — the cc-ci server
-has **no self-service rebuild path**, so the orchestrator stages the repo + rebuilds it. The
-deploy clone on the host is `/root/builder-clone`; staging a fresh copy avoids disturbing it.
-
-**(a) Stage the PR branch on the host:**
-```
-# from the orchestrator, on the PR branch, clean tree:
-rsync -a --delete --exclude='.git' /home/loops/work/cc-ci-server-update/ root@cc-ci:/root/cc-ci-deploy/
-ssh cc-ci 'chown -R root:root /root/cc-ci-deploy'
-```
-(A `--delete` rsync from a clean tree = exactly the PR head. Excluding `.git` keeps it a plain path
-flake — see (c).)
-
-**(b) Copy the operator-held sops secrets** (NOT in git, required for the build):
-```
-ssh cc-ci 'mkdir -p /root/cc-ci-deploy/secrets && cp /etc/cc-ci/secrets/secrets.yaml /root/cc-ci-deploy/secrets/secrets.yaml && chmod 600 /root/cc-ci-deploy/secrets/secrets.yaml'
-```
-(The age key is at `/var/lib/sops-nix/key.txt` on the host — already in place from prior rebuilds.
-Without `secrets/secrets.yaml` the build FATAs `secrets/secrets.yaml does not exist`.)
-
-**(c) Build first (catch errors before switching):**
-```
-ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild build --flake .#cc-ci' 2>&1 | tail -30
-```
- Target is `.#cc-ci` (= `.#cc-ci-hetzner` = `nix/hosts/cc-ci-hetzner/configuration.nix`).
- nixpkgs is now your bumped pin, so this **will** rebuild more than a same-pin no-op — expect a
-  longer build (docker/traefik/keycloak/python toolchain may all rebuild against the new nixpkgs).
-  That's expected; this is a real nixpkgs bump.
- If the build fails: read the error, fix on the PR branch (e.g. a renamed option, a removed
-  package), re-push, re-stage, re-build. Do NOT `switch` a failed build. Common breakages: NixOS
-  module option renames across channels (check the release notes), python package removals, a
-  sops-nix incompatibility (go back to step 3b and pick a compatible sops-nix rev).
-
-**(d) Test-activate FIRST (general policy for nix deploys to this server AND the orchestrator
-host): `nixos-rebuild test` before any `switch`:**
-```
-# detach it so a dropped SSH can't kill the activation mid-flight:
-ssh cc-ci 'cd /root/cc-ci-deploy && setsid nohup nixos-rebuild test --flake .#cc-ci \
-  > /root/cc-ci-deploy/test-switch.log 2>&1 < /dev/null & echo launched'
-# poll until reachable again, then check the essentials survived:
-ssh cc-ci 'readlink /run/current-system; ip route show default; systemctl --failed --no-legend'
-```
-`test` activates the new configuration **without touching the bootloader or the system profile**,
-so if the activation kills networking (as the 2026-08-03 26.05 bump did — no default route), a
-plain reboot (Hetzner API power-cycle if SSH is gone) lands back on the last-known-good
-generation. Only proceed to `switch` once the test activation is up, reachable, and healthy.
-Note: the detached test leaves a transient `nixos-rebuild-switch-to-configuration` unit; if the
-follow-up `switch` fails with "Unit ... was already loaded", wait for it to finish or
-`systemctl stop nixos-rebuild-switch-to-configuration.service` first.
-
-**(e) Switch (make it permanent — only after (d) verified):**
-```
-ssh cc-ci 'cd /root/cc-ci-deploy && nixos-rebuild switch --flake .#cc-ci' 2>&1 | tail -30
-```
-This is the moment the server moves to the new nixpkgs. The abra reconcile oneshots
-(`deploy-proxy`, `warm-keycloak`, …) re-run as part of the switch — watch the tail for any that
-fail. If a reconcile unit fails, do NOT roll back blindly — diagnose first (a warm-keycloak blip
-may be transient; a proxy failure is real).
-
-### 6. Verify the server is healthy (health gate — do not skip)
-
-```
-ssh cc-ci 'echo "=== version ==="; nixos-version; readlink /run/current-system; \
-  echo "=== failed units ==="; systemctl --failed --no-legend; \
-  echo "=== infra services ==="; systemctl is-active deploy-proxy deploy-dashboard deploy-drone \
-    deploy-bridge deploy-reports deploy-backupbot drone-runner-exec | paste - - - - - - -; \
-  echo "=== docker stacks ==="; docker stack ls --format "{{.Name}} {{.Services}}"; \
-  echo "=== service replicas (any 0/N?) ==="; docker service ls --format "{{.Name}} {{.Replicas}}" | grep -c "0/"; \
-  echo "=== endpoints ==="; curl -sS -o /dev/null -w "traefik:%{http_code}\n" https://traefik.ci.commoninternet.net/api/version; \
-  curl -sS -o /dev/null -w "dashboard:%{http_code}\n" https://ci.commoninternet.net/; \
-  echo "=== disk ==="; df -h / | tail -1'
-```
-Compare against the step-1 baseline:
- `nixos-version` moved (or the current-system path changed) — the bump landed.
- **No failed units** (the `--failed` output is empty).
- All 7 infra services `active` (or `active (exited)` for the oneshots).
- `docker service ls` shows no `0/N` replicas (everything converged).
- Endpoints `200` (traefik `/api/version`, dashboard root).
- Disk didn't spike (a nixpkgs bump adds a new system generation; old ones are GC'd eventually).
-
-If **anything regressed**: diagnose. A real regression → roll back to the previous generation:
-`ssh cc-ci 'nixos-rebuild switch --rollback'` (this switches to the pre-bump system path; the
-flake input change on the PR branch is still your source of truth, but the running host reverts
-while you fix). Report the rollback in the PR.
-
-### 7. Merge the PR + report
-
-On a **verified** run (health gate green): post a PR comment summarizing what was deployed (revs,
-any incidents, verification results), then **merge the PR directly** — the skill invocation is the
-authorization; the PR exists for visibility and historical record. On a rolled-back or failed run,
-leave the PR **open** with a comment explaining the state.
-
-Print + record a one-line result + the PR URL. Write a short note to
-`/srv/cc-ci/.cc-ci-logs/server-update-<YYYY-MM-DD>.md`:
- `RESULT: SUCCESS — cc-ci server rebuilt on nixpkgs <new-rev[:8]>, no failed units, endpoints 200, PR merged: <url>`
- `RESULT: ROLLED-BACK — nixpkgs <new-rev> rebuild failed/ regressed (<reason>); host reverted to <old-rev>; PR <url> left open`
- `RESULT: FAILED — <step>: <reason>`
-
-The end-of-run report to the operator MUST include **links to every merged PR** plus a summary of
-what changed in each, so the operator can review them after the fact if they want to.
-
-### 8. Optional cold-boot proof
-
-A `nixos-rebuild switch` proves the new generation works on the running kernel. A **reboot** proves
-it boots cleanly from cold — the stronger guarantee (cf. the pxgate cold-boot proof). Only do this
-if the operator wants it, and only outside a CI window:
-```
-ssh cc-ci 'systemctl reboot'
-# wait ~60s, re-check ssh cc-ci + the step-6 health gate
-```
-Record the new `boot_id` if you do this (it's a genuine reboot — log it to
-`cc-ci-plan/REBOOTS.md`? No — that file is for the **orchestrator** host. The cc-ci server's reboots
-are tracked in its own journal; a note in the server-update report is enough).
-
-## Guardrails
-
- **Live rebuild is authorized by invocation.** The operator's choice to run this skill IS the
-  go-ahead for the `nixos-rebuild switch` — do not stop mid-run to re-confirm. A switch cycles
-  reconcile oneshots and can disrupt an in-flight `!testme`, so prefer a quiescent moment (the
-  step-1 check), but don't gate on a second authorization.
- **Single-writer:** dedicated branch in a separate clone; never push `main`; never touch the
-  build loops' `/cc-ci` / `/cc-ci-adv` clones or `/root/builder-clone` (stage to
-  `/root/cc-ci-deploy` instead).
- **Open the PR for visibility, merge it directly.** The live rebuild is the deploy; the PR is the
-  reviewable historical record. Merge only after the health gate passes; a rolled-back/failed run
-  leaves the PR open with an explanatory comment. Always report merged-PR links + change summaries.
- **Build before test, test before switch.** Never `nixos-rebuild switch` a build that didn't build
-  cleanly first, and never `switch` without a healthy `nixos-rebuild test` activation first — `test`
-  leaves the bootloader untouched, so a reboot always recovers to the last-known-good generation.
-  This is general policy for nix deploys to the cc-ci server **and** the orchestrator host.
- **Health gate is mandatory.** No "it switched, ship it" — run the step-6 verification and compare
-  to the baseline. Roll back on regression.
- **Stable channel only** for the cc-ci server (never `nixos-unstable`).
- **Keep sops-nix.follows** so the two inputs stay aligned; don't fork them.
- **Secrets never in git.** The operator-held `secrets/secrets.yaml` is copied on the host only.
- **Rollback is always available:** `nixos-rebuild switch --rollback` reverts to the prior
-  generation without touching the flake input revs.
-
-## Relationship to an automatic update
-
-This skill is the **manual trigger** for what should ideally run on a periodic timer: bump the cc-ci
-flake inputs, build, switch, verify. If/when an automatic `cc-ci-server-update.timer` is added
-(NixOS systemd timer on the orchestrator, analog of `cc-ci-upgrade-all.timer`), it should invoke
-this skill via a dedicated one-shot agent (the way `cc-ci-upgrader` runs `/upgrade-all`). The
-skill stays as the authoritative procedure either way — automatic just means "run on a schedule
-without a human prompt."
+If what you actually wanted is "the host should run cc-ci `main` as of now" (a harness or test
+change), that is `/cc-ci-orchestrator-update --cc-ci-only`.
@@ -58,35 +58,62 @@ For each open PR: age (now − created_at), and whether the PR title/body/report
 - Verified-green PRs awaiting operator merge are normal — list them as "ready to merge", not
  as failures.

-### 5. Server + orchestrator host update recency
+### 5. The cc-ci host: update recency + the weekly auto-update

-For BOTH hosts:
-```
-# cc-ci server:      ssh cc-ci 'nixos-version; cd /root/cc-ci-deploy && nix flake metadata --json' (or builder-clone)
-# orchestrator:      nixos-version; cd /srv/cc-ci-orch && nix flake metadata --json
-git ls-remote https://github.com/NixOS/nixpkgs <channel>   # current tip
-ls -t /srv/cc-ci-orch/.cc-ci-logs/server-update-*.md /srv/cc-ci-orch/.cc-ci-logs/orchestrator-update-*.md | head -2
-```
- Report: days since last update log + how far the running nixpkgs rev lags the channel tip.
- **Lagging >30 days** (or a NixOS release behind) → recommend `/cc-ci-server-update` /
-  `/cc-ci-orchestrator-update`.
-
-### 6. Host + service health (both machines)
+Since 2026-09-07 the CI server and the orchestrator are ONE Hetzner host (`cc-ci`,
+195.201.88.249, no tailnet), built from `recipe-maintainers/cc-ci-orchestrator`'s
+`nixosConfigurations.cc-ci` (which imports cc-ci's `nixosModules.cc-ci-server`). From the
+notplants-orchestrator box `ssh cc-ci` reaches it as root; on the host itself `ssh cc-ci` is loopback.

 ```
-ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; docker service ls --format "{{.Name}} {{.Replicas}}"'
-systemctl --failed --no-legend; df -h / | tail -1; tmux ls
+ssh cc-ci 'nixos-version; readlink /run/current-system; readlink /nix/var/nix/profiles/system'
+ssh cc-ci 'cd /srv/cc-ci-orch && nix flake metadata --json | jq -r ".locks.nodes | to_entries[] | \"\(.key) \(.value.locked.rev // \"-\") \(.value.locked.lastModified // \"-\")\""'
+git ls-remote https://github.com/NixOS/nixpkgs nixos-26.05 | cut -c1-12   # channel tip
+# the weekly self-update (nix/modules/auto-update.nix, Tue 03:00 UTC, busy-gated):
+ssh cc-ci 'cat /srv/cc-ci/.cc-ci-logs/auto-update-state; systemctl list-timers --no-pager cc-ci-auto-update.timer | sed -n 2p; journalctl -u cc-ci-auto-update --no-pager -n 3'
 ```
- Failed units, core swarm services not 1/1 (warm-* spares flapping is a known benign pattern —
-  note, don't page), disk **>65% (server)** / >85% (orchestrator) → findings. Server unreachable →
-  HIGH: recommend `hetzner-server-recovery`.
-  > **65%, not 80%, on the server — it is not a steady-state measure.** Two concurrent recipe runs
-  > pull images and write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run.
-  > That is exactly what happened on 2026-08-11: 63GB of unused images had accumulated (nothing ever
-  > pruned them), the filesystem filled during a run, and the harness died at startup with
-  > `OSError: [Errno 28] No space left on device`. Remedy: `docker image prune -af` on cc-ci — it
-  > spares anything a container references, so infra and warm-* canonicals are untouched. Do NOT
-  > `docker volume prune`: warm-* canonical volumes are data-warm and legitimately dangling.
+- The state file is one line: `<UTC time> result=ok|unchanged|skipped|failed generation=<store path> note=<why>`.
+  - `ok` / `unchanged` within the last 8 days → healthy.
+  - `skipped` (busy) two weeks in a row, `failed`, or no state line / timer inactive for >8 days →
+    finding: *"host auto-update <state> since <date>: <note>"* → recommend
+    `/cc-ci-orchestrator-update` (it runs the same unit by hand and reads its journal).
+  - `failed` also means the host is running the previous generation and `flake.lock` was restored —
+    say so; the journal (`journalctl -u cc-ci-auto-update`) names the health check that failed.
+- Running `/run/current-system` must equal the profile's boot generation (`system` link) — a
+  mismatch means a `test`-only activation was left behind (finding: switch or reboot decision).
+- nixpkgs rev lagging the `nixos-26.05` channel tip by >30 days → finding → `/cc-ci-orchestrator-update`.
+- `cc-ci` (the CI server input) lagging cc-ci `main` → note; the auto-update moves it weekly, or
+  `/cc-ci-orchestrator-update --cc-ci-only`.
+
+### 6. Host + service health (one machine)
+
+```
+ssh cc-ci 'systemctl --failed --no-legend; df -h / | tail -1; free -m | sed -n 2p'
+ssh cc-ci 'docker service ls --format "{{.Name}} {{.Replicas}}"; docker stack ls --format "{{.Name}}"'
+ssh cc-ci 'for h in ci report.ci drone.ci oc.ci; do printf "%s " $h; curl -s -o /dev/null -w "%{http_code} ssl=%{ssl_verify_result}\n" https://$h.commoninternet.net/; done'
+ssh cc-ci 'ls -s /run/secrets/test_secret; systemctl is-active acme-dns fail2ban nginx opencode-web drone-runner-exec; fail2ban-client status sshd | grep -E "Currently banned|Total banned"; fail2ban-client status nginx-http-auth | grep -E "Currently banned|Total banned"'
+ssh cc-ci 'systemctl list-timers --no-pager cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer cc-ci-auto-update.timer nightly-sweep.timer | tail -n +2 | head -4'
+ssh cc-ci 'sudo -iu loops tmux ls'      # cc-ci-orchestrator must exist (the opencode orchestrator agent); cc-ci-upgrader only during a run
+```
+- Failed units, swarm services not at their replica count (`warm-*` spares flapping is benign —
+  note, don't page), a front door not 200 (`oc.ci` must be **401**: basic auth in front of the
+  opencode UI; 404 there = the `ccci-opencode-ui` stack is gone, `systemctl restart opencode-ui-route`
+  and check the weekly sweep's keep-list), `ssl≠0` (cert), `/run/secrets` empty (sops), acme-dns /
+  fail2ban / nginx / opencode-web / drone-runner-exec inactive, any timer missing, the orchestrator
+  tmux session missing (`systemctl restart cc-ci-orchestrator`) → findings. Disk **>65%** → finding.
+  Host unreachable → HIGH: `hetzner-server-recovery` (token in `/srv/cc-ci/.hcloud-token` on the
+  notplants box while the operator keeps it active).
+  > **65%, not 80% — it is not a steady-state measure.** Two concurrent recipe runs pull images and
+  > write volumes worth tens of GB, so a host sitting at 73% still hits 100% mid-run (2026-08-11:
+  > 63GB of unused images, `OSError: [Errno 28] No space left on device`). Remedy: `docker image
+  > prune -af` — it spares anything a container references. Do NOT `docker volume prune`: warm-*
+  > canonical volumes are data-warm and legitimately dangling.
+- fail2ban: a non-zero "Currently banned" is information, not a finding (that is the jail
+  working); the notplants box's own address showing up banned IS a finding (it means a skill
+  misbehaved) — `fail2ban-client set <jail> unbanip <ip>`.
+- Secrets hygiene: `ssh cc-ci 'ls /secrets/files'` must list ONLY cc-ci.testenv, cc-ci-local-ed25519(.pub),
+  autonomic-bot-cc-ci-ed25519(.pub), opencode-auth.json. Anything else there (a tailscale key, a
+  master age key, another host's ssh key…) → finding: *"secret sprawl on the cc-ci host"*.
 - **!testme actually produces results** (the check that would have caught the above days earlier):
  the newest few `/var/lib/cc-ci-runs/<build>/` dirs must each contain `results.json`. A build that
  dies before the harness writes one leaves an EMPTY dir — and the PR badge still says "failure", so
@@ -143,14 +170,17 @@ minutes, no PRs). If it is instead that a known CVE is sitting unpatched, recomm
 `/cve-check` over waiting for the next weekly run whenever the question is "are we exposed?".

 `ALL HEALTHY` requires: recent successful weekly run + published report, no stale tests, no
-CVE PR open >14 days, both hosts <30 days behind their channel, zero failed units, recent builds all
-producing results.json, disk under
-thresholds, bridge clean, maintained-set consistent. Anything else is a finding — even minor
+CVE PR open >14 days, the host <30 days behind its channel with the weekly auto-update `ok`/`unchanged`
+within 8 days and running generation == boot generation, zero failed units, every front door
+answering (oc.ci with 401), sops decrypted, fail2ban up, the orchestrator agent session present,
+recent builds all producing results.json, disk under threshold, bridge clean, maintained-set
+consistent, secrets inventory exact. Anything else is a finding — even minor
 ones get a recommended next step. Order findings by priority (CVE/unreachable-host first).

 ## Guardrails

 - **Read-only.** This skill diagnoses and recommends; it invokes nothing and changes nothing.
- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else.
+- Use the pseudo-TTY wrap for any abra call; plain ssh for everything else. `ssh cc-ci` from the
+  notplants box is root on 195.201.88.249 (loops ssh config); there is no tailnet.
 - Don't double-count: a finding that explains another (bridge 401 → !testme "failures") gets
  reported once, at the root cause.
@@ -1,12 +1,11 @@
 ---
 name: cc-ci-update
-description: Run both cc-ci maintenance updates in one pass — first /cc-ci-server-update (bump the cc-ci server host's nixpkgs/sops-nix flake inputs, rebuild + verify the live server), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
+description: Run both cc-ci maintenance updates in one pass — first /cc-ci-orchestrator-update (the cc-ci host: nixpkgs, sops-nix and the cc-ci input, test → health check → switch, lock committed to main), then /cc-ci-tests-update (sweep all maintained recipes for stale cc-ci tests, update the stale ones, verify each via the standard PR + !testme -> CI flow). The combined "keep cc-ci itself current" skill: host deps + test suite, both done end-to-end with PRs opened for visibility and merged directly once verified (the skill invocation is the authorization), a healthy server at the end, and a report listing every merged PR link + a summary of what changed in each. Invoke as /cc-ci-update.
 ---

 # cc-ci-update

-The combined **keep cc-ci itself current** pass: run `/cc-ci-server-update` (host dependency bump +
-rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
+The combined **keep cc-ci itself current** pass: run `/cc-ci-orchestrator-update` (host dependency bump + health-checked rebuild) **then** `/cc-ci-tests-update` (stale-test sweep + fixes), in that order, end-to-end. This
 is just sequencing the two skills — it doesn't add or change any behavior. Read each sub-skill's
 SKILL.md for the full procedure; this file is the orchestration + the why-of-the-order.

@@ -21,7 +20,7 @@ SKILL.md for the full procedure; this file is the orchestration + the why-of-the
 ## Arguments

 None. (If you only want one of the two updates, invoke the sub-skill directly:
-`/cc-ci-server-update` or `/cc-ci-tests-update`.)
+`/cc-ci-orchestrator-update` or `/cc-ci-tests-update`.)

 ## Preconditions

@@ -34,12 +33,12 @@ Both sub-skills' preconditions hold:

 ## Procedure

-### 1. Run `/cc-ci-server-update` (host first)
+### 1. Run `/cc-ci-orchestrator-update` (host first)

-Run the full `/cc-ci-server-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
+Run the full `/cc-ci-orchestrator-update` procedure: snapshot baseline → bump flake inputs (nixpkgs +
 sops-nix) → commit + open cc-ci PR → stage on host → build → `nixos-rebuild switch` → **health
 gate** (no failed units, infra up, endpoints 200, disk free). See
-`.claude/skills/cc-ci-server-update/SKILL.md`.
+`.claude/skills/cc-ci-orchestrator-update/SKILL.md`.

 **Gate on the result before continuing:**
 - `SUCCESS` → the server is on the new nixpkgs, healthy. Proceed to step 2.
@@ -90,7 +89,7 @@ operator review.
 Inherited from both sub-skills (read their Guardrails sections). The combined-specific ones:
 - **Server-first, gated.** Don't run the tests sweep on a sick/rolled-back host. The server-update
  health gate is a hard go/no-go for step 2.
- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-server-update`.
+- **No-CI window for the rebuild** (the server half) — same as `/cc-ci-orchestrator-update`.
 - **Serialize the tests half on the shared Swarm** — same as `/cc-ci-tests-update`; don't run it
  concurrent with `/upgrade-all` or active build loops.
 - **Never merge anything** — server PR, test PRs, paired recipe upgrade PRs all operator-merged.
@@ -30,15 +30,19 @@ the orchestrator watches from outside.

 Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
 to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
-watchdog auto-resume the saved phase. The orchestrator session itself is NOT auto-started — the
-operator reconnects to it (that's why the startup notification matters). The orchestrator now runs on
-a **Hetzner `cpx22`** cloud server (`cc-ci-orchestrator-1`, tailnet `100.84.190.30`, public
-`168.119.126.100`, flake host `cc-ci-orchestrator-hetzner`) — see
-`cc-ci-plan/plan-orchestrator-hetzner-migration.md`. The earlier Pi→Incus-VM move is the historical
-`cc-ci-plan/plan-orchestrator-migration.md`. Rebuild this host with
-`nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch` — but **always
+watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
+`cc-ci-orchestrator.service` (`agents.py up orchestrator`) as an **opencode agent** the operator
+steers from https://oc.ci.commoninternet.net (no Claude on the cc-ci host; Claude sessions run on
+the notplants-orchestrator box and reach cc-ci over ssh). Since 2026-09 the orchestrator runs on the **same Hetzner
+host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
+`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
+repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
+The full provisioning + deploy guide is `README.md`; the move is recorded in
+`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
+shared with notplants) are in `archive/`. Rebuild this host with
+`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
 `nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
-before the `switch`** (general policy for nix deploys to this host and the cc-ci server: `test`
+before the `switch`** (general policy for nix deploys to this host: `test`
 leaves the bootloader and system profile untouched, so a reboot always recovers to the
 last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
 `.cc-ci-logs/server-update-2026-08-03.md`).
@@ -1,57 +1,414 @@
 # cc-ci-orchestrator

-Orchestrator workspace for building the **cc-ci** Co-op Cloud recipe CI server. The plan, launch
-tooling, and loop prompts live in [`cc-ci-plan/`](cc-ci-plan/); see [`AGENTS.md`](AGENTS.md) for the
-roles and operating model. Secrets (`.testenv`) are gitignored — never commit them.
+The **cc-ci orchestrator**: the agent loops that built the cc-ci Co-op Cloud recipe CI server,
+the operator's steering session, and the weekly autonomous recipe-upgrade run — plus the NixOS
+host they run on. Since 2026-09 that host is **the same Hetzner server as the CI server itself**:
+one `nixos-rebuild` from this repo builds both, because this flake imports the CI server as a
+module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) repo.

-## Run the orchestrator in tmux (survives disconnects + closing your laptop)
+| | where |
+|---|---|
+| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
+| The host contract those need (loops user, opencode CLI, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
+| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
+| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
+| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
+| Skills the orchestrator runs (`/upgrade-all`, `/recipe-upgrade`, `/cc-ci-status`, …) | `.claude/skills/`, `.opencode/skills/` |
+| How it used to be built (Pi → Incus VM → shared Hetzner box) | `archive/` |

-Keep this supervising session alive on the host with tmux, and use `--remote-control` so you can
-watch/steer it from **claude.ai/code** (or the mobile app).
+Secrets (`.testenv`, `upgrader.env`, `.sops/`, everything under `/secrets`) are gitignored — never
+commit them.
+
+---
+
+# Deploying a cc-ci host from scratch
+
+This is the whole path from "nothing" to a working CI server + orchestrator on one Hetzner
+server. It was last done on 2026-09-07 for `195.201.88.249` and is written so a person or an LLM
+can repeat it. Read it once before starting; the order matters.
+
+## 0. What you need in hand
+
+- A **Hetzner Cloud** project you can create servers in (console login or an API token).
+- **SSH keys**: yours, and the orchestrator's own key so the automation can reach the box. The
+  public keys that get root are tracked in `nix/hosts/cc-ci/ssh-keys` (one per line). Password
+  login is disabled and fail2ban watches sshd, so a key is the only way in.
+- Read access to `recipe-maintainers/cc-ci`, `recipe-maintainers/cc-ci-orchestrator` (both public
+  read) and the **private** `recipe-maintainers/cc-ci-secrets` (the `autonomic-bot` account has
+  it; the host gets its own deploy key on that account, §4b/§4c — and the bot password, in
+  `.testenv`, is what registers it).
+- The out-of-band secrets listed in §4. If you are migrating, they come from the old host; if
+  you are starting fresh you create them (each row says how).
+- Control of the DNS zone (Gandi for `commoninternet.net`) for the cutover in §7.
+
+## 1. Provision the server on Hetzner (Debian image)
+
+In the Hetzner Cloud console (or with `hcloud server create`):
+
+| setting | value | why |
+|---|---|---|
+| Image | **Debian 13** (any recent Debian/Ubuntu works with nixos-infect) | it is replaced by NixOS in §2 |
+| Type | **x86**, **8 GB RAM**, 4 vCPU — e.g. `cpx32` (dedicated AMD) or `cx33`. **Never `cax*`** (ARM): the flakes are `x86_64-linux`. | swarm + recipe deploys + 3–6 agent sessions; 4 GB is too small |
+| Disk | the type's default 150+ GB NVMe | docker layers alone are ~60 GB after a few weeks |
+| Network | public **IPv4** required; IPv6 optional (leave enabled or not, NixOS config ignores it) | cc-ci serves 80/443 and DNS on 53 publicly |
+| SSH keys | add every key from `nix/hosts/cc-ci/ssh-keys` you want to log in with, at least the orchestrator's | nixos-infect carries `/root/.ssh/authorized_keys` over |
+| Name | `cc-ci` | becomes the hostname |
+| Firewall | if a Hetzner Cloud Firewall is attached it must allow **22/tcp, 80/tcp, 443/tcp, 53/tcp, 53/udp** in, and ICMP | the NixOS firewall is separate and is configured by the flake |
+
+Check you can log in: `ssh root@<ip> hostname`.
+
+## 2. Convert Debian → NixOS with nixos-infect
+
+[nixos-infect](https://github.com/elitak/nixos-infect) installs NixOS over the running Debian and
+reboots. Run it detached so the SSH session dropping does not kill it:

 ```bash
-# 0. Exit any running orchestrator session first — a conversation can't be resumed while it's live:
-#      /exit        (inside Claude)   or Ctrl-D
-
-# 1. Start a detachable tmux session on this host
-tmux new -s orchestrator
-
-# 2. Inside tmux, resume the orchestrator conversation WITH remote control:
-claude --resume autonomous-orchestrator \
-  --remote-control "autonomous-orchestrator" \
-  --dangerously-skip-permissions
-#    - If name-resume opens a picker instead of resuming directly, choose "autonomous-orchestrator".
-#    - Or resume by the stable session id (more deterministic in a fresh pane):
-#        claude --resume 34a80a99-b37e-4809-b8da-ccc9fafe785e \
-#          --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
-
-# 3. Detach — the process keeps running:  press Ctrl-b, then d
+ssh root@<ip> 'cat > /root/infect.sh <<"EOF"
+#!/usr/bin/env bash
+set -euo pipefail
+# Pinned nixos-infect revision (same one that built the previous cc-ci hosts).
+INFECT_SHA="40f62a680bb0e8f2f607d79abfaaecd99d59401c"
+export NIX_CHANNEL="nixos-26.05"   # must match the nixpkgs channel in flake.nix
+export PROVIDER="hetznercloud"     # GRUB + Hetzner networking
+export NIXOS_IMPORT=""             # the real config comes from the flake in §5
+# The Debian 13 cloud image mounts /tmp as tmpfs; nixos-infect makes a temporary swapfile
+# there and swapon fails with "Invalid argument". 8 GB RAM needs no extra swap: skip it.
+export NO_SWAP=true
+curl -fsSL "https://raw.githubusercontent.com/elitak/nixos-infect/${INFECT_SHA}/nixos-infect" | bash -x
+EOF
+chmod +x /root/infect.sh
+nohup /root/infect.sh > /var/log/nixos-infect.log 2>&1 &'
 ```

-**Reconnect later**
- On this host: `tmux attach -t orchestrator`
- From anywhere: **claude.ai/code** → the `autonomous-orchestrator` session
+It downloads Nix, builds a NixOS system (5–10 min; follow with
+`ssh root@<ip> tail -f /var/log/nixos-infect.log`), then reboots. The SSH host key changes:
+`ssh-keygen -R <ip>` and confirm `ssh root@<ip> nixos-version` prints a 26.05 version.

-**Why it survives:** tmux keeps the `claude` process alive across SSH disconnects and your laptop
-closing; remote-control runs *outbound* from this host to Anthropic, so it stays connected
-regardless of the viewer. After a host reboot, re-run steps 1–2.
+### 2a. What went wrong on 2026-09-07, and the fixes (Debian 13 image, NixOS 26.05)

-> Two different "names": `--resume <name|id>` selects the *conversation* to restore (shown in the
-> `/resume` picker); the `--remote-control "<name>"` value is only the web display label and resumes
-> nothing. Resuming reuses the same session id each time (stays `34a8…`) — don't pass
-> `--fork-session` unless you intend to branch a new conversation.
->
-> Already inside a live session and just want the web surface? Run `/remote-control` — no exit/resume.
+All three bit on the first attempt; the script above and §3 already include the fixes, this is
+so you recognise them if they come back in another form.
+
+1. **`swapon: /tmp/nixos-infect.XXXX.swp: Invalid argument`** right at the start, script exits.
+   The Debian 13 cloud image mounts `/tmp` as tmpfs and a swapfile cannot live there.
+   Fix: `NO_SWAP=true` (in the script above). An 8 GB box does not need the temporary swap.
+2. **The box never comes back after the reboot: it boots NixOS, but nearly every unit fails**
+   (`dbus`, `systemd-logind`, `sshd`, networking …) with
+   `Could not start dynamically linked executable: /usr/bin/dbus-daemon` in the journal.
+   nixos-infect leaves the old Debian root in place and relies on NixOS's first boot to move it
+   to `/old-root` (`/etc/NIXOS_LUSTRATE`). With NixOS 26.05's systemd-based initrd that
+   lustration did not happen, so Debian's `/etc/systemd/system/*.service` files shadowed the
+   NixOS units and started Debian binaries. Fix, from Hetzner **rescue mode**
+   (`enable_rescue` + `reset` in the API/console, ssh in, `mount /dev/sda1 /mnt/root`):
+   move everything except `nix`, `boot`, `swapfile`, `lost+found`, `var/log`, `var/empty`,
+   `etc/nixos`, `etc/resolv.conf`, `etc/NIXOS`, `etc/machine-id`, `etc/ssh/ssh_host_*`,
+   `root/.nix-*`, `root/.ssh` into `/mnt/root/old-root`, delete `etc/NIXOS_LUSTRATE`, unmount,
+   `disable_rescue`, `reset`. (`/old-root`, ~1 GB, can be deleted once the host is in service.)
+3. **Boots, units fine, but no network.** The generated `networking.nix` has
+   `defaultGateway = "172.31.1.1";` — a bare string. Since NixOS 25.05 that yields no default
+   route. Fix: `defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };` (this is what
+   `nix/hosts/cc-ci/networking.nix` carries). To apply it from rescue mode, chroot into the
+   mounted root and rebuild the boot entry — the nix sandbox cannot `pivot_root` inside a chroot,
+   so turn it off for that one build:
+   ```bash
+   for d in proc sys dev dev/pts; do mount --bind /$d /mnt/root/$d; done
+   mount -t tmpfs tmpfs /mnt/root/run; cp -L /etc/resolv.conf /mnt/root/etc/resolv.conf
+   chroot /mnt/root /nix/var/nix/profiles/system/sw/bin/bash -c '
+     export PATH=/nix/var/nix/profiles/system/sw/bin NIX_REMOTE= HOME=/root
+     export NIX_PATH=nixos-config=/etc/nixos/configuration.nix:nixpkgs=/root/.nix-defexpr/channels/nixos
+     ln -sfn /nix/var/nix/profiles/system /run/current-system
+     nixos-rebuild boot --option sandbox false'
+   ```
+   The `journalctl -D /mnt/root/var/log/journal -b 0` trick (reading the dead system's journal
+   from rescue mode) is what told these apart.
+
+> Rescue mode without a console: `POST /servers/<id>/actions/enable_rescue` with your ssh key
+> id, then `…/actions/reset`; afterwards `disable_rescue` **and check `rescue_enabled` is false
+> before** the next `reset`, or it boots the rescue image again. `scripts/recovery/hetzner.py`
+> wraps these (token in `/srv/cc-ci/.hcloud-token`).
+
+## 3. Capture the machine-specific config into this repo
+
+nixos-infect wrote `/etc/nixos/{hardware-configuration,networking,configuration}.nix`. Only the
+first two matter; the flake replaces `configuration.nix`.
+
+```bash
+scp root@<ip>:/etc/nixos/hardware-configuration.nix nix/hosts/cc-ci/hardware.nix
+scp root@<ip>:/etc/nixos/networking.nix            nix/hosts/cc-ci/networking.nix
+```
+
+Then in `nix/hosts/cc-ci/`:
+
+- `hardware.nix`: keep as generated (GRUB EFI with `efiInstallAsRemovable`, `/boot/efi` by UUID,
+  `/dev/sda1` root). Do not copy another host's file — the UUIDs are per machine.
+- `networking.nix`: keep the static IPv4 + Hetzner gateway `172.31.1.1`. Make sure
+  `networking.defaultGateway` has **both** `address` and `interface = "eth0"` (§2a item 3). If
+  the generated IPv6 block has an empty address, delete the IPv6 parts; a real global address
+  (as on the 2026-09 box) can stay.
+- `configuration.nix`: set `cc-ci.publicIPv4` to the server's IPv4 and check `system.stateVersion`
+  is the release you installed (never change it later).
+- `ssh-keys`: the root keys.
+
+Commit on a branch; the rebuild in §5 can use the local checkout before the PR merges.
+
+## 4. Stage the workspace and secrets on the new host
+
+Everything in this section is **outside git**. Do it as root over SSH, in this order.
+
+### 4a. No tailscale
+
+The combined host is NOT on the tailnet (operator decision 2026-09-07): ssh is key-only on the
+public IP, the CI front doors are public via traefik, and the opencode UI is public on the same
+443 — traefik routes `oc.ci.commoninternet.net` to an nginx vhost on the docker bridge that
+enforces basic auth, via traefik labels on a tiny swarm relay stack (`ccci-opencode-ui`, on the weekly sweep's keep-list) that
+`opencode-ui-route.service` deploys. (Not via traefik's file *directory* provider: enabling
+that in the coop-cloud traefik recipe REPLACES its provider file, which holds the wildcard cert
+and the `security` middleware, and every front door goes down — it did, for two minutes, on
+2026-09-07.) fail2ban guards sshd and that login
+(`nix/hosts/cc-ci/configuration.nix`; the nginx jail bans in the DOCKER-USER chain because the
+traffic is docker-forwarded, not host INPUT).
+
+### 4b. The CI server's checkout and its one out-of-band secret
+
+```bash
+# the bot deploy key for the private cc-ci-secrets submodule — generated ON this host by loops
+# (§4c) and registered on the autonomic-bot Gitea account; root only points at it:
+install -d -m700 /root/.ssh
+cat > /root/.ssh/config <<'SSHCFG'
+Host git.autonomic.zone
+    Port 2222
+    User git
+    IdentityFile /secrets/files/autonomic-bot-cc-ci-ed25519
+    IdentitiesOnly yes
+SSHCFG
+# the deployed checkout: nightly-sweep runs from it, sops reads secrets/secrets.yaml from it
+git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci.git /etc/cc-ci
+# the age identity sops-nix decrypts with. FIRST deploy of a host that is not yet a recipient:
+# the master (recovery) key, temporarily. Once the host is a recipient (below): its own
+# ssh-host-key-derived identity, and the master key leaves the box.
+install -d -m0700 /secrets/host
+install -m600 <master-age.txt> /secrets/host/sops-age-key.txt
+```
+
+`/etc/cc-ci/secrets/secrets.yaml` is encrypted to the master key and the *previous* hosts' SSH
+host keys. The master key is enough for the FIRST deploy. Then make the new host a recipient so
+the master key can leave the box again (it belongs off-box, with the operator):
+`ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub` → add it to `.sops.yaml` in cc-ci-secrets AND in
+cc-ci, `sops updatekeys secrets.yaml` (needs the master key, so do it where that lives), push,
+bump the submodule in cc-ci, `git -C /etc/cc-ci pull --recurse-submodules` on the host, then
+replace `/secrets/host/sops-age-key.txt` with the host-derived identity
+(`ssh-to-age -private-key -i /secrets/host/ssh_host_ed25519_key`), rebuild, and delete the master
+key from the host.
+
+### 4c. The orchestrator's workspace (as the `loops` user — it exists after the first rebuild, so
+run §5 once first if this is a fresh host, then come back)
+
+```bash
+sudo -iu loops
+git clone --recursive https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git /srv/cc-ci-orch
+sudo ln -sfn /srv/cc-ci-orch /srv/cc-ci          # every script and unit says /srv/cc-ci
+cd /srv/cc-ci-orch
+git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci        # Builder clone
+git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv    # Adversary clone
+mkdir -p .cc-ci-logs .sops
+```
+
+**`/secrets` is the authoritative location for every secret on this host.** Consumers either read
+from it directly (where the nix config owns the path) or reach it by a symlink (where the software
+fixes the path), so `find /secrets -type f` is the complete inventory — including the ssh host
+keys. The directory is `0711` (traversable, not listable) with three subdirectories, each owned by
+whoever must read it; the layout is declared in `nix/hosts/cc-ci/configuration.nix`:
+
+| dir | mode | holds |
+|---|---|---|
+| `/secrets/files` | `loops:users 0700` | the agent's secrets: `.testenv`, opencode auth, its two ssh keys |
+| `/secrets/host` | `root:root 0700` | host identity: the ssh **host** keys and the sops age identity |
+| `/secrets/nginx` | `root:nginx 0750` | `oc-htpasswd` for the opencode UI (nginx must read it) |
+
+**Only cc-ci's secrets go there** — nothing from other projects (no tailscale, lichen, tangled,
+b1, borg…), and every private key is generated **on the host**, never copied from another machine.
+`/secrets/README.txt` lists each file, its purpose, and the derived material that legitimately
+lives elsewhere (`/run/secrets`, `/root/.docker/config.json`, the ACME cert) so a later sweep does
+not "clean" it away.
+
+> **The ed25519 host key is load-bearing beyond ssh.** Its age identity is a recipient of
+> `cc-ci-secrets`, so regenerating it makes every cc-ci secret undecryptable. sshd is pointed at
+> `/secrets/host/...` directly (not through the `/etc/ssh` symlinks) precisely so it can never
+> write a fresh key through a dangling link. To move a host: copy the key, verify
+> `ssh-to-age` yields the same identity, then activate.
+
+| runtime path → `/secrets/files/…` | what | source |
+|---|---|---|
+| `/srv/cc-ci/.testenv` → `cc-ci.testenv` | `GITEA_PASSWORD` (autonomic-bot: PR/API calls), `DOCKERHUB_USERNAME/TOKEN` (harness image pulls). Nothing else: no tailscale key, no third-party model API keys (opencode's own auth is `opencode-auth.json`; add `TINFOIL_API_KEY` back only if `LOOP_MODEL` moves to a tinfoil model). | old host's copy minus the tailscale line; fresh: create each credential |
+| `/srv/cc-ci/upgrader.env` (not a secret, lives in the checkout) | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL`, `SUPERVISOR_MODEL` for the weekly run. This host is on the **OpenCode Go subscription** (`LOOP_TIER=go`, `opencode-go/…` models); the ZEN tier is pay-as-you-go credit and its key is not on this host | old host, or copy the example in `AGENTS.md` |
+| `~loops/.ssh/cc-ci-local-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host. **Generated on the host**, its pub in `nix/hosts/cc-ci/ssh-keys` | `ssh-keygen -t ed25519 -C cc-ci-loops-to-root@cc-ci` as loops |
+| `~loops/.ssh/autonomic-bot-cc-ci-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's ssh config points at the same file to clone the private cc-ci-secrets submodule. **Generated on the host** and registered on the bot's Gitea account (`POST /api/v1/user/keys` with the bot password) | `ssh-keygen -t ed25519 -C autonomic-bot@cc-ci-host` as loops |
+| `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret — it must contain NO inline `apiKey`) | opencode provider auth + config. Exactly ONE provider, `opencode-go` — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
+| `/secrets/nginx/oc-htpasswd` (root:nginx 0640; the bcrypt line only — the plaintext stays with the operator, not on the host) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
+
+`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
+`127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.
+
+## 5. Build and activate
+
+From the checkout with the §3 commit (root can build from the loops-owned checkout via sudo):
+
+```bash
+# as root, detached (the activation restarts sshd; a dropped session must not kill it).
+# Three things the FIRST rebuild on a bare infect system needs, none of which the converged
+# host needs afterwards: `git` on PATH (nix's flake fetcher shells out to it and the infect
+# system has none — hence nix-shell), HOME=/root (so root's `git config --global
+# safe.directory '*'` applies to the loops-owned checkout), and a login shell (`bash -l`, for
+# NIX_SSL_CERT_FILE and friends from /etc/set-environment).
+git config --global --add safe.directory '*'
+systemd-run --unit=ccci-rebuild --collect -E HOME=/root -p WorkingDirectory=/srv/cc-ci-orch \
+  bash -lc 'nix-shell -p git --run "nixos-rebuild test --flake /srv/cc-ci-orch#cc-ci"'
+journalctl -fu ccci-rebuild          # ~10 min the first time (image pulls + two OCI image builds)
+```
+
+`test` first, always: it activates WITHOUT touching the bootloader, so if the activation breaks
+networking or sshd a reboot from the Hetzner console lands on the last known-good generation.
+(Earlier hosts were on the tailnet, and their rebuilds had to be detached because activation
+restarted tailscale under the session; this one is plain public ssh, but detached is still the
+safe habit.)
+Later rebuilds are simply `sudo nixos-rebuild test|switch --flake .#cc-ci` from the checkout.
+
+The first activation takes a while: it pulls the traefik/drone/keycloak images, builds the bridge
+and dashboard OCI images with Nix, initialises the swarm and runs the serialized reconcile
+oneshots (`swarm-init → deploy-proxy → deploy-drone → deploy-bridge → deploy-dashboard →
+deploy-reports`, `deploy-backupbot`, `warm-keycloak`). Verify:
+
+```bash
+systemctl is-system-running        # running — or list-units --failed and read journalctl -u <unit>
+docker service ls                  # traefik app+socket-proxy, drone, bridge, dashboard, reports, backups: 1/1
+systemctl status cc-ci-loops cc-ci-orchestrator opencode-web nginx acme-dns fail2ban
+fail2ban-client status sshd; fail2ban-client status nginx-http-auth
+systemctl list-timers 'cc-ci-*' nightly-sweep
+sudo -iu loops tmux ls             # cc-ci-orchestrator (+ loops sessions if a phase is active)
+# the CI front doors, before DNS points here (expect 200 / 200 / 303 and ssl_verify=0 once
+# /var/lib/acme is restored or a cert has been issued):
+curl -s --resolve ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://ci.commoninternet.net/
+curl -s --resolve report.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://report.ci.commoninternet.net/
+curl -s --resolve drone.ci.commoninternet.net:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://drone.ci.commoninternet.net/
+dig +short @<ip> ns-acme.commoninternet.net   # acme-dns answering on the public 53
+```
+
+The opencode UI: `curl -s --resolve oc.ci.commoninternet.net:443:<ip> -o /dev/null -w '%{http_code}' https://oc.ci.commoninternet.net/`
+→ 401 without credentials, 200 with `-u oc:<password>`; `fail2ban-client status nginx-http-auth`
+counts the failures (it reads nginx's journal — NixOS nginx logs to stderr, not to
+/var/log/nginx/error.log).
+
+When it is healthy: `sudo nixos-rebuild switch --flake .#cc-ci` (same config, now also the boot
+default). **If you are migrating from another host, do §6 before letting it serve anything**: right
+after the first activation scale the `!testme` bridge to 0 and mask the two orchestrator timers so
+the new box does not process PR comments or start a second weekly run while the old host is live:
+
+```bash
+docker service scale ccci-bridge_app=0
+systemctl mask --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer
+```
+
+## 6. Migrating: restore state from the previous host
+
+Over ssh between the hosts (`ssh root@<old> 'tar -C / -cf - <paths>' | ssh root@<new> 'tar -C / -xpf - --numeric-owner'`, or rsync), with the matching service
+stopped on the new host while its directory is copied:
+
+| path | holds | notes |
+|---|---|---|
+| `/var/lib/cc-ci-reports` | the published weekly report pages (`report.ci…`) | |
+| `/var/lib/cc-ci-runs` | per-run artifacts the dashboard shows | |
+| `/var/lib/ci-warm` | warm-canonical state + alerts | recipe warm *volumes* are caches: not copied, rebuilt by the Sunday sweep / first use |
+| `/var/lib/acme` | the Let's Encrypt cert + account **and `acme-dns-accounts.json`** — the account the permanent `_acme-challenge` CNAME points at | without it a fresh registration + a new CNAME at Gandi is needed (registration is disabled in `acme-dns.nix`) |
+| `/var/lib/acme-dns` | the acme-dns zone DB | |
+| `/var/lib/ci-certs` | the copy traefik is handed | then `systemctl restart cc-ci-acme-traefik-handoff` |
+| `/root/.abra` | abra's per-app env files for the deployed stacks | |
+| Drone data volume `/var/lib/docker/volumes/drone_ci_commoninternet_net_data` | Drone's DB: the Gitea OAuth grant, repo activation, build history | `docker service scale drone_ci_commoninternet_net_app=0` on the new host, copy, scale back to 1. Otherwise run `scripts/bootstrap-drone-oauth.sh` (cc-ci repo) with the bot password and re-activate repos |
+| `/srv/cc-ci-orch/.cc-ci-logs`, `/srv/cc-ci-orch/cc-ci-plan/upstream/`, `REBOOTS.md`, `JOURNAL.md` | orchestrator history, the upgrader's per-recipe release-note registry | as loops; do the final sync after stopping the orchestrator on the old host |
+
+## 7. Cutover and verification
+
+1. **DNS** (operator, Gandi zone `commoninternet.net`): A records `ci`, `*.ci` and `ns-acme` →
+   the new IPv4. `acme NS ns-acme` and `_acme-challenge.ci CNAME <account>.acme…` stay as they
+   are. Wait for propagation (`dig +short ci.commoninternet.net`) — the records carry a 3 h TTL,
+   and resolvers that cached the old answer (the Hetzner ones in particular) keep serving it that
+   long. The new host must not list such a resolver: with a mixed set, every other lookup of its
+   own names went to the old server on 2026-09-07 (runner/bridge 404s). `networking.nix` drops
+   the Hetzner resolvers and `configuration.nix` pins the host's own names.
+2. Check the new host answers on the new IP before DNS moves: `dig @<new-ip> ns-acme.commoninternet.net`
+   (acme-dns), `curl --resolve ci.commoninternet.net:443:<new-ip> https://ci.commoninternet.net/`
+   (dashboard, valid cert), same for `report.ci` and `drone.ci`.
+3. Old host: `docker service scale ccci-bridge_app=0 drone_ci_commoninternet_net_app=0`;
+   `systemctl disable --now cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer` on the old
+   orchestrator. New host: `docker service scale ccci-bridge_app=1`;
+   `systemctl unmask cc-ci-upgrade-all.timer cc-ci-upgrade-supervisor.timer && systemctl start` both.
+4. End to end: post `!testme` on an open recipe PR and watch it turn green on the new Drone;
+   open `https://ci.commoninternet.net` and `https://report.ci.commoninternet.net`.
+5. The orchestrator: as loops on the new host `cd /srv/cc-ci-orch && python3 cc-ci-plan/agents.py up orchestrator`
+   (or just `systemctl restart cc-ci-orchestrator`), attach with `claude --resume` or from
+   claude.ai/code. Its startup routine (AGENTS.md) reports phase + reboot count.
+6. Keep the old host as a cold standby for a week, then delete it (and its tailnet node — the
+   old servers were on the tailnet; the new one is not).
+
+## 8. Day 2
+
+- **The host updates itself weekly**: `cc-ci-auto-update.timer` (Tuesday 03:00 UTC,
+  `nix/modules/auto-update.nix`) runs `nix flake update` on all inputs, builds, activates with
+  `test`, runs the cc-ci health checks (swarm replicas, front doors, sops, acme-dns, fail2ban…),
+  and only then commits the generation + bootloader and pushes the health-checked `flake.lock` to
+  `main`; on any failure it re-activates the previous generation and restores the lock. It skips
+  itself while a CI run, the weekly upgrader, the report or the Sunday sweep is in flight. Outcome
+  in `/srv/cc-ci/.cc-ci-logs/auto-update-state` (one line) and `journalctl -u cc-ci-auto-update`;
+  `/cc-ci-status` §5 reads it. `/cc-ci-orchestrator-update` runs the same unit by hand.
+  The `cc-ci` input follows this flake's nixpkgs, so the CI server is rebuilt on the same nixpkgs.
+- **Update only cc-ci's code** (harness/tests/modules): merge in the cc-ci repo, then
+  `nix flake update cc-ci` here and rebuild; also `git -C /etc/cc-ci pull --recurse-submodules`
+  so the deployed checkout the sweep runs from matches. Run the lock update **on this host**: a
+  lock written by a newer Nix elsewhere once failed here with `NAR hash mismatch in input
+  git+https://…cc-ci.git…&shallow=1`; re-running `nix flake update cc-ci` on the host fetched the
+  input properly and the rebuild went through.
+- **Something is down**: `systemctl --failed`, `journalctl -u deploy-<x>`, `docker service ps <svc>`;
+  the cc-ci repo's `docs/runbook.md`. Host unreachable: Hetzner console → reboot lands on the last
+  `switch`ed generation; rescue mode + `nixos-enter` for anything worse (skill
+  `/hetzner-server-recovery`).
+
+---
+
+# Operating the orchestrator
+
+The cc-ci orchestrator on this host is an **opencode agent** (`cc-ci-plan/agents.toml`:
+backend `opencode`, model `opencode/glm-5.2`), launched on boot by `cc-ci-orchestrator.service`
+via `cc-ci-plan/agents.py up orchestrator` into tmux session `cc-ci-orchestrator`, attached to
+the shared opencode web server. Steer it from the web UI, **https://oc.ci.commoninternet.net**
+(basic auth, user `oc`), where the weekly upgrader's and report's sessions show up too.
+There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and
+operate the CI from there over ssh (`Host cc-ci` in that box's loops ssh config → this host).
+
+```bash
+# on the host
+sudo -iu loops tmux attach -t cc-ci-orchestrator     # the agent's terminal
+systemctl restart cc-ci-orchestrator                 # relaunch it (fresh opencode session)
+sudo -iu loops tmux ls                               # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run
+```
+
+## Weekly upgrade run
+
+`cc-ci-upgrade-all.timer` (Thu 22:00 America/New_York) starts `launch-upgrader.py`, which runs
+`/upgrade-all` as an opencode agent (`upgrader.env`: `LOOP_MODEL`/`REPORT_MODEL`) in tmux
+`cc-ci-upgrader` with a usage-limit watchdog; the hourly `cc-ci-upgrade-supervisor.timer` drives a
+stalled run to completion. Start one by hand with `systemctl start cc-ci-upgrade-all`; follow
+`/srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log`; it ends with `UPGRADE RUN COMPLETE`, a summary in
+`.cc-ci-logs/upgrades/upgrade-all-<date>.md`, and a report published to
+`https://report.ci.commoninternet.net/week-<date>.html`.

 ## Kick off / supervise the loops

 ```bash
 cd /srv/cc-ci/cc-ci-plan
-./launch.sh start                       # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
+./launch.sh start                       # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
 ./launch.sh status                      # session + DONE state
 ./launch.sh logs builder|adversary|watchdog
 ./launch.sh stop
 ```

-Full supervision guide, credential map, and the Incus VM fallback are in
-[`cc-ci-plan/kickoff.md`](cc-ci-plan/kickoff.md) and [`cc-ci-plan/plan.md`](cc-ci-plan/plan.md) §1.5.
+Full supervision guide, credential map and history are in `cc-ci-plan/kickoff.md`,
+`cc-ci-plan/plan.md` §1.5 and `cc-ci-plan/JOURNAL.md`.
@@ -0,0 +1,22 @@
+# archive/ — how cc-ci and its orchestrator were built and moved, before the combined host
+
+Historical record only. Nothing in here is deployed or evaluated. It was moved out of the live
+tree on 2026-09-07 when the CI server and the orchestrator were consolidated onto one Hetzner
+host (`nixosConfigurations.cc-ci` in `../flake.nix`; deploy guide in `../README.md`; the plan
+that did it is `../cc-ci-plan/plan-cc-ci-combined-host.md`).
+
+| path | what it was |
+|---|---|
+| `nix/configuration-incus-vm.nix` | Channel-based NixOS config of the first orchestrator VM on b1 (Incus, 2 GB). Ran the loops as root; hard-coded the dead Incus cc-ci IP. Replaced by the Hetzner host 2026-05-31. |
+| `nix/README.md` | The README for that Incus VM config. |
+| `nix/cc-ci-orchestrator-hetzner/` | The orchestrator's own Hetzner `cpx22` host (`168.119.126.100`, tailnet `cc-ci-orchestrator-1`), 2026-05-31 → 2026-09. From 2026-08-20 the live copy of this config was `notplants-nix`'s `notplants-orchestrator` host (the box became a shared agent host for several projects); this one had drifted and still carried lichen/project-orchestrator units. Superseded by `../nix/hosts/cc-ci` + `../nix/modules/orchestrator-host.nix`. |
+| `nix/atproto-likes.nix` | A notplants (not cc-ci) service that lived on the shared box; kept by notplants-nix. |
+| `terraform/` | OpenTofu for the `cpx22` orchestrator server (Debian 12 → nixos-infect at `nixos-24.11`). The combined host was provisioned by hand instead; the README documents that path. Note its `user-data.sh` would fail on the Debian 13 image (nixos-infect's temp swapfile on a tmpfs `/tmp`) — see the README's `NO_SWAP=true` note. |
+| `plans/plan-orchestrator-migration.md` | Pi → Incus VM move of the orchestrator (2026-05). |
+| `plans/plan-orchestrator-hetzner-migration.md` | Incus VM → Hetzner `cpx22` move of the orchestrator (2026-05-31). Has the reboot-resilience design (`cc-ci-loops.service`). |
+| `plans/plan-migrate-cc-ci-to-hetzner.md`, `plans/plan-cc-ci-hetzner-migration.md`, `plans/plan-cc-ci-hetzner-terraform.md` | The CI server's own move from the `cc-nix-test` Incus VM to Hetzner `cpx32` (`91.98.47.73`, 2026-05-31), and the terraform that provisioned it (lives in the cc-ci repo). |
+| `plans/plan-repo-consolidation.md` | The earlier repo layout consolidation. |
+
+The cc-ci server's own history (machine-docs, decisions, the clean-room rebuild that proved
+"two repos + one age key + one `nixos-rebuild switch`") is in the cc-ci repo under
+`machine-docs/` and `docs/`.
@@ -976,3 +976,229 @@ certificate. The operator-managed public gateway at `143.244.213.108` currently
 closes TLS before presenting a certificate, so its passthrough to cc-ci needs repair/verification.
 The cc-ci plan explicitly marks the gateway as operator infrastructure; do not change the DNS
 delegation or add a Gandi token to work around it.
+
+## Session 2026-08-31 20:45 UTC — weekly DeepSeek Flash run and report complete
+
+**Completed:** Started the weekly upgrader explicitly on `opencode/deepseek-v4-flash`; its parent
+and every recipe subagent were confirmed with that persisted model. All nine eligible recipe PRs
+completed `!testme` successfully. The public report was then restarted before publication when the
+first handoff was found to use the old GLM model; the replacement report session was confirmed
+`opencode/deepseek-v4-flash` and published successfully at
+`https://report.ci.commoninternet.net/week-2026-08-31.html` (HTTP 200).
+
+**Follow-up fixes:** Orchestrator PR #17 (`1b75d98`) made `testme-on-pr.sh` apply the documented
+public Gitea-host default. cc-ci PR #31 (`769fd29`) adds that public hostname to `.env.public`.
+Both commits were scanned clean and contain no coauthor trailers. No recipe PR was merged.
+
+**Security note:** A subagent briefly enabled shell tracing while debugging the verifier, exposing
+runtime credentials in its private agent trace. No values were committed or put in this journal,
+but rotate the affected `/srv/cc-ci/.testenv` credentials as a precaution.
+
+## Session 2026-09-07 19:30 UTC — Claude Fable 5.1 orchestrator (re)launch, startup check
+
+**What happened:** Orchestrator relaunched on the `claude` backend (`agents.toml` now says
+`backend = "claude"`, `model = "claude-fable-5-1"`, operator change today, uncommitted). Ran the
+AGENTS.md on-startup routine. NOT a reboot: host uptime 15 days, REBOOTS.md still shows 5 reboots
+(last 2026-08-23 03:11 UTC). `cc-ci-loops.service` was restarted at 14:50 and 15:14 UTC today by a
+`nixos-rebuild test --flake /srv/notplants-nix#notplants-orchestrator`, which re-ran `launch.sh start`;
+the phase sequence immediately re-concluded (all 15 phases DONE, "entire build finished"), so
+builder/adversary/watchdog being stopped is the expected terminal state. Did NOT relaunch the loops.
+
+**Current state:**
+- Weekly `/upgrade-all` 2026-09-04 completed: 8 upgrade PRs extended (custom-html, ghost,
+  lasuite-docs/drive/meet, matrix-synapse, mattermost-lts, n8n), 0 failed, nothing merged. Report
+  `week-2026-09-04.html` returns 200. Next timer run Fri 2026-09-11 02:00 UTC.
+- Hourly supervisor (XX:07) fires and stands down in ~1s — nothing to drive.
+- Open operator items from the 09-04 run: review/merge the 8 PRs; `warm-gitea` canonical
+  crash-looping on read-only `/etc/gitea` (pre-existing); deployed `/root/cc-ci/tests` on the CI host
+  lags server-repo `main` (missing `tests/wordpress`).
+- Uncommitted in this checkout (left alone, operator WIP): `agents.toml` backend switch,
+  auto-appended 2026-08-23 line in `REBOOTS.md`, and the untracked `plan-agent-orchestrator.md` /
+  `plan-phase-ao*.md` / `cc-ci-conc/` set.
+
+## Session 2026-09-07 20:00 UTC — start of the cc-ci + orchestrator consolidation onto one Hetzner host
+
+**Operator request:** move the cc-ci CI server AND the orchestrator to a new Hetzner box
+(`195.201.88.249`, 8 GB), leave everything notplants-side on this host, keep cc-ci's nix in the
+cc-ci repo and the orchestrator's in cc-ci-orchestrator with the latter including the former,
+add `archive/` + a from-scratch deploy README, and (last) move to `autonomic.zone` subdomains.
+Plan + live log: `cc-ci-plan/plan-cc-ci-combined-host.md` (on the branch; copy here).
+
+**Done this session:**
+- New ssh key `notplants-orchestrator` (`/secrets/files/notplants-orchestrator-ed25519`), on the new box.
+- nixos-infect on the new box (Debian 13 → NixOS 26.05). Gotcha: `/tmp` is tmpfs on that image,
+  nixos-infect's temp swapfile fails → `NO_SWAP=true`. It built and rebooted ~19:50 UTC and had
+  NOT come back by 20:00 (no ping) — operator to check the Hetzner console / give an API token.
+- cc-ci branch `feat/nixos-module-export` (9b99f81, pushed): `nixosModules.cc-ci-server`
+  (`nix/modules/default.nix`), options `cc-ci.publicIPv4` + `cc-ci.sopsFile`; standalone `#cc-ci`
+  drv byte-identical before/after.
+- cc-ci-orchestrator branch `feat/combined-cc-ci-host` (31af820, pushed): flake input `cc-ci`
+  (follows), `nixosConfigurations.cc-ci`, `nix/modules/orchestrator-host.nix`, `nix/hosts/cc-ci/`
+  (hardware/networking PROVISIONAL until the infect output is captured), README deploy guide,
+  `archive/` (old host configs, terraform, migration plans), AGENTS.md + update-skill refs.
+  `#cc-ci` evaluates. Work is in git worktrees under the session scratchpad, not in this checkout.
+
+**Next:** box reachable → capture hardware/networking → stage secrets → `nixos-rebuild test`
+→ data copy → DNS cutover → move the orchestrator → notplants-nix PR dropping cc-ci → autonomic.zone.
+
+## Session 2026-09-07 20:30 UTC — new combined host is UP, pre-cutover
+
+- nixos-infect trouble root-caused from Hetzner rescue mode (operator gave an API token, stored
+  at `/srv/cc-ci/.hcloud-token`, server id 165014541, cpx32 nbg1): (1) `NO_SWAP=true` for tmpfs
+  /tmp; (2) 26.05's systemd initrd did NOT lustrate — Debian's units shadowed NixOS's, every
+  service failed; fixed by moving the old root to `/old-root` by hand; (3) bare-string
+  `defaultGateway` → no default route; fixed + chroot `nixos-rebuild boot --option sandbox false`.
+  All documented in the new README §2a.
+- cc-ci PR #32 merged (module export). cc-ci-orchestrator PR #19 merged (combined host). Both
+  branches scanned clean by the commit hook.
+- New box: `nixos-rebuild test` → verified → `switch`; reboot test OK. Data restored: acme (+
+  acme-dns account), acme-dns, ci-certs, reports, runs, ci-warm, /root/.abra, Drone volume (with
+  drone scaled to 0 during the copy). Dashboard/reports/drone answer on the new IP with the valid
+  LE cert; acme-dns answers on public 53.
+- Pre-cutover quarantine on the new box: `ccci-bridge_app` scaled to 0, both cc-ci timers
+  `mask --runtime`, cc-ci-orchestrator/loops units stopped (these do NOT survive a reboot — redo).
+- Staged for loops: ~/.claude, opencode config+state, ssh keys, .testenv, upgrader.env,
+  .sops/master-age.txt, .cc-ci-logs; nginx oc-* files (root:nginx 0640).
+- Open: tailscale auth key revoked (`invalid key: API key does not exist`) → operator issues a
+  new one. DNS cutover at Gandi (ci, *.ci, ns-acme → 195.201.88.249) → operator.
+
+## 2026-09-07 22:15 UTC — first weekly upgrade run on the new host: GREEN, report published
+
+Started by hand 21:23 UTC (`systemctl start cc-ci-upgrade-all` on 195.201.88.249, opencode /
+deepseek-v4-flash); `UPGRADE RUN COMPLETE` 22:02 (39 min). Everything ran on the new host — old
+server's Drone/bridge at 0/0, no new run dirs or report there. 20 recipes surveyed, 2 upgrade PRs
+extended and `!testme` GREEN on the new Drone (lasuite-docs #8 → v5.6.1, build 1338; n8n #7 →
+2.38.4, build 1339), 1 PR closed as merged upstream (custom-html #7), 18 skipped as up-to-date or
+covered. Summary: `.cc-ci-logs/upgrades/upgrade-all-2026-09-07.md`. Report agent published
+https://report.ci.commoninternet.net/week-2026-09-07.html (200, 42 KB, indexed) at 22:11.
+One side effect: the run's orphan sweep removed the `opencode-ui` swarm stack (traefik route to
+the opencode web UI) — redeployed, renamed `ccci-opencode-ui`, added to the sweep keep-list.
+
+## 2026-09-08 02:05 UTC — cc-ci host auto-update + skills rewritten for the combined host
+
+**Auto-update (own module, no notplants-nix dependency):** `nix/modules/auto-update.nix`,
+`cc-ci-auto-update.timer` Tue 03:00 UTC. Busy gate (CI run / weekly upgrader / report / sweep /
+running Drone builds) → `nix flake update` (all inputs) → build as loops → `switch-to-configuration
+test` → 45 s settle → health check (sshd, 0 failed units, docker+acme-dns+fail2ban+nginx+
+opencode-web+runner active, every swarm service at replica count, sops decrypted,
+dashboard/reports/drone 200, opencode UI 401) → profile + bootloader → flake.lock committed as
+`auto-update` and pushed to main → `/etc/cc-ci` fast-forwarded. Failure re-activates the previous
+generation and restores the lock. One-line outcome in `.cc-ci-logs/auto-update-state`.
+
+**Tested for real, both paths:**
+- Full run 22:23–22:27 UTC: nixpkgs 20260803.531670d → 20260906.c257840 and sops-nix moved,
+  built, activated, `HEALTH OK`, committed (`d1f5e91`, pushed), running == boot generation.
+- Busy gate: with a fake `cc-ci-upgrader` tmux session the unit exited 0 with
+  `result=skipped note=busy` and changed nothing.
+
+**Skills (canonical in `.opencode/skills`, `.claude` pointer descriptions synced):**
+- `/cc-ci-orchestrator-update` — THE host update; drives the auto-update unit by hand, has the
+  failure playbook and `--cc-ci-only` for a cc-ci-main-only move.
+- `/cc-ci-server-update` — now delegates to it, and says why the old procedure (rebuilding the
+  cc-ci repo's standalone `#cc-ci`) must never run on this host.
+- `/cc-ci-update` — chains orchestrator-update then tests-update.
+- `/cc-ci-status` — §5/§6 rewritten for one host: auto-update state + staleness, running vs boot
+  generation, front doors (oc.ci must be 401), sops, fail2ban, all four timers, the orchestrator
+  agent session, and a secrets-inventory check that flags sprawl. Verdict updated.
+  Verified against the box: 0 failed units, 10/10 swarm services, disk 33%, all timers armed.
+
+## 2026-09-08 16:55 UTC — cc-ci host moved to the operator's OpenCode **Go** key; extra credentials purged
+
+**The key.** `AUTONOMIC_OPENCODE_KEY` (operator, in `/secrets/files/cc-ci.testenv`) is an OpenCode
+**Go subscription** key, not a ZEN one. Installed into the ZEN slot it authenticates but every
+request dies `Insufficient balance` (ZEN = pay-as-you-go credit); on the `opencode-go` endpoint the
+same key answers immediately. Diagnosis was the operator's — worth remembering as the first thing
+to check when an opencode key "does not work".
+
+- `auth.json` (→ `/secrets/files/opencode-auth.json`) now holds **exactly one** provider,
+  `opencode-go`, carrying that key. The previous ZEN key is deleted; `opencode/glm-5.2` now errors.
+- `agents.toml` orchestrator → `opencode-go/glm-5.2`; `upgrader.env` → `LOOP_TIER=go` and
+  LOOP/REPORT/SUPERVISOR models all `opencode-go/glm-5.2`; cc-ci PR #36 moved the subagent model in
+  `cc-ci/opencode.json` the same way.
+- **Model note:** `opencode-go/deepseek-v4-flash` (what the weekly run used on ZEN) is China-hosted
+  on the Go tier and returns an opt-in error until the operator enables it at
+  `https://opencode.ai/workspace/wrk_01M20RVAY3D3B28XAYY1Z2Q6AZ/go`. `glm-5.2` is used meanwhile.
+- Verified: `opencode-go/glm-5.2` answers, the orchestrator agent restarted on it, ZEN gone.
+
+**Purged while there** (same "only what cc-ci strictly needs" rule): `auth.json` had also carried
+`openai` (oauth), `opencode-go` (old) and `infomaniak` credentials copied from the notplants box,
+and `~/.config/opencode/opencode.jsonc` carried a **Tinfoil API key inline in plaintext**. All
+removed; the config is now a 4-line file with no `apiKey` anywhere. That Tinfoil key still lives in
+the notplants-orchestrator config it came from — **operator: consider rotating it**, since it was
+briefly present on a second host.
+
+**Also:** `vim`/`vi` (absent, the operator hit it over ssh), `sqlite`, `bat`, `bc`, `moreutils`,
+`pv`, `man-pages` added to the host toolbox with `EDITOR=vim`; on PATH for root and loops.
+Stale `oc.commoninternet.net` URLs in the launchers → `oc.ci.commoninternet.net`.
+## Session 2026-09-08 16:52 UTC — OpenCode GPT-5.6-luna startup supervision
+
+**Left off:** Read the kickoff prompt, confirmed the host loop service is active, and performed the
+required `RESUME_PHASE=1` relaunch; the completed phase sequence immediately stood down again.
+
+**Phase / loop state:**
+- Phase **ghost [15/15]**, `STATUS-ghost.md` is `## DONE`; the full phase sequence completed.
+- `cc-ci-loops.service` is active (exited successfully); builder, adversary, and watchdog are
+  stopped because there is no remaining phase work.
+- Reboot count remains **5**, latest logged reboot 2026-08-23 03:11 UTC.
+
+**Open items:** none for loop supervision.
+
+**Notes:** `cc-ci-plan/launch.py status` is not executable directly in this checkout, so the
+  equivalent `python3 cc-ci-plan/launch.py status` was used. The required launch was run via
+  `RESUME_PHASE=1 cc-ci-plan/launch.sh start`; it re-confirmed completion at 16:52:34 UTC.
+## 2026-09-08 17:10 UTC — secrets audit of the cc-ci host (operator asked for a re-verify)
+
+**Result: `/secrets` is exactly what it should be** — 6 files + the inventory README, every one of
+them cross-checked as actually referenced by something on the host:
+`cc-ci.testenv` (GITEA_PASSWORD, DOCKERHUB_*, AUTONOMIC_OPENCODE_KEY), `opencode-auth.json`
+(one provider, `opencode-go`), and two ssh keypairs **generated on this host**
+(`cc-ci-local-ed25519` → `ssh cc-ci`; `autonomic-bot-cc-ci-ed25519` → Gitea, key id 156).
+
+**Proved absent, not just "removed":** the sops **master key** is gone and
+`/var/lib/sops-nix/key.txt` is provably this host's OWN identity (its public half equals
+`ssh-to-age` of the host ssh key: `age1tmvg…`). No tailscale key, no `.netrc`/`.git-credentials`/
+`.aws`/`.npmrc`, no third-party model keys, no inline `apiKey` in any opencode config, and the old
+ZEN key is absent from files, the opencode database and its logs.
+
+**One real finding, fixed.** The agent transcripts I rsynced from the old orchestrator box
+(`/srv/cc-ci/.cc-ci-logs/*.log`, ~145 MB) contained **live credential values in plaintext** — a
+`tskey-auth-…` tailscale key and an older OpenCode ZEN key `sk-iLmK5…` — because earlier sessions
+had pasted the values into logged `grep` commands while running their own secret scans. 133
+occurrences across 7 logs, redacted in place (`<REDACTED-BY-SECRET-SWEEP>`); a whole-host rescan is
+now clean. **Operator: rotate both of those** if either is still live — they sat on a second host.
+Lesson for future sweeps: scanning `/secrets` is not enough, agent logs are a secret sink.
+
+**Legitimately outside `/secrets`** (documented in its README so a later sweep does not "clean"
+them): `/run/secrets/*`, `/root/.docker/config.json`, `/etc/nginx/oc-htpasswd`,
+`/var/lib/ci-certs/live/*.pem`, `/var/lib/sops-nix/key.txt`, `/etc/ssh/ssh_host_*`.
+
+## 2026-09-08 17:30 UTC — /secrets made authoritative (incl. ssh host keys); weekly-run models
+
+**Operator rule:** every secret lives in `/secrets`; consumers read from there or symlink to it.
+Implemented declaratively in `nix/hosts/cc-ci/configuration.nix` (cc-ci-orchestrator e7aa055):
+`/secrets` `0711` with `files/` (loops), `host/` (root: ssh host keys + sops age identity) and
+`nginx/` (root:nginx: the UI htpasswd). `find / -type f` for key material now returns **only**
+`/secrets`; `/etc/ssh/ssh_host_*`, `~loops/.ssh/*`, `~loops/.local/share/opencode/auth.json` and
+`/srv/cc-ci/.testenv` are all symlinks into it.
+
+**The risky part, done safely.** The ed25519 host key is a sops recipient (`age1tmvg…`), so a
+regenerated key would make every cc-ci secret undecryptable. Therefore: keys **copied** (never
+moved-then-regenerated), the age identity verified identical before activation, `services.openssh.
+hostKeys` and `sops.age.{sshKeyPaths,keyFile}` pointed at `/secrets/host` **directly** rather than
+through the `/etc/ssh` symlinks (a dangling link would let sshd write a NEW key), `test` before
+`switch`, and the proof was a fresh `StrictHostKeyChecking=yes` connection plus 8/8 sops secrets.
+Fingerprint unchanged: `SHA256:r1H85Ybif…`. Then the duplicates outside `/secrets` were deleted
+(`/var/lib/sops-nix/key.txt`, `/etc/nginx/oc-htpasswd`) and the unused ecdsa host key removed.
+
+**Logs cleared** (operator: "we don't need them anymore"): 30 agent transcripts + 1 archived gz,
+558 MB → 2 MB. Kept `upgrades/` (233 weekly records), `state/`, the `*-update-*.md` records and the
+launcher dotfiles.
+
+**Weekly-run models** — operator enabled China-hosted models on the workspace, so both verified on
+the host: `opencode-go/deepseek-v4-flash` and `opencode-go/glm-5.3-flash` answer.
+- **subagents** (per-recipe workers): `opencode-go/deepseek-v4-flash` — cc-ci PR #37, pulled into
+  both loops clones and `/etc/cc-ci`.
+- **main agents** (weekly parent, report, hourly supervisor): `opencode-go/glm-5.3-flash` —
+  `upgrader.env` (`LOOP_TIER=go` maps to the `opencode-go` auth entry; `LOOP_MODEL` overrides the
+  tier default). Next fire Fri 2026-09-11 02:00 UTC.
+- The steering orchestrator agent stays on `opencode-go/glm-5.2` (not asked to change).
@@ -22,3 +22,4 @@ restarts the loops on boot. Count the lines below to see how often it's happenin
  auto-logging. Auto-logging is live from the next reboot onward.
 - 2026-05-30 17:03:05 BST — reboot detected; loops auto-started by systemd (resuming phase index 6). boot_id=f565f752-0463-42db-b787-9e0db35a5e3f
 - 2026-05-31 03:38:29 UTC — reboot detected; loops auto-started by systemd (resuming phase index 5). boot_id=51c17fc3-8391-4109-bce2-413fbee6f26d
+- 2026-08-23 03:11:24 UTC — reboot detected; loops auto-started by systemd (resuming phase index 14). boot_id=8215fc5e-8032-4844-b2b8-e0f8c0d9b13e
@@ -53,8 +53,9 @@ log_dir = "/srv/cc-ci/.cc-ci-logs"
 [[agent]]
 name    = "orchestrator"       # tmux session: cc-ci-orchestrator
 kind    = "persistent"
-backend = "opencode"           # attached to the shared opencode web server (oc.commoninternet.net)
-model   = "opencode/glm-5.2"   # opencode backend; visible at https://oc.commoninternet.net
+backend = "opencode"              # operator 2026-09-07 (evening): NO Claude on the cc-ci host — the
+model   = "opencode-go/glm-5.2"   # orchestrator is an opencode agent, steered via the opencode web UI
+                               # (https://oc.ci.commoninternet.net). Claude sessions live on notplants-orchestrator.
 resume  = false                # opencode backend has no --resume equivalent (fresh session each launch)
 watch   = "heal"               # restart if dead / FATAL / backend-mismatch; never stall-reboot
 wake    = { interval = 3600, prompt_file = "ai-progress-monitor-prompt.txt" }
@@ -116,7 +116,7 @@ def start(mode="resume"):
            f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {WORKDIR}"
        )
        log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
-        log("  visible at http://oc.commoninternet.net (tailnet only)")
+        log("  visible at https://oc.ci.commoninternet.net")
    else:
        die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")

@@ -158,7 +158,7 @@ def start(mode="resume"):
            f"{model_env}NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {shlex.quote(WORKDIR)}"
        )
        log(f"starting {SESSION} (backend=opencode, model={LOOP_MODEL or 'default'})")
-        log(f"  visible at http://oc.commoninternet.net (tailnet only)")
+        log(f"  visible at https://oc.ci.commoninternet.net")
    else:
        die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")

@@ -212,7 +212,7 @@ def main():
    else:
        backend_note = (
            "claude:   --resume preserves conversation across reboots; viewable at claude.ai/code\n"
-            "  opencode: fresh session each launch (no --resume); viewable at http://oc.commoninternet.net"
+            "  opencode: fresh session each launch (no --resume); viewable at https://oc.ci.commoninternet.net"
        )
        print(f"""cc-ci orchestrator launcher

@@ -124,7 +124,7 @@ def start(mode, date):
        _lu._pin_new_session(_prev_ids)
    if BACKEND == "opencode":
        if OPENCODE_SHARE:
-            log(f"  attached to {OPENCODE_SERVER} → http://oc.commoninternet.net  +public --share link")
+            log(f"  attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net  +public --share link")
        # Watchdog for the report too: it runs on glm-5.2 sharing the same opencode-go budget the
        # upgrade run just drained, so a 429 stall is likely. Reuse launch-upgrader.py's watchdog,
        # pointed at THIS (cc-ci-report) session with a report-specific marker + resume prompt. It
@@ -67,7 +67,7 @@ REMOTE_CONTROL = os.environ.get("REMOTE_CONTROL", "1") == "1"
 OPENCODE_BIN    = os.environ.get("OPENCODE_BIN",    "/home/loops/.local/bin/opencode")
 OPENCODE_SERVER = os.environ.get("OPENCODE_SERVER", "http://127.0.0.1:4096")
 # Web visibility for the opencode backend: attach the session to the shared opencode
-# web server (viewable at http://oc.commoninternet.net, tailnet-only) AND optionally
+# web server (viewable at https://oc.ci.commoninternet.net (basic auth)) AND optionally
 # create a public opencode.ai --share link. Default both on so the run is monitorable.
 OPENCODE_SHARE = os.environ.get("OPENCODE_SHARE", "1") == "1"

@@ -220,7 +220,7 @@ def start(mode="use-or-create"):
            f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
            f"--title '{SESSION}' \"$(cat '{kf}')\""
        )
-        log(f"  attached to {OPENCODE_SERVER} → http://oc.commoninternet.net (tailnet only)"
+        log(f"  attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net"
            + ("  +public --share link (printed in the session)" if OPENCODE_SHARE else ""))
    else:
        die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
@@ -655,7 +655,7 @@ Model:   {MODEL}  (LOOP_MODEL or UPGRADER_MODEL env var)
 Args:    {UPGRADER_ARGS or '<none>'}  (UPGRADER_ARGS env var, passed to /upgrade-all)

 claude:   viewable at claude.ai/code
-opencode: viewable at http://oc.commoninternet.net  server={OPENCODE_SERVER}
+opencode: viewable at https://oc.ci.commoninternet.net  server={OPENCODE_SERVER}
 """)


@@ -327,7 +327,7 @@ def start_agent(role, session, workdir):
            f"NO_COLOR=1 {OPENCODE_BIN} attach {OPENCODE_SERVER} --dir {session_cwd}"
        )
        log(f"starting {session} (backend=opencode, phase={pid}, model={model or 'default'})")
-        log(f"  visible at http://oc.commoninternet.net (tailnet only)")
+        log(f"  visible at https://oc.ci.commoninternet.net")
    else:
        die(f"unknown BACKEND '{BACKEND}' — set LOOP_BACKEND=claude or LOOP_BACKEND=opencode")

@@ -0,0 +1,127 @@
+# Plan — one Hetzner host for cc-ci (CI server) + cc-ci-orchestrator
+
+**Status:** IN PROGRESS (started 2026-09-07). Operator request: move the cc-ci CI server AND the
+cc-ci orchestrator onto one new Hetzner server (`195.201.88.249`, 8 GB, 150 GB, Debian 13 image),
+cleanly split off from the shared `notplants-orchestrator` box, which keeps everything else
+(lichen, project-orchestrator, notplants agents). Nix config ownership: cc-ci's config in
+`recipe-maintainers/cc-ci`, the orchestrator's in `recipe-maintainers/cc-ci-orchestrator`, and the
+orchestrator flake **includes** cc-ci's module so one `nixos-rebuild` produces the combined host.
+Last step (separate, after everything works on the current names): move both to `autonomic.zone`
+subdomains.
+
+## Facts (2026-09-07)
+
+| | old cc-ci server | old orchestrator host (stays, becomes notplants-only) | **new combined host** |
+|---|---|---|---|
+| public IP | 91.98.47.73 (fsn1, Hetzner 134485294) | 168.119.126.100 (nbg1, Hetzner 134487234) | **195.201.88.249** |
+| tailnet | `cc-ci` 100.95.31.88 | `cc-ci-orchestrator-1` 100.84.190.30 | `cc-ci` (new node) |
+| RAM / disk | 8 GB / 150 GB (83 GB used, 59 GB docker) | 4 GB + 4 GB swap / 75 GB + 250 GB `/mnt/data` | 8 GB / 150 GB, one disk |
+| built by | `cc-ci` flake `#cc-ci` (nixpkgs 26.05 rev 531670d) | `notplants-nix` flake `#notplants-orchestrator` (26.05 channel), importing `cc-ci-orchestrator`'s `nixosModules.cc-ci` | `cc-ci-orchestrator` flake `#cc-ci` importing `cc-ci`'s `nixosModules.cc-ci-server` |
+| DNS | `ci.`, `*.ci.`, `ns-acme.commoninternet.net` → 91.98.47.73 (Gandi, direct, no gateway) | `oc.commoninternet.net` → 100.84.190.30 | operator repoints at cutover |
+
+Data on the old cc-ci server that must move: `/var/lib/cc-ci-reports` (published reports),
+`/var/lib/cc-ci-runs` (dashboard artifacts, 1.7 G), `/var/lib/ci-warm` (1.4 G), `/var/lib/acme`
+(LE cert valid to 2026-11-29 + **acme-dns account json** that the `_acme-challenge` CNAME points at),
+`/var/lib/acme-dns` (the authoritative zone DB), `/var/lib/ci-certs`, `/root/.abra` (app env files),
+`/etc/cc-ci` (deployed checkout the Sunday sweep runs from), Drone's `drone_ci_commoninternet_net_data`
+volume (Gitea OAuth grant + repo activation + build history). Warm recipe volumes are caches and get
+rebuilt on first use / the Sunday sweep. Docker swarm secrets/configs cannot be copied; the reconcile
+oneshots recreate them from sops.
+
+Out-of-band secrets the new host needs (never in git): `/var/lib/sops-nix/key.txt` (= the master age
+key, `/srv/cc-ci/.sops/master-age.txt` here — the new host's SSH host key is not a sops recipient),
+`/etc/ts-auth-key`, `/srv/cc-ci/.testenv`, `/srv/cc-ci/upgrader.env`, `/srv/cc-ci/.sops/master-age.txt`,
+`~loops/.ssh/{cc-ci-root,autonomic-bot-gitea,tangled}-ed25519`, `/etc/nginx/oc-*` (self-signed cert +
+htpasswd for the opencode UI), claude/opencode/codex auth under `~loops`.
+
+## Design
+
+**cc-ci repo** (`feat/nixos-module-export`):
+- `nixosModules.cc-ci-server` = `nix/modules/default.nix`: imports all service modules + the
+  host-generic cc-ci settings that used to sit in the host file (UTC, docker/swarm firewall 80/443,
+  `environment.systemPackages = ccciRuntimeTools`, allowUnfree). No hardware, no networking, no
+  tailscale, no root keys, no stateVersion — the host supplies those.
+- New options under `cc-ci.*`: `publicIPv4` (acme-dns listen + the `ns-acme` A record),
+  `sopsFile` (absolute path to the decrypted-at-activation `secrets.yaml`, default the submodule
+  path so `#cc-ci` keeps working), `repoPath` (`/etc/cc-ci`, used by nightly-sweep).
+- `nixosConfigurations.cc-ci` (old host) keeps building unchanged via the same module.
+
+**cc-ci-orchestrator repo** (`feat/combined-cc-ci-host`):
+- flake input `cc-ci` (https, public) with `nixpkgs`/`sops-nix` `follows` so one nixpkgs + one sops-nix.
+- `nixosModules.cc-ci-orchestrator` (the existing `nix/modules/cc-ci.nix`, kept exported as
+  `nixosModules.cc-ci` too so notplants-nix keeps evaluating until it drops the input) — the loops,
+  orchestrator session and the weekly/hourly timers.
+- `nix/modules/orchestrator-host.nix`: the host contract the module assumes — `loops` user + sudo,
+  nix-ld, claude/opencode/codex installers, `opencode-web`, the tailnet-only nginx `oc.` vhost
+  (on the tailscale IP, port **8443**, because traefik owns 80/443), tool packages, PATH.
+- `nixosConfigurations.cc-ci` = `nix/hosts/cc-ci/{configuration,hardware,networking}.nix` importing
+  both modules. `/srv` is a plain directory (no `/mnt/data`), 8 GB swapfile, root keys, tailscale
+  `--hostname=cc-ci`, firewall 22 (+ what cc-ci-server opens: 80, 443, 53).
+- `loops`' ssh config `Host cc-ci` → `127.0.0.1` so every `ssh cc-ci …` in skills/scripts keeps working.
+- `archive/`: the retired Incus/Hetzner-orchestrator host config, old terraform, historical plans.
+- `README.md`: provisioning (Hetzner Debian → nixos-infect → NixOS), secrets staging, the one
+  `nixos-rebuild`, data restore, cutover, verification — written so a person or an LLM can redo it.
+
+**notplants-nix** (`chore/drop-cc-ci`, after cutover): remove the `cc-ci` input, module import, the
+four cc-ci units' mount gating, `loopsSshConfig`, `opencode-web` + the `oc.` vhost (unless something
+notplants-side uses it), tailscale hostname → `notplants-orchestrator`.
+
+## Steps
+
+1. [x] nixos-infect the new box (`NIX_CHANNEL=nixos-26.05 PROVIDER=hetzner`); capture
+   `hardware-configuration.nix` + `networking.nix`.
+2. [x] cc-ci: module export + options; verify `#cc-ci` still evaluates; PR.
+3. [x] cc-ci-orchestrator: input + host + modules + archive/ + README + terraform refresh; verify
+   `#cc-ci` evaluates; PR.
+4. [x] Stage secrets + clones on the new host; `nixos-rebuild test` → verify → `switch`.
+   Immediately after: scale the new `ccci-bridge_app` to 0 and mask the two cc-ci timers so the
+   new host does not double-process `!testme` or run a second weekly upgrade before cutover.
+5. [x] Copy data (rsync over tailscale): reports, runs, ci-warm, acme, acme-dns, ci-certs,
+   /root/.abra, /etc/cc-ci; Drone volume with Drone scaled to 0 during the copy.
+6. [x] Pre-cutover verification on the new IP (`curl --resolve`, port 53, dashboard, reports,
+   drone, one direct `cc-ci-run` on custom-html-tiny).
+7. [x] Operator: Gandi A records `ci`, `*.ci`, `ns-acme` → 195.201.88.249. Then: old bridge +
+   drone + timers off, new bridge up, one real `!testme` end-to-end, a `!testme`-driven report page.
+8. [x] Move the orchestrator: stop cc-ci units here, final rsync of `/srv/cc-ci-orch` + agent
+   state, enable on the new host, operator reconnects there; notplants-nix PR removing cc-ci.
+9. [ ] Old cc-ci server: cold standby ~1 week, then operator deletes it and the stale tailnet node.
+10. [ ] Domain move to `autonomic.zone` — separate plan, after 1–9 are proven.
+
+## Log
+
+- 2026-09-07 19:40 UTC — recon done, plan written, ssh to the new box verified as root with
+  `notplants-orchestrator-ed25519`.
+- 2026-09-07 20:05 UTC — nixos-infect started on 195.201.88.249 (rev 40f62a6, nixos-26.05,
+  PROVIDER=hetznercloud). Two false starts: the Debian 13 image has /tmp on tmpfs, so
+  nixos-infect's temp swapfile fails `swapon: Invalid argument`; fixed with `NO_SWAP=true`.
+  Build ran, box rebooted ~20:11 UTC and has not answered ping/ssh since (>25 min) — needs the
+  Hetzner console (no API token for that project on this host).
+- 2026-09-07 20:40 UTC — cc-ci branch `feat/nixos-module-export` (9b99f81) pushed: the standalone
+  `#cc-ci` drv is byte-identical before/after. Orchestrator branch `feat/combined-cc-ci-host`:
+  `#cc-ci` evaluates (gcnwq4fy…-nixos-system-cc-ci-26.05.20260803.531670d.drv) with PROVISIONAL
+  hardware/networking copied from the old CI server — to be replaced by the infect output.
+- 2026-09-07 20:30 UTC — rescue-mode diagnosis + fixes (README §2a); `#cc-ci` test → switch →
+  reboot OK on the new box; data restored incl. Drone volume; PRs cc-ci #32 + cc-ci-orchestrator
+  #19 merged. Quarantine on (bridge 0, timers runtime-masked, orchestrator units stopped).
+  Waiting on operator: new tailscale auth key; Gandi DNS cutover.
+- 2026-09-07 21:10 UTC — operator: NO tailscale on the new host; opencode UI behind its basic
+  auth on 443 (traefik labels on the `opencode-ui` socat relay → nginx on docker_gwbridge:8090);
+  fail2ban (sshd + nginx-http-auth in DOCKER-USER). Operator repointed Gandi (ci, *.ci, ns-acme)
+  → 195.201.88.249 ~20:40; propagated at once at public resolvers. CUTOVER DONE: old bridge +
+  drone scaled to 0, old-host timers masked, new bridge up; `!testme` on custom-html #7 = Drone
+  build 1326 on the new box. Two self-inflicted incidents, both documented in README: traefik
+  FILE_PROVIDER_DIRECTORY_ENABLED took the front doors down 2 min; the Hetzner resolvers in
+  networking.nix served the stale address (3 h TTL) → runner/bridge 404s until dropped.
+  `nixos-rebuild switch` = generation 4. The `switch` re-enabled cc-ci-orchestrator on the new
+  box: a fresh Claude orchestrator session runs there (tmux `cc-ci-orchestrator`, remote-control
+  label `cc-ci-orchestrator`) since 20:46 UTC; the old host's cc-ci units are runtime-masked and
+  the notplants-nix branch `chore/drop-cc-ci` (pushed, evaluates) removes them for good —
+  operator merges (that repo auto-deploys nightly).
+- Remaining: (9) delete old CI server 134485294 + its tailnet node after ~1 week; revoke the
+  Hetzner token; (10) autonomic.zone domains. Also re-key cc-ci-secrets to the new host's ssh host
+  key (optional; master key works) and re-run the cc-ci self-test build (1324/1325 failed during
+  the resolver mess).
+- 2026-09-07 21:20 UTC — notplants-nix `chore/drop-cc-ci` merged (9c785c1) and applied on the old
+  host (test → switch, generation 57, healthy, cc-ci units gone). cc-ci push self-test green again
+  after #33 (ruff format, pre-existing since 2026-08-31) + #34 (statix: acme-dns.nix systemd
+  attrset): builds 1330/1331 success. New host on generation 6 with cc-ci main 0437210.
@@ -137,3 +137,37 @@
  2.37.5 withdrawn). 2.36.9 holds the Stable/Latest badge; 2.37.x remains Pre-release on GitHub
  (consistent precedent). Re-verified 2.37.3→2.37.6 (pure core bugfixes), no breaking changes beyond
  the already-flagged 2.37.0 API behavior pair. Rolling upgrade safe. Recommended release: `-y`.
+- 2.37.7 (2026-09-01, patch): core/editor bugfixes (distroless runners glibc/libatomic copy; thread
+  execution id through dynamic-credential storage; scope active workflow IDs to user's projects;
+  route secrets pasted in AI chat to real credential forms; gate autosave on document hydration).
+- 2.37.8/2.37.9/2.37.10/2.37.11 (2026-09-02..04, patches): mostly core/editor/ai-builder bugfixes.
+  2.37.9 and 2.37.10 are noted on the Releases page; 2.37.11 = 1 core fix (thread execution id through
+  dynamic-credential storage backport). No breaking compose/env/migration changes.
+- 2.38.0 (2026-09-01, Pre-release): the 2.38 feature minor — a large feature/bugfix release (Confluence
+  page/space operations, Databricks user-delegated OAuth2, Moonshot/MiniMax/Qwen Cloud model providers
+  in Agent Builder, background sub-agent delegation, nonce-based Content-Security-Policy on HTML pages,
+  chat trigger publish validation + workflow:execute access control, restore mutating array methods on
+  $json in expressions, MongoDB bulkWrite, etc.). No breaking compose/env/migration changes; no
+  `N8N_*` env renames; the 2.37.0 API behavior pair (JSON content-type on decorator body routes;
+  binary-data endpoint adapt) still applies to HTTP API callers only.
+- 2.38.1 (2026-09-01, Pre-release): a large patch — bugfixes (mongodb sort/projection parameter
+  binding; S3 startup connection timeout; domain-restricted credential in own node; OAuth token grant
+  binding; restore mutating array methods on $json in expressions; etc.) + features (n8n credits
+  section in agent tools modal, Confluence page get comments/labels/space, Moonshot/MiniMax/Qwen Cloud
+  model providers, nonce-based CSP, split hosted chat page into trusted shell + sandboxed frame).
+  No breaking compose/config changes.
+- 2.38.2 (2026-09-02, Pre-release): core bugfix (restore ...) — see Releases page.
+- 2.38.3 (2026-09-0?, Pre-release): patch bugfixes (bundle backports to 2.38.x release track —
+  egress filtering, credential-destination grants, proxy-agent behavior, shared-workflow/user queries;
+  nonce-based CSP on HTML pages). No breaking compose/config changes.
+- 2.38.4 (2026-09-0?, Pre-release; **newest 2.38.x tag** — abra lists it as the top upgrade candidate):
+  patch bugfixes. No breaking compose/env/migration changes; rolling upgrade safe (sqlite, TypeORM
+  auto-migrate on boot). The 2.37.0 API behavior pair (JSON content-type on decorator body routes;
+  binary-data endpoint adapt) still applies — HTTP API callers only, no recipe impact.
+- 2026-09-07 run: PR #7 extended 2.34.4 → **2.38.4** (newest tag abra lists; 2.37.5/2.38.0-era withdrawn
+  tags skipped). 2.38.x line remains Pre-release on GitHub (consistent tracking-the-newest-tag
+  precedent: 2.34.2/2.35.2/2.36.3/2.37.3/2.37.6 were all Pre-release when taken). Rolling upgrade safe
+  (sqlite, TypeORM auto-migrate on boot). No breaking compose/config changes for the recipe. Operator
+  flag (unchanged from 2.37.0): the two API behavior changes only affect HTTP API callers. Recommended
+  release: `-y` (feature minor across 2.34.4 → 2.38.4). CVE-2026-73088 was adjudicated NOT-FIXED in the
+  2.x window by the 2026-09-04 parent run; not blocking.
@@ -1,12 +1,35 @@
 {
  "nodes": {
+    "cc-ci": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs"
+        ],
+        "sops-nix": [
+          "sops-nix"
+        ]
+      },
+      "locked": {
+        "lastModified": 1788820034,
+        "narHash": "sha256-mkktTMeGKutfPp3pn3AhoPoGSCm4iRSnPvpmbEpMmwA=",
+        "ref": "refs/heads/main",
+        "rev": "b11cc0b7385aee9fb89bd9a7ed23896401d683e5",
+        "revCount": 1543,
+        "type": "git",
+        "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.autonomic.zone/recipe-maintainers/cc-ci.git"
+      }
+    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1785734586,
-        "narHash": "sha256-ODZkEK9Gy50yg6h98u7KkitZ3oc/uuTFK00bh1CRdNA=",
+        "lastModified": 1788690626,
+        "narHash": "sha256-+v4I4LawmRD/mVxO7QIAerRrCkElp3YImzWkkUnvOTg=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "531670d871c0e29724a02f3cbcac170adc65b58c",
+        "rev": "c25784012c9982bca5b3e0de87e90bbdac8927d3",
        "type": "github"
      },
      "original": {
@@ -18,6 +41,7 @@
    },
    "root": {
      "inputs": {
+        "cc-ci": "cc-ci",
        "nixpkgs": "nixpkgs",
        "sops-nix": "sops-nix"
      }
@@ -29,11 +53,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1783174389,
-        "narHash": "sha256-aCWC8ngycU7OdJrU2+Je3qf+1a2ykuBvpPhZT/9tXMc=",
+        "lastModified": 1788337237,
+        "narHash": "sha256-gkSH8VUtCo6hnysNmb9DbTuDepH2t5pv+QWjP75xKAk=",
        "owner": "Mic92",
        "repo": "sops-nix",
-        "rev": "f1406619a3884cd5c47992a70b8b35c9c0fcb4c9",
+        "rev": "fbf759290e0cb0a98dfc813a4eb7d53ad1dacb57",
        "type": "github"
      },
      "original": {
@@ -1,37 +1,55 @@
 {
-  description = "cc-ci-orchestrator — NixOS host for the cc-ci loops runtime (Builder/Adversary/Watchdog)";
+  description = "cc-ci-orchestrator — the cc-ci orchestrator (loops, steering session, weekly upgrader) and the NixOS host it shares with the cc-ci CI server";

  inputs = {
-    # Follow the current stable release channel (operator 2026-08-01), was a hard rev pin at
-    # nixpkgs 24.11 (50ab7937, 2025-06-30) kept "the same as the cc-ci server". This host runs
-    # agents/tmux/nginx/docker, not recipe CI, so it does not need to match that server — and a
-    # frozen rev only accrues unpatched CVEs. `nix flake update` now actually moves.
+    # Stable release channel (operator 2026-08-01). `nix flake update` moves it; the cc-ci input
+    # below FOLLOWS it, so one nixpkgs builds the whole combined host and CVEs get patched once.
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

-    # sops-nix follows nixpkgs below, so it no longer needs its own matching pin.
    sops-nix.url = "github:Mic92/sops-nix";
    sops-nix.inputs.nixpkgs.follows = "nixpkgs";
+
+    # The cc-ci CI server, as a NixOS module (`nixosModules.cc-ci-server`). HTTPS, anonymous read:
+    # nix evaluates every input for every output, so the input must be fetchable without
+    # credentials. The private secrets submodule is deliberately NOT fetched through this input —
+    # the host reads the deployed --recursive checkout's secrets.yaml at activation instead
+    # (`cc-ci.sopsFile`). Both `follows` are REQUIRED: without them cc-ci's own nixpkgs/sops-nix
+    # pins would produce a second sops-nix module tree and a second nixpkgs in one system.
+    cc-ci.url = "git+https://git.autonomic.zone/recipe-maintainers/cc-ci.git";
+    cc-ci.inputs.nixpkgs.follows = "nixpkgs";
+    cc-ci.inputs.sops-nix.follows = "sops-nix";
  };

-  outputs = { nixpkgs, sops-nix, ... }:
+  outputs = { self, nixpkgs, sops-nix, cc-ci, ... }:
    let
      system = "x86_64-linux";
    in
    {
-      # The cc-ci part of a host, on its own, so a host that runs cc-ci can import just this and
-      # keep its own (unrelated) configuration separate. Split out 2026-08-20; consumed by
-      # notplants-nix's `notplants-orchestrator` host.
-      nixosModules.cc-ci = ./nix/modules/cc-ci.nix;
+      nixosModules = {
+        # The orchestrator itself: loops supervisor, steering session, weekly/hourly timers.
+        cc-ci-orchestrator = ./nix/modules/cc-ci.nix;
+        # The host contract those units assume: loops user, claude/opencode CLIs, opencode web
+        # server + tailnet UI, nix-ld, tool set, `ssh cc-ci` config.
+        orchestrator-host = ./nix/modules/orchestrator-host.nix;
+        # Weekly health-gated self-update of the host (busy-gated around CI runs).
+        auto-update = ./nix/modules/auto-update.nix;
+        # Old name of cc-ci-orchestrator, kept while notplants-nix still imports it (2026-09).
+        cc-ci = ./nix/modules/cc-ci.nix;
+      };

-      # Hetzner cpx11 host (nixos-infect generated hardware.nix + orchestrator config).
-      # Provision with terraform/ then run Stage 2 per terraform/README.md.
-      nixosConfigurations.cc-ci-orchestrator-hetzner = nixpkgs.lib.nixosSystem {
-        inherit system;
-        modules = [
-          sops-nix.nixosModules.sops
-          ./nix/hosts/cc-ci-orchestrator-hetzner/hardware.nix
-          ./nix/hosts/cc-ci-orchestrator-hetzner/configuration.nix
-        ];
+      nixosConfigurations = {
+        # THE live host: cc-ci CI server + cc-ci orchestrator on one Hetzner cpx32-class box
+        # (195.201.88.249, since 2026-09). README.md is the deploy guide.
+        cc-ci = nixpkgs.lib.nixosSystem {
+          inherit system;
+          modules = [
+            cc-ci.nixosModules.cc-ci-server
+            self.nixosModules.cc-ci-orchestrator
+            self.nixosModules.orchestrator-host
+            self.nixosModules.auto-update
+            ./nix/hosts/cc-ci/configuration.nix
+          ];
+        };
      };
    };
 }
@@ -10,9 +10,10 @@ metadata:
 The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx22** as of
 2026-05-31, replacing the Incus VM (100.116.55.106).

- Hetzner server **134487234**, public **168.119.126.100**, tailnet **cc-ci-orchestrator-1** @
-  **100.84.190.30**. Flake host **cc-ci-orchestrator-hetzner**.
- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci-orchestrator-hetzner` from `/srv/cc-ci-orch`
+- Since 2026-09-07: ONE Hetzner host for CI server + orchestrator, public **195.201.88.249**,
+  tailnet **cc-ci**, flake host **`.#cc-ci`** (this repo). Before: orchestrator on Hetzner
+  134487234 (168.119.126.100 / 100.84.190.30, `cc-ci-orchestrator-hetzner`), shared with notplants.
+- Rebuild: `sudo nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch`
  (`/srv/cc-ci` is a symlink to it). The Bash tool runs as user **loops** (uid 1000, passwordless
  sudo) — plain `nixos-rebuild switch` fails on the profile symlink; use `sudo`.
 - Reboot-resilience: `cc-ci-loops.service` is **enabled** (wantedBy multi-user.target); ExecStartPre
@@ -23,4 +24,4 @@ The cc-ci orchestrator (loops + watchdog + this session) runs on a **Hetzner cpx
  identity unknown". Set per-repo to match prior commits: `autonomic-bot
  <autonomic-bot@git.autonomic.zone>`.

-Full record: `cc-ci-plan/plan-orchestrator-hetzner-migration.md`.
+Full record: `archive/plans/plan-orchestrator-hetzner-migration.md`.
@@ -0,0 +1,144 @@
+# cc-ci — ONE Hetzner Cloud host running both the cc-ci CI server and the cc-ci orchestrator.
+#
+# This file is only what is physical or identity about the machine: hardware, networking,
+# root SSH keys, firewall + fail2ban, swap, stateVersion. Everything functional comes from modules:
+#   cc-ci.nixosModules.cc-ci-server        recipe-maintainers/cc-ci — swarm, traefik, drone,
+#                                          runner, bridge, dashboard, reports, acme-dns, harness
+#   self.nixosModules.cc-ci-orchestrator   nix/modules/cc-ci.nix — loops, orchestrator, timers
+#   self.nixosModules.orchestrator-host    nix/modules/orchestrator-host.nix — loops user, CLIs
+# See README.md for provisioning (Hetzner Debian → nixos-infect → this flake) and staging.
+{ lib, pkgs, ... }:
+{
+  imports = [
+    ./hardware.nix
+    ./networking.nix
+  ];
+
+  networking.hostName = "cc-ci";
+
+  # ---- cc-ci server identity --------------------------------------------------------------
+  # Public address: acme-dns binds to it and publishes it as the `ns-acme` glue record; the
+  # Gandi A records for ci / *.ci / ns-acme .commoninternet.net point here.
+  cc-ci.publicIPv4 = "195.201.88.249";
+  # cc-ci is a plain flake input here (no private submodule), so the sops file is the one in
+  # the deployed --recursive checkout the weekly sweep runs from (README "Stage the workspace").
+  cc-ci.sopsFile = "/etc/cc-ci/secrets/secrets.yaml";
+
+  # ---- orchestrator identity --------------------------------------------------------------
+  # The CI server is this very host, so `ssh cc-ci` goes to loopback (the module default).
+  cc-ci-orchestrator.ciSshHost = "127.0.0.1";
+  # Weekly self-update (Tue 03:00 UTC; skips itself while CI is busy; see nix/modules/auto-update.nix).
+  cc-ci-orchestrator.autoUpdate.enable = true;
+
+  # The opencode UI: traefik (public 443, the *.ci.commoninternet.net cert) → nginx basic auth.
+  cc-ci-orchestrator.opencodeUiHost = "oc.ci.commoninternet.net";
+  cc-ci-orchestrator.opencodeUiTraefikNetwork = "proxy";
+
+  # ---- no tailscale on this host (operator 2026-09-07) --------------------------------------
+  # Everything is reached over the public IP: ssh (keys only), the CI front doors via traefik,
+  # and the opencode UI on 443 (traefik → nginx basic auth). fail2ban below guards the two logins.
+
+  # ---- /secrets is THE authoritative location for this host's secret material ---------------
+  # Operator rule (2026-09-08): every secret lives under /secrets; anything that needs one either
+  # reads it from there directly (where we own the path) or reaches it by a symlink (where the
+  # consuming software fixes the path). One directory to audit, back up, and reason about.
+  #
+  #   /secrets/files   loops:users 0700  the agent's secrets (testenv, opencode auth, its ssh keys)
+  #   /secrets/host    root:root   0700  host identity: ssh host keys + the sops age identity
+  #   /secrets/nginx   root:nginx  0750  the opencode UI htpasswd (nginx must read it)
+  #
+  # /secrets itself is 0711: traversable so nginx can reach its own subdirectory, not listable.
+  systemd.tmpfiles.rules = [
+    "d /secrets       0711 root root  -"
+    "d /secrets/host  0700 root root  -"
+    "d /secrets/nginx 0750 root nginx -"
+    # Convenience symlinks at the conventional paths, so an operator (or a tool that assumes the
+    # usual location) still finds the host keys. NOT load-bearing: sshd and sops-nix below are
+    # pointed at /secrets directly, precisely so nothing depends on symlink/activation ordering.
+    "L+ /etc/ssh/ssh_host_ed25519_key     - - - - /secrets/host/ssh_host_ed25519_key"
+    "L+ /etc/ssh/ssh_host_ed25519_key.pub - - - - /secrets/host/ssh_host_ed25519_key.pub"
+    "L+ /etc/ssh/ssh_host_rsa_key         - - - - /secrets/host/ssh_host_rsa_key"
+    "L+ /etc/ssh/ssh_host_rsa_key.pub     - - - - /secrets/host/ssh_host_rsa_key.pub"
+  ];
+
+  # sops-nix: the cc-ci server module hard-codes /etc/ssh/... and /var/lib/sops-nix/key.txt.
+  # Override both to the authoritative copies. THE ED25519 HOST KEY IS LOAD-BEARING BEYOND SSH:
+  # its age identity (age1tmvg…) is a recipient of cc-ci-secrets, so replacing or regenerating it
+  # makes every cc-ci secret undecryptable. Move it, never re-create it.
+  sops.age.sshKeyPaths = lib.mkForce [ "/secrets/host/ssh_host_ed25519_key" ];
+  sops.age.keyFile = lib.mkForce "/secrets/host/sops-age-key.txt";
+
+  # ---- ssh ----------------------------------------------------------------------------------
+  services.openssh = {
+    enable = true;
+    settings.PermitRootLogin = "yes";
+    # Host keys live in /secrets (above). sshd is pointed here directly rather than through the
+    # /etc/ssh symlinks, so it can never write a NEW key through a dangling link — that would
+    # silently rotate the age identity that decrypts cc-ci-secrets.
+    hostKeys = [
+      { path = "/secrets/host/ssh_host_ed25519_key"; type = "ed25519"; }
+      { path = "/secrets/host/ssh_host_rsa_key"; type = "rsa"; bits = 4096; }
+    ];
+  };
+  # Root keys: PUBLIC keys, tracked deliberately in ./ssh-keys (one per line, blank lines ok).
+  users.users.root.openssh.authorizedKeys.keys =
+    builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+  # The loops user can also be reached directly (same keys) — handy for rsync of its workspace.
+  users.users.loops.openssh.authorizedKeys.keys =
+    builtins.filter (s: s != "") (lib.splitString "\n" (builtins.readFile ./ssh-keys));
+  services.openssh.settings.PasswordAuthentication = false;
+  services.openssh.settings.KbdInteractiveAuthentication = false;
+
+  # ---- firewall -------------------------------------------------------------------------------
+  # 80/443 (traefik) and 53 (acme-dns) are opened by the cc-ci-server module; the opencode UI
+  # rides 443 through traefik (orchestrator-host.nix), so only ssh is opened here.
+  networking.firewall = {
+    enable = true;
+    allowedTCPPorts = [ 22 ];
+  };
+
+  # ---- fail2ban: sshd (password auth is off, this stops the log noise and slow brute force) and
+  # the opencode UI's basic auth (nginx logs 401s with the real client IP to the journal; the
+  # built-in nginx-http-auth filter matches them). Those clients arrive through traefik's
+  # docker-published 443, which iptables FORWARDs rather than INPUTs, so the ban for that jail
+  # goes into the DOCKER-USER chain — an INPUT rule would never see the traffic.
+  services.fail2ban = {
+    enable = true;
+    maxretry = 5;
+    bantime = "1h";
+    bantime-increment = { enable = true; maxtime = "48h"; factor = "4"; };
+    ignoreIP = [ "127.0.0.0/8" "::1" ];
+    jails.nginx-http-auth.settings = {
+      enabled = true;
+      filter = "nginx-http-auth";
+      # NixOS nginx logs errors to stderr → the journal, not /var/log/nginx/error.log (which
+      # exists but stays empty). Read the unit's journal instead.
+      backend = "systemd";
+      journalmatch = "_SYSTEMD_UNIT=nginx.service";
+      banaction = "iptables-allports";
+      chain = "DOCKER-USER";
+    };
+  };
+
+  networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
+  # This host's own public names resolve to itself regardless of external DNS state (host
+  # processes: the drone runner, the harness, the orchestrator; containers use the resolvers
+  # above). Per-run recipe domains are random and cannot be pinned — those follow public DNS.
+  networking.hosts."195.201.88.249" = [
+    "ci.commoninternet.net"
+    "drone.ci.commoninternet.net"
+    "report.ci.commoninternet.net"
+    "traefik.ci.commoninternet.net"
+    "warm-keycloak.ci.commoninternet.net"
+    "oc.ci.commoninternet.net"
+  ];
+
+  # ---- memory: 8 GB RAM shared by the swarm (recipe deploys) and 3–6 agent sessions ---------
+  swapDevices = [ { device = "/swapfile"; size = 8192; } ];
+
+  # ssh client for root (the orchestrator's `ssh cc-ci` goes through the loops user's own config).
+  environment.systemPackages = [ pkgs.openssh ];
+
+  # Fresh NixOS 26.05 install (nixos-infect, 2026-09-07). Never change this on an existing host.
+  system.stateVersion = "26.05";
+}
@@ -0,0 +1,19 @@
+# Generated by nixos-infect on this machine (2026-09-07), captured verbatim per README §3.
+# The ESP UUID is specific to THIS server; a new server gets a new file.
+{ modulesPath, ... }:
+{
+  imports = [ (modulesPath + "/profiles/qemu-guest.nix") ];
+  boot.loader = {
+    efi.efiSysMountPoint = "/boot/efi";
+    grub = {
+      efiSupport = true;
+      efiInstallAsRemovable = true;
+      device = "nodev";
+    };
+  };
+  fileSystems."/boot/efi" = { device = "/dev/disk/by-uuid/E079-7D41"; fsType = "vfat"; };
+  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "xen_blkfront" "vmw_pvscsi" ];
+  boot.initrd.kernelModules = [ "nvme" ];
+  fileSystems."/" = { device = "/dev/sda1"; fsType = "ext4"; };
+  
+}
@@ -0,0 +1,40 @@
+# Generated by nixos-infect on this machine (2026-09-07), captured per README §3, with ONE edit:
+# `defaultGateway` as an attrset WITH `interface = "eth0"`. The generated bare-string form leaves
+# NixOS ≥25.05 without a default route (the host boots and is unreachable) — see README §2.
+{ lib, ... }: {
+  # This file was populated at runtime with the networking
+  # details gathered from the active system.
+  networking = {
+    # The Hetzner resolvers nixos-infect wrote here are deliberately gone: configuration.nix sets
+    # 1.1.1.1/8.8.8.8. During the 2026-09-07 cutover the Hetzner caches kept the OLD address for
+    # the full 3h TTL while the public resolvers already had the new one, and every other lookup
+    # of drone.ci.commoninternet.net from this box went to the old server (404s in the runner and
+    # the bridge). One consistent resolver set, and the host's own names pinned in
+    # configuration.nix, keep the CI independent of that.
+    defaultGateway = { address = "172.31.1.1"; interface = "eth0"; };
+    defaultGateway6 = {
+      address = "fe80::1";
+      interface = "eth0";
+    };
+    dhcpcd.enable = false;
+    usePredictableInterfaceNames = lib.mkForce false;
+    interfaces = {
+      eth0 = {
+        ipv4.addresses = [
+          { address="195.201.88.249"; prefixLength=32; }
+        ];
+        ipv6.addresses = [
+          { address="2a01:4f8:1c1c:a9b::1"; prefixLength=64; }
+{ address="fe80::2ff8:e3ea:bbb8:aa39"; prefixLength=64; }
+        ];
+        ipv4.routes = [ { address = "172.31.1.1"; prefixLength = 32; } ];
+        ipv6.routes = [ { address = "fe80::1"; prefixLength = 128; } ];
+      };
+      
+    };
+  };
+  services.udev.extraRules = ''
+    ATTR{address}=="92:00:09:d5:ec:0d", NAME="eth0"
+    
+  '';
+}
@@ -0,0 +1,11 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGZGp/DQTFuD1GvsyTzCVBUTmoWqcb5T+Z7zZo5nYLXO
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhgo41nt8/L+Cr0PKd8jQK45mw/A+h041j6LQ8JWZisEVaQOzr6s9rxPL8VT5ML4P3/4bMblzdDiXWlJxymcb+yk5S5TnVrMavzHEDhWHwEvTRMe6xNTmsU6cmmhRw7PJqqQ+0GTlQalu3I4jkC0kTF7kuPwduUOgUuSpJqxvDTwYiXoyVnOQHAIygh+BmQvYUz0PBfQgIhgcbYmGZ++T0DnMzdGFzW2UB/iy5mymnpmbaZCgLy0w8AoDE+0YLtUc4gwTXc183nvqO1i7LQr+3jBYkv5ZthCCc52vXFHDSw9xZ5ohsOrBvoi5foRbqinmU5/t0aTK7SSrat7xXm/odIOyS+S7PJyeEcsXN6d5zdxbabAy5vLfodEaKGZd4rqQeDCxOTPAS/BlrBV/EV714n4E+fSOAllAuMBO4IibJM/gLJrh2Dql3co50QW9HEDeSC7iqp2lxRBDxvUs3rIEzy7o4HSN8chqBUK1bbBY6B17fuNHIpBAw4akRVVvPnVM= trav@trav480sweet
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6jrKj7iZUNRLBTZG0vZM1D/BXtARhhB4+GrvpyuqmPb9iw2ifT9YqRUwgyGrOW9U6nIAR9yFnfp9+FkyhEKWByqEBbe/zYKlGLRGjfsIdDdW29QQ3hvmqNyboCkXLxZGat93poYhnoomqicmGD/xST4s0OUhcK9E494lUmenlD9dcMZW1aKpJ+9O4Dq6A7nk2z1e4KFcZdrZDI2Hgg+gfEdsKZQqd/R3Mls/eVKpzhfv3Y8BiNoHssUChVf8IGESqTOBOR7Dk7FsU5Z2ZcnQ1coxY7VlBn4fPjTWmz/Ac0jLqgcpCLpNyQzFPDVMYZKYrPVoqBeKVhN5YnfwR5OVP8YsakT/obLwC43sx/esXfjhVGcsRoGpiLOfazzNw/eC8s6FlS8cesOubEM37a7F25z4UEG3d487oM7EjQ39gBCCj/KRgUimCKMWsm6yIas4OSctBWEAo/NhZp0gwulSRxleW6eJCNNwzOmWjdzYIVWoVP0EIeM95Tq8PVUN7gpc= aadil@t480
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMyHSi12R0+HCVBz7+d9fyOBnoJi8Nsj5D7vQ9UQO8a5
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJVlfoLBPseQ9fA9534KmRg2KWcksKZGzAJIpHJ2JpsI mfowler.email@protonmail.com
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAQFuqUB2qNZSDNjDsjjhVA/WnnQNVAMmsUscW6OgMDN
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHOcLo0YBa0UYi7i/l8K/Y/7cF2OclmDqSTlAsHM0dOS notplants-orchestrator
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMniNzAzuI527bfk/EipqFILFayUCwYXDoZ3R7+QgYq6
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOk8NaeBdPbS2gfUvbny8h0AkZlVjGYHzx4QPXSJ38gd claude@claude-vm
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAcyTGb/wVgdhg5oBCZZvBaR1RuUQRY/3WHnOQpNDCsp claude-cc-ci-sandbox@20260526
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKmGDZC6wrOQNJAW5PPDpxgEXXrcsnIU4b3QJLtq05RQ cc-ci-loops-to-root@cc-ci
@@ -0,0 +1,199 @@
+# auto-update.nix — weekly, health-gated self-update of the combined cc-ci host.
+#
+# Modelled on notplants-nix's modules/auto-update.nix (the nightly updater the old orchestrator
+# box had), with the two things the cc-ci host needs on top:
+#   * a BUSY GATE — a `switch` restarts docker/traefik/drone/the runner, so the update skips
+#     (and simply retries next week) while a CI run, the weekly recipe-upgrade run or the
+#     Sunday canonical sweep is in flight;
+#   * cc-ci HEALTH CHECKS — beyond "sshd + no failed units": every swarm service at its replica
+#     count, the front doors (dashboard, reports, drone, the opencode UI) answering, sops having
+#     decrypted, acme-dns and fail2ban up.
+#
+# Sequence (as in the notplants module — read its comments for the whys):
+#   flake update (all inputs: nixpkgs, sops-nix, cc-ci) → nixos-rebuild build (as loops) →
+#   switch-to-configuration test (bootloader untouched) → settle → health check →
+#   on failure: re-activate the previous generation, restore flake.lock, exit 1
+#   on success: set the system profile + bootloader, commit flake.lock as "auto-update", push
+#   main (the repo stays the source of truth), refresh /etc/cc-ci.
+# Nothing is committed that did not pass the health check. Never fires a missed run at boot.
+# `/cc-ci-status` reads the outcome from the state file this writes.
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.cc-ci-orchestrator.autoUpdate;
+  builderHome = config.users.users.${cfg.buildUser}.home;
+  stateFile = "${cfg.flakePath}/.cc-ci-logs/auto-update-state";
+  curl = "${pkgs.curl}/bin/curl";
+
+  busyGate = pkgs.writeShellScript "cc-ci-auto-update-busy-gate" ''
+    set -u
+    busy() { echo "BUSY: $1 — skipping this week's update"; exit 3; }
+    ${pkgs.procps}/bin/pgrep -f run_recipe_ci >/dev/null && busy "a CI run is in flight"
+    ${pkgs.systemd}/bin/systemctl is-active --quiet nightly-sweep.service && busy "the canonical sweep is running"
+    ${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-upgrader 2>/dev/null \
+      && busy "the weekly recipe-upgrade run is in flight (tmux cc-ci-upgrader)"
+    ${pkgs.util-linux}/bin/runuser -u ${cfg.buildUser} -- ${pkgs.tmux}/bin/tmux has-session -t cc-ci-report 2>/dev/null \
+      && busy "the weekly report is being written (tmux cc-ci-report)"
+    if [ -r /run/secrets/bridge_drone_token ]; then
+      running=$(${curl} -s -m 20 -H "Authorization: Bearer $(cat /run/secrets/bridge_drone_token)" \
+        "https://drone.${cfg.ciDomain}/api/repos/recipe-maintainers/cc-ci/builds?per_page=10" \
+        | ${pkgs.gnugrep}/bin/grep -o '"status":"running"' | ${pkgs.coreutils}/bin/wc -l)
+      [ "''${running:-0}" -eq 0 ] || busy "$running Drone build(s) running"
+    fi
+    echo "not busy"
+  '';
+
+  healthCheck = pkgs.writeShellScript "cc-ci-auto-update-health-check" ''
+    set -u
+    fail() { echo "HEALTH FAIL: $1"; exit 1; }
+    ${pkgs.systemd}/bin/systemctl is-active --quiet sshd || fail "sshd not active"
+    ${pkgs.iproute2}/bin/ss -tlnH | ${pkgs.gnugrep}/bin/grep -q ':22 ' || fail "nothing listening on :22"
+    n=$(${pkgs.systemd}/bin/systemctl --failed --no-legend | ${pkgs.coreutils}/bin/wc -l)
+    [ "$n" -eq 0 ] || fail "$n failed unit(s): $(${pkgs.systemd}/bin/systemctl --failed --no-legend --plain | ${pkgs.gawk}/bin/awk '{print $1}' | ${pkgs.coreutils}/bin/tr '\n' ' ')"
+    for u in docker acme-dns fail2ban nginx opencode-web drone-runner-exec; do
+      ${pkgs.systemd}/bin/systemctl is-active --quiet "$u" || fail "$u not active"
+    done
+    # The reconcile oneshots re-run on activation; give the swarm up to 10 minutes to converge.
+    for i in $(${pkgs.coreutils}/bin/seq 1 40); do
+      short=$(${pkgs.docker}/bin/docker service ls --format '{{.Name}} {{.Replicas}}' \
+        | ${pkgs.gawk}/bin/awk '{ split($2,a,"/"); if (a[1] != a[2]) print $1 }')
+      [ -z "$short" ] && break
+      ${pkgs.coreutils}/bin/sleep 15
+    done
+    [ -z "$short" ] || fail "swarm services not at their replica count: $(echo "$short" | ${pkgs.coreutils}/bin/tr '\n' ' ')"
+    [ -s /run/secrets/test_secret ] || fail "sops did not decrypt (/run/secrets/test_secret missing)"
+    code() { ${curl} -s -m 20 -o /dev/null -w '%{http_code}' --resolve "$1:443:127.0.0.1" "https://$1/$2"; }
+    [ "$(code ${cfg.ciDomain} "")" = 200 ] || fail "dashboard not 200"
+    [ "$(code report.${cfg.ciDomain} "")" = 200 ] || fail "reports not 200"
+    [ "$(code drone.${cfg.ciDomain} version)" = 200 ] || fail "drone /version not 200"
+    [ "$(code ${cfg.opencodeUiHost} "")" = 401 ] || fail "opencode UI not answering with its auth challenge"
+    echo "HEALTH OK"
+  '';
+in
+{
+  options.cc-ci-orchestrator.autoUpdate = {
+    enable = lib.mkEnableOption "weekly health-gated self-update of the cc-ci host";
+    flakePath = lib.mkOption {
+      type = lib.types.str;
+      default = "/srv/cc-ci-orch";
+      description = "The cc-ci-orchestrator checkout (owned by buildUser) whose flake.lock is updated, committed and pushed.";
+    };
+    flakeRef = lib.mkOption {
+      type = lib.types.str;
+      default = "${cfg.flakePath}#cc-ci";
+      description = "Flake reference to build.";
+    };
+    buildUser = lib.mkOption {
+      type = lib.types.str;
+      default = "loops";
+      description = "Owner of the checkout; runs the update, build, commit and push (root activates).";
+    };
+    onCalendar = lib.mkOption {
+      type = lib.types.str;
+      default = "Tue *-*-* 03:00:00 UTC";
+      description = ''
+        When to run. Must stay clear of the weekly recipe-upgrade run (Thu 22:00 America/New_York
+        = Fri 02:00/03:00 UTC, several hours) and the Sunday 03:00 UTC canonical sweep (up to 6 h);
+        the busy gate covers the rest.
+      '';
+    };
+    remote = lib.mkOption { type = lib.types.str; default = "origin"; };
+    branch = lib.mkOption { type = lib.types.str; default = "main"; };
+    ciDomain = lib.mkOption {
+      type = lib.types.str;
+      default = "ci.commoninternet.net";
+      description = "The CI apex; report./drone. are checked under it.";
+    };
+    opencodeUiHost = lib.mkOption {
+      type = lib.types.str;
+      default = config.cc-ci-orchestrator.opencodeUiHost;
+      description = "The opencode UI host name (expected to answer 401).";
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    systemd.services.cc-ci-auto-update = {
+      description = "Weekly cc-ci host self-update (flake update → build → test → health check → commit)";
+      after = [ "network-online.target" "docker.service" ];
+      wants = [ "network-online.target" ];
+      path = with pkgs; [ nixos-rebuild nix git openssh coreutils gnugrep gawk systemd util-linux bash python3 docker curl ];
+      serviceConfig = { Type = "oneshot"; TimeoutStartSec = "3h"; };
+      # A unit that performs the switch must never be stopped or restarted BY that switch
+      # (notplants-orchestrator, 2026-09-07: it was, and the host ran half-switched for 12 h).
+      stopIfChanged = false;
+      restartIfChanged = false;
+      script = ''
+        set -u
+        cd ${cfg.flakePath}
+        as_builder() { runuser -u ${cfg.buildUser} -- env HOME=${builderHome} "$@"; }
+        state() { printf '%s result=%s generation=%s note=%s\n' "$(date -u +%FT%TZ)" "$1" "$2" "$3" > ${stateFile}; chown ${cfg.buildUser} ${stateFile} || true; }
+        PREV=$(readlink -f /run/current-system)
+        LOCK_BAK=${cfg.flakePath}/.flake.lock.auto-update-prev
+
+        if ! ${busyGate}; then
+          state skipped "$PREV" busy
+          exit 0
+        fi
+
+        revert() {
+          echo "REVERT: $1"
+          as_builder cp "$LOCK_BAK" flake.lock 2>/dev/null || true
+          "$PREV"/bin/switch-to-configuration test || echo "re-activation of $PREV failed"
+          state failed "$PREV" "$1"
+          exit 1
+        }
+
+        as_builder cp flake.lock "$LOCK_BAK"
+        echo "=== checkout at: $(as_builder git rev-parse --short HEAD) on $(as_builder git rev-parse --abbrev-ref HEAD)"
+        echo "=== flake update (all inputs)"
+        as_builder nix flake update || revert "flake update failed"
+        if as_builder git diff --quiet -- flake.lock; then
+          echo "=== nothing moved; done"
+          state unchanged "$PREV" "lock already current"
+          exit 0
+        fi
+        as_builder git --no-pager diff --stat -- flake.lock
+
+        echo "=== build (cannot affect the running system)"
+        as_builder nixos-rebuild build --flake ${cfg.flakeRef} || revert "build failed"
+        NEW=$(readlink -f result) || revert "no build result"
+        as_builder rm -f result
+        echo "=== built: $NEW"
+
+        echo "=== activate WITHOUT touching the bootloader"
+        "$NEW"/bin/switch-to-configuration test || revert "test activation failed"
+
+        echo "=== settle, then health check"
+        sleep 45
+        ${healthCheck} || revert "health check failed"
+
+        echo "=== healthy: committing (system profile + bootloader)"
+        nix-env -p /nix/var/nix/profiles/system --set "$NEW" || revert "could not set system profile"
+        "$NEW"/bin/switch-to-configuration boot || revert "bootloader update failed"
+        state ok "$NEW" "committed"
+
+        # Bookkeeping from here on — never revert a healthy, committed generation over it.
+        if as_builder git -c user.name="auto-update" -c user.email="auto-update@cc-ci" \
+             commit -q -m "flake.lock: weekly auto-update, health-checked" -- flake.lock; then
+          echo "=== flake.lock committed: $(as_builder git rev-parse --short HEAD)"
+          as_builder git push -q ${cfg.remote} HEAD:${cfg.branch} \
+            && echo "=== pushed ${cfg.branch}" \
+            || echo "WARNING: push failed; the lock is committed locally, push it by hand"
+        else
+          echo "WARNING: could not commit flake.lock; the system IS committed and healthy"
+        fi
+        # Keep the deployed cc-ci checkout (sweep + sops file) on the same cc-ci rev as the flake input.
+        git -C /etc/cc-ci pull -q --ff-only --recurse-submodules || echo "WARNING: /etc/cc-ci pull failed"
+        echo "=== auto-update committed: $NEW"
+      '';
+    };
+
+    systemd.timers.cc-ci-auto-update = {
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        OnCalendar = cfg.onCalendar;
+        Persistent = false;
+        RandomizedDelaySec = "10min";
+      };
+    };
+  };
+}
@@ -1,22 +1,23 @@
-# cc-ci.nix — everything on this host that exists FOR cc-ci, and nothing else.
+# cc-ci.nix — the cc-ci ORCHESTRATOR: the Builder/Adversary loops supervisor, the operator's
+# steering session, and the weekly-upgrade + hourly-supervisor timers. Nothing else.
 #
-# Split out of the orchestrator host config on 2026-08-20. The host it runs on is a general
-# agent/orchestration box that also serves several unrelated projects; this module is the cc-ci
-# part of it, so that the two can evolve (and be reviewed) independently. It is exported from this
-# repo's flake as `nixosModules.cc-ci` and imported by whichever host runs cc-ci.
+# Exported from this repo's flake as `nixosModules.cc-ci-orchestrator` (and, for the host that
+# used to import it under the old name, `nixosModules.cc-ci`). Split out of the shared agent
+# host config on 2026-08-20; since 2026-09 it runs on the same Hetzner host as the CI server
+# itself (`#cc-ci` in flake.nix), next to recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
 #
 # All of it assumes the cc-ci workspaces exist on the host:
-#   /srv/cc-ci        the loops workspace (+ .cc-ci-logs, upgrader.env)
-#   /srv/cc-ci-orch   this repo (the orchestrator's own working dir)
+#   /srv/cc-ci        the loops workspace (+ .cc-ci-logs, upgrader.env) — a symlink to
+#   /srv/cc-ci-orch   this repo (the orchestrator's own working dir), with cc-ci/ checked out
 # and that a `loops` user, tmux, python3 and the standalone claude/opencode CLIs are present —
-# those are host concerns, provided by the host config, not by this module.
+# those are host concerns, provided by nix/modules/orchestrator-host.nix, not by this module.
 { config, pkgs, lib, ... }:
 {
  # cc-ci-loops supervisor — workspace staged 2026-05-31, so ENABLED for reboot-resilience.
  systemd.services.cc-ci-loops = {
    description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
    wantedBy = [ "multi-user.target" ];  # enabled after workspace staged (Hetzner cutover)
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    after = [ "network-online.target" "opencode-install.service" ];
    wants = [ "network-online.target" ];
    serviceConfig = {
      # KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
@@ -31,14 +32,14 @@
      # Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
      ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
    };
-    # CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
-    # backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
-    # `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
-    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    # The loops backend is persisted in .cc-ci-logs/.loop-backend (opencode on the cc-ci host —
+    # no Claude there since 2026-09-07). The build phases are all DONE; on boot launch.sh just
+    # re-confirms that and exits.
+    environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; };
    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
    script = ''
-      # Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
-      # tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
+      # Put the standalone opencode binary on PATH. On a cold boot this is the env the tmux
+      # server (and thus every agent session) inherits.
      export PATH="/home/loops/.local/bin:$PATH"
      [ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
        echo "workspace not staged yet — skipping loop start"
@@ -48,17 +49,16 @@
  # cc-ci-orchestrator supervisor — the operator's steering session. Same shape as
  # lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
  # tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
-  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml on
-  # the OPencode backend (backend = "opencode", model = "opencode/glm-5.2"), so on boot it
-  # attaches to the shared opencode web server (opencode-web.service below) and is reachable for
-  # Remote Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project. The harness
-  # watchdog (started by `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent
-  # supervisor must not be killed just for idling). Added 2026-08-03 to give the cc-ci orchestrator
-  # the same reboot-resilience the other two orchestrators already have.
+  # session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
+  # (backend/model chosen there: an opencode agent, opencode/glm-5.2, attached to the shared
+  # opencode web server — opencode-web.service in orchestrator-host.nix — and steered from its
+  # web UI; no Claude on this host, operator 2026-09-07). The harness watchdog (started by
+  # `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
+  # be killed just for idling). Added 2026-08-03 for reboot-resilience.
  systemd.services.cc-ci-orchestrator = {
-    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator, opencode backend";
+    description = "cc-ci orchestrator (operator steering session) — agents.py up orchestrator";
    wantedBy = [ "multi-user.target" ];
-    after = [ "network-online.target" "tailscaled.service" "opencode-web.service" ];
+    after = [ "network-online.target" "opencode-web.service" ];
    wants = [ "network-online.target" ];
    serviceConfig = {
      # KillMode=process: see the note on cc-ci-loops — a rebuild that merely touches this unit
@@ -85,7 +85,7 @@
  # wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
  systemd.services.cc-ci-upgrade-all = {
    description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
-    after = [ "network-online.target" "tailscaled.service" "claude-install.service" ];
+    after = [ "network-online.target" "opencode-install.service" ];
    wants = [ "network-online.target" ];
    serviceConfig = {
      Type = "oneshot";  # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
@@ -103,7 +103,7 @@
      # timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
      EnvironmentFile = "-/srv/cc-ci/upgrader.env";
    };
-    environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
+    environment = { HOME = "/home/loops"; };
    path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
    script = ''
      export PATH="/home/loops/.local/bin:$PATH"
@@ -129,7 +129,7 @@
  # short-lived glm-5.2 agent that diagnoses the blockage and drives the run to a clean DONE.
  systemd.services.cc-ci-upgrade-supervisor = {
    description = "cc-ci hourly weekly-run supervisor (glm-5.2 — drives a stalled /upgrade-all to completion)";
-    after = [ "network-online.target" "tailscaled.service" ];
+    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
    serviceConfig = {
      Type = "oneshot";  # launch-supervisor.py check: gate now, spawn the agent into tmux, return
@@ -0,0 +1,270 @@
+# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
+# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
+# the standalone opencode CLI, the shared opencode web server and its basic-auth web UI,
+# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
+#
+# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
+# `nixosModules.cc-ci-orchestrator`; the combined CI-server + orchestrator host (`#cc-ci`) also
+# imports recipe-maintainers/cc-ci's `nixosModules.cc-ci-server`.
+#
+# History: until 2026-09 this lived (twice, drifting) in nix/hosts/cc-ci-orchestrator-hetzner/
+# configuration.nix here and in notplants-nix's hosts/notplants-orchestrator/configuration.nix,
+# the shared agent box that also ran lichen + project-orchestrator. The cc-ci half moved to its
+# own host; this file is that half.
+{ config, lib, pkgs, ... }:
+let
+  cfg = config.cc-ci-orchestrator;
+in
+{
+  options.cc-ci-orchestrator = {
+    ciSshHost = lib.mkOption {
+      type = lib.types.str;
+      default = "127.0.0.1";
+      example = "100.95.31.88";
+      description = ''
+        Where `ssh cc-ci` (used by every skill and script that drives the CI server) connects to,
+        as root with ~loops/.ssh/cc-ci-local-ed25519 (a key generated ON the host — nothing
+        copied from another machine). On the combined host the CI server IS this
+        machine, so the default is loopback; a standalone orchestrator points it at the CI
+        server's tailnet address.
+      '';
+    };
+
+    opencodeUiBackendPort = lib.mkOption {
+      type = lib.types.port;
+      default = 8090;   # not 8080: acme-dns's local API has it on the combined host
+      description = ''
+        Plain-HTTP port nginx listens on for the opencode UI, reachable ONLY from the docker
+        bridge (firewall rule on docker_gwbridge). Traefik — which owns the public 443 on the
+        combined host — terminates TLS for opencodeUiHost and forwards here; nginx adds the basic
+        auth and logs failures for fail2ban with the real client IP.
+      '';
+    };
+
+    opencodeUiTraefikNetwork = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      example = "proxy";
+      description = ''
+        Name of the swarm overlay network the cc-ci traefik watches (cc-ci's swarm.nix creates
+        `proxy`). When set, a one-container swarm stack `ccci-opencode-ui` (a socat TCP relay to
+        nginx on the docker bridge) is deployed with traefik labels routing opencodeUiHost on
+        the `web-secure` entrypoint — the same label mechanism every cc-ci service and recipe
+        uses, so it coexists with the traefik recipe's own file provider (the wildcard cert and
+        the `security` middleware live there; switching traefik to a file *directory* replaces
+        that file and takes every front door down — learned 2026-09-07). null = no route.
+      '';
+    };
+
+    opencodeUiHost = lib.mkOption {
+      type = lib.types.str;
+      default = "oc.commoninternet.net";
+      description = "nginx server_name for the opencode web UI (TLS + basic auth).";
+    };
+
+    opencodeUiHtpasswdFile = lib.mkOption {
+      type = lib.types.str;
+      default = "/secrets/nginx/oc-htpasswd";
+      description = ''
+        htpasswd file for the opencode UI (`oc:<bcrypt>`), created out of band — a store path
+        would be world-readable. Default is under /secrets, the authoritative location for this
+        host's secrets; it must be readable by the `nginx` user (root:nginx 0640 in a directory
+        nginx can traverse). **nginx refuses to start if it is missing**, and its config check
+        runs as the nginx user, so a root-only file fails the check even though the path exists.
+      '';
+    };
+
+  };
+
+  config = {
+    # ---- the loops user -------------------------------------------------------------------
+    # claude sessions run as non-root (--dangerously-skip-permissions is refused for root).
+    users.users.loops = {
+      isNormalUser = true;
+      uid = 1000;                 # fixed: workspace files are rsynced between hosts by uid
+      home = "/home/loops";
+      shell = pkgs.bash;
+      extraGroups = [ "wheel" "docker" ];
+    };
+    security.sudo.wheelNeedsPassword = false;
+    security.sudo.extraRules = [{
+      users = [ "loops" ];
+      commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
+    }];
+
+    # /home/loops/.local/bin holds the standalone opencode binary; it must be first on every PATH
+    # (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
+    environment.variables.PATH = lib.mkForce
+      "/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
+
+    # ---- nix-ld: the standalone opencode CLI is a foreign dynamic ELF binary --------------------
+    programs.nix-ld.enable = true;
+    programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
+
+    # ---- the toolbox every agent (and every operator ssh session) on this box gets ----------
+    # Bar for adding something: an agent or an operator doing ordinary work would otherwise waste
+    # a turn discovering it is absent. Installed system-wide, so it is on PATH for BOTH root and
+    # loops via /run/current-system/sw/bin (which the forced PATH below keeps for every user).
+    environment.systemPackages = with pkgs; [
+      git tmux python3 jq curl cacert
+      gnused gawk coreutils gnugrep findutils util-linux nettools openssh
+      age sops ssh-to-age
+      wget gnutar gzip unzip zip xz
+      ripgrep fd tree file less which
+      procps psmisc htop lsof strace ncdu
+      dnsutils socat netcat-gnu iproute2 iputils
+      openssl gnumake gcc pkg-config
+      yq-go diffutils patch rsync bubblewrap
+      # Editors: `vim` ships `vi` too, so both names resolve for anyone who types either.
+      # `nano` comes from the base system. EDITOR is set below so git/systemctl/visudo agree.
+      vim nano
+      # `sqlite` is not a nicety here: Drone's build/step logs live in its sqlite volume and the
+      # runbook (and /cc-ci-status) tell you to read them there; without it every such check
+      # needed an ad-hoc `nix-shell -p sqlite`.
+      sqlite
+      bat bc moreutils pv man-pages
+    ];
+
+    # So `git commit`, `systemctl edit`, `visudo` etc. open something that exists on this host.
+    environment.variables.EDITOR = "vim";
+
+    # ---- ssh config for the loops user: `ssh cc-ci` = the CI server (root) -----------------
+    # Written only if absent so a manual customisation survives rebuilds.
+    system.activationScripts.loopsSshConfig = ''
+      mkdir -p /home/loops/.ssh && chown loops:users /home/loops/.ssh && chmod 700 /home/loops/.ssh
+      if [ ! -f /home/loops/.ssh/config ]; then
+        cat > /home/loops/.ssh/config <<'SSHCFG'
+Host cc-ci
+    HostName ${cfg.ciSshHost}
+    User root
+    IdentityFile /home/loops/.ssh/cc-ci-local-ed25519
+    IdentitiesOnly yes
+    StrictHostKeyChecking accept-new
+    ServerAliveInterval 30
+
+Host git.autonomic.zone
+    HostName git.autonomic.zone
+    Port 2222
+    User git
+    IdentityFile /home/loops/.ssh/autonomic-bot-cc-ci-ed25519
+    IdentitiesOnly yes
+SSHCFG
+        chmod 600 /home/loops/.ssh/config
+        chown loops:users /home/loops/.ssh/config
+      fi
+    '';
+
+    # ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
+    # No Claude Code on this host (operator 2026-09-07): the orchestrator and the weekly upgrader
+    # are opencode agents; Claude sessions run on the notplants-orchestrator box and reach this
+    # host over ssh.
+    systemd.services.opencode-install = {
+      description = "Install opencode CLI for loops user (idempotent)";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
+      environment = { HOME = "/home/loops"; };
+      path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip pkgs.unzip ];
+      script = ''
+        if [ ! -x "$HOME/.local/bin/opencode" ]; then
+          echo "installing opencode CLI for loops user..."
+          curl -fsSL https://opencode.ai/install | bash || echo "install failed — retry on next activation"
+          # The installer puts the binary in ~/.opencode/bin; every unit here expects ~/.local/bin.
+          if [ -x "$HOME/.opencode/bin/opencode" ]; then
+            mkdir -p "$HOME/.local/bin" && ln -sfn "$HOME/.opencode/bin/opencode" "$HOME/.local/bin/opencode"
+          fi
+        fi
+      '';
+    };
+
+    # ---- opencode web server: one shared instance the opencode-backed agents attach to -------
+    # Provider creds come from /srv/cc-ci/.testenv (out of band, see README).
+    systemd.services.opencode-web = {
+      description = "opencode web server for cc-ci agents";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "network-online.target" "opencode-install.service" ];
+      wants = [ "network-online.target" ];
+      serviceConfig = {
+        Type = "simple";
+        User = "loops"; Group = "users";
+        WorkingDirectory = "/srv/cc-ci-orch/cc-ci";
+        EnvironmentFile = [ "-/srv/cc-ci/cc-ci/.env.public" "/srv/cc-ci/.testenv" ];
+        ExecStartPre = "${pkgs.coreutils}/bin/rm -rf /tmp/opencode";
+        ExecStart = "/home/loops/.local/bin/opencode serve --hostname 127.0.0.1 --port 4096";
+        Restart = "on-failure";
+        RestartSec = "5s";
+      };
+      environment = {
+        HOME = "/home/loops";
+        PATH = lib.mkForce "/run/wrappers/bin:/home/loops/.local/bin:/run/current-system/sw/bin:/usr/bin:/bin:/etc/profiles/per-user/loops/bin:/nix/var/nix/profiles/default/bin";
+      };
+      path = [ pkgs.bash pkgs.coreutils pkgs.git pkgs.python3 pkgs.openssh pkgs.tmux pkgs.nettools ];
+    };
+
+    # ---- nginx: basic auth for the opencode UI, behind traefik -----------------------------
+    # Traefik (public 443, the CI wildcard cert) routes opencodeUiHost to this plain-HTTP vhost
+    # on the docker bridge address. nginx enforces HTTP basic auth (the opencode web UI has no
+    # auth of its own and can drive agent sessions), and — via real_ip from traefik's
+    # X-Forwarded-For — logs the CLIENT address on a 401, which is what the fail2ban jail bans.
+    # The htpasswd is created out of band (a store path would be world readable); nginx FAILS TO
+    # START without it, and its config check runs as the nginx user:
+    #   /etc/nginx/oc-htpasswd   root:nginx 0640   (`oc:<bcrypt>`; plaintext kept in /secrets)
+    # Rotate with: printf 'oc:%s\n' "$(mkpasswd -m bcrypt "$P")" > /etc/nginx/oc-htpasswd && systemctl reload nginx
+    services.nginx = {
+      enable = true;
+      recommendedProxySettings = true;
+      virtualHosts.${cfg.opencodeUiHost} = {
+        listen = [ { addr = "0.0.0.0"; port = cfg.opencodeUiBackendPort; } ];
+        basicAuthFile = cfg.opencodeUiHtpasswdFile;
+        extraConfig = ''
+          # traefik sits on the docker networks (ingress 10.0.0.0/24, gwbridge 172.18.0.0/16)
+          set_real_ip_from 172.16.0.0/12;
+          set_real_ip_from 10.0.0.0/8;
+          real_ip_header X-Forwarded-For;
+        '';
+        locations."/" = {
+          proxyPass = "http://127.0.0.1:4096";
+          proxyWebsockets = true;
+        };
+      };
+    };
+    # Only docker's bridge may reach the plain-HTTP backend; the public interface stays closed.
+    networking.firewall.interfaces.docker_gwbridge.allowedTCPPorts = [ cfg.opencodeUiBackendPort ];
+
+    # The traefik side of the route: a swarm service carrying the router labels. Named ccci-* so
+    # the weekly run's orphan sweep (skills/upgrade-all/sweep-orphans.sh keep-list) leaves it be. A plain
+    # TCP relay (socat) from the overlay network to nginx on the docker bridge; traefik's
+    # X-Forwarded-For passes through untouched, which is what nginx's real_ip reads.
+    systemd.services.opencode-ui-route = lib.mkIf (cfg.opencodeUiTraefikNetwork != null) {
+      description = "swarm stack ccci-opencode-ui: traefik labels ${cfg.opencodeUiHost} -> nginx basic auth";
+      wantedBy = [ "multi-user.target" ];
+      after = [ "deploy-proxy.service" "docker.service" "nginx.service" ];
+      wants = [ "deploy-proxy.service" ];
+      path = [ pkgs.docker ];
+      serviceConfig = { Type = "oneshot"; RemainAfterExit = true; };
+      script = ''
+        docker stack deploy --detach=true -c ${pkgs.writeText "opencode-ui-stack.yml" ''
+          # Deployed by opencode-ui-route.service (nix/modules/orchestrator-host.nix). Do not edit.
+          version: "3.8"
+          services:
+            relay:
+              image: alpine/socat:1.8.0.3
+              command: ["TCP-LISTEN:${toString cfg.opencodeUiBackendPort},fork,reuseaddr", "TCP:172.18.0.1:${toString cfg.opencodeUiBackendPort}"]
+              networks: [ ${cfg.opencodeUiTraefikNetwork} ]
+              deploy:
+                replicas: 1
+                labels:
+                  - "traefik.enable=true"
+                  - "traefik.http.routers.opencode-ui.rule=Host(`${cfg.opencodeUiHost}`)"
+                  - "traefik.http.routers.opencode-ui.entrypoints=web-secure"
+                  - "traefik.http.routers.opencode-ui.tls=true"
+                  - "traefik.http.services.opencode-ui.loadbalancer.server.port=${toString cfg.opencodeUiBackendPort}"
+          networks:
+            ${cfg.opencodeUiTraefikNetwork}:
+              external: true
+        ''} ccci-opencode-ui
+      '';
+    };
+  };
+}
Author	SHA1	Message	Date
notplantsandClaude Opus 5	f3c871665a	README + journal: the /secrets layout, and the weekly run's flash models Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 17:30:18 +00:00
notplantsandClaude Opus 5	e7aa055784	/secrets is the authoritative location for every secret, incl. the ssh host keys Operator rule: secrets live in /secrets and consumers reach them from there. Three subdirectories with the ownership each consumer needs — files/ (loops), host/ (root: ssh host keys + sops age identity), nginx/ (root:nginx: the opencode UI htpasswd) — under a 0711 /secrets so nginx can traverse to its own without the directory being listable. sshd's hostKeys and sops-nix's sshKeyPaths/keyFile are pointed at /secrets DIRECTLY rather than through symlinks: the ed25519 host key is load-bearing beyond ssh, since its age identity (age1tmvg…) is a recipient of cc-ci-secrets, and a dangling symlink would let sshd write a NEW key and silently make every cc-ci secret undecryptable. The /etc/ssh symlinks are added for discoverability only, so nothing depends on activation ordering. nginx's htpasswd path becomes an option (opencodeUiHtpasswdFile) defaulting under /secrets, rather than a hard-coded /etc/nginx path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 17:23:28 +00:00
notplantsandClaude Opus 5	360999d623	journal: secrets audit of the cc-ci host; leaked keys found in rsynced agent logs and redacted Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 17:07:23 +00:00
cc-ci-orchestrator	8af6c5fb45	journal: orchestrator startup entry (opencode agent, phase sequence confirmed DONE)	2026-09-08 16:53:57 +00:00
notplantsandClaude Opus 5	c14c56147d	launchers: the opencode UI is oc.ci.commoninternet.net (not tailnet-only); journal the key switch Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 16:53:20 +00:00
notplantsandClaude Opus 5	79f69b0d35	cc-ci host on the OpenCode Go subscription key (opencode-go), not ZEN The operator's AUTONOMIC_OPENCODE_KEY is a Go subscription key. Put in the `opencode` (ZEN) slot it authenticates but every request fails "Insufficient balance", because ZEN is pay-as-you-go credit; on the `opencode-go` endpoint the same key answers fine. So the orchestrator agent moves to opencode-go/glm-5.2, and README records that this host is a Go host and that its opencode config must carry no inline apiKey. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 16:50:20 +00:00
notplantsandClaude Opus 5	7abea67ad4	host toolbox: vi/vim, sqlite, bat, bc, moreutils, pv, man-pages; set EDITOR `vi` and `vim` were both absent on the cc-ci host (operator hit it over ssh); the vim package ships a `vi` binary, so one entry covers both names, and EDITOR=vim makes git/systemctl/visudo open something that exists. sqlite earns its place beyond convenience: Drone's build and step logs live in its sqlite volume, and the runbook plus /cc-ci-status tell you to read them there — until now every such check needed an ad-hoc `nix-shell -p sqlite`. Installed via environment.systemPackages, so they are on PATH for BOTH root and loops through /run/current-system/sw/bin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 16:32:45 +00:00
auto-update	ec8b2fef0f	flake.lock: weekly auto-update, health-checked	2026-09-08 03:06:27 +00:00
notplantsandClaude Opus 5	1eb176cf09	gitignore the auto-update lock backup; journal the tested auto-update + skill rewrites Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-08 02:02:26 +00:00
auto-update	d1f5e916f3	flake.lock: weekly auto-update, health-checked	2026-09-07 22:27:16 +00:00
notplantsandClaude Fable 5.1	a8af8429f2	weekly health-gated auto-update of the cc-ci host; skills rewritten for the combined host nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday 03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone builds → skip), `nix flake update` → build → switch-to-configuration test → cc-ci health checks (sshd, 0 failed units, core units, every swarm service at replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401) → profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci fast-forwarded; revert + lock restore on failure; one-line state file for /cc-ci-status. Skills (.opencode canonical, .claude pointers' descriptions synced): - cc-ci-orchestrator-update: THE host update — drives the auto-update unit by hand; --cc-ci-only for a cc-ci-main-only move; failure playbook. - cc-ci-server-update: delegates to it and explains why the old procedure (rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host. - cc-ci-update: chains orchestrator-update then tests-update. - cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator agent session, secrets inventory; verdict updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 22:22:50 +00:00
notplantsandClaude Fable 5.1	48226dd78f	opencode UI stack is ccci-opencode-ui, on the weekly sweep's keep-list; journal: first weekly run on the new host The 2026-09-07 weekly run's step-0 orphan sweep removed the `opencode-ui` swarm stack (the traefik route to the opencode web UI) because it was not on sweep-orphans.sh's keep-list. Renamed to ccci-opencode-ui alongside the other control-plane stacks and added to KEEP_RE. Journal: the run itself (lasuite-docs #8, n8n #7 GREEN on the new Drone, report week-2026-09-07 published) ran entirely on the new host. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 22:13:43 +00:00
autonomic-bot	be9687962c	upstream(n8n): release-notes sources 2.38.x	2026-09-07 21:49:50 +00:00
notplantsandClaude Fable 5.1	b360bfbb7c	journal: evening changes on the cc-ci host (opencode agent, secrets minimised, weekly run started) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:34:58 +00:00
notplantsandClaude Fable 5.1	77f156318e	README §4b: the bot deploy key is generated on the host; master age key only for the first deploy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:32:50 +00:00
notplantsandClaude Fable 5.1	415d530baf	cc-ci host: keys generated on the host, nothing copied from another machine Operator: no ssh keys or non-essential secrets from the old orchestrator box on the cc-ci host. `ssh cc-ci` uses cc-ci-local-ed25519 (generated on the host, pub in nix/hosts/cc-ci/ssh-keys); pushes to Gitea use autonomic-bot-cc-ci-ed25519 (generated on the host, registered on the bot account); root reuses that file for the cc-ci-secrets submodule. README §4: the /secrets/files inventory shrinks to exactly what cc-ci needs, and §4b now says to make the host a sops recipient and take the master key away. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:31:13 +00:00
notplantsandClaude Fable 5.1	a72a816b48	cc-ci host: the orchestrator is an opencode agent; no Claude on the box; /secrets convention Operator 2026-09-07 (evening): Claude sessions stay on notplants-orchestrator; on the cc-ci host the orchestrator is an opencode agent (agents.toml: opencode/glm-5.2) steered from https://oc.ci.commoninternet.net, next to the upgrader/report sessions. claude-install and CLAUDE_BIN are gone from the modules; launch-upgrader/report print the real UI URL; README "Operating the orchestrator" rewritten and a "Weekly upgrade run" section added. Secrets: only cc-ci's, under /secrets/files with runtime paths symlinked (README §4 table), nothing from other projects — tangled key and the tailscale line dropped from the host. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:27:23 +00:00
notplantsandClaude Fable 5.1	01caa90e76	README day-2: regenerate the cc-ci lock on the host; plan: steps 8 + lint follow-ups done Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:20:11 +00:00
notplantsandClaude Fable 5.1	841f8a9d1f	flake.lock: cc-ci → main with the lint fixes (#33 , #34 ) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:18:20 +00:00
notplantsandClaude Fable 5.1	536d621865	fail2ban nginx jail: read the journal (NixOS nginx logs errors to stderr, not error.log) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:08:22 +00:00
notplantsandClaude Fable 5.1	666509d53d	plan + journal: cutover done 2026-09-07, orchestrator moved, remaining items Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:06:09 +00:00
notplantsandClaude Fable 5.1	bf84245d4b	cc-ci host: one resolver set (no Hetzner caches), pin the host's own names During the DNS cutover the Hetzner resolvers nixos-infect had left in networking.nix still served the old address (3h TTL) while 1.1.1.1/8.8.8.8 had the new one, so every other lookup of drone.ci.commoninternet.net from the box itself went to the old server: runner "cannot request stage 404", bridge "drone trigger failed 404", traefik probes alternating 200/404. Drop those resolvers; pin ci/drone/report/traefik/warm-keycloak/oc names to this host in networking.hosts. README §7 records it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 21:03:00 +00:00
notplantsandClaude Fable 5.1	72d4a31910	opencode UI route via traefik labels on a swarm relay, not the file-directory provider Enabling FILE_PROVIDER_DIRECTORY_ENABLED in the coop-cloud traefik recipe REPLACES its single provider file (wildcard cert + `security` middleware), which took every cc-ci front door down for two minutes on 2026-09-07. Reverted. The route is now what every recipe does: a one-container swarm stack (`opencode-ui`, alpine/socat relay to nginx on the docker bridge) carrying the traefik router labels, deployed by opencode-ui-route.service. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:57:36 +00:00
notplantsandClaude Fable 5.1	6cc78cf758	orchestrator-host: opencode UI backend on 8090 (8080 is acme-dns's API); note the traefik FILE_PROVIDER_DIRECTORY_ENABLED requirement Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:51:53 +00:00
notplantsandClaude Fable 5.1	b23ad96616	opencode UI on 443 via traefik (Host routing), nginx basic auth on the docker bridge Operator: standard 443, routed by domain. Traefik already owns 443 with the *.ci.commoninternet.net cert, so opencode-ui-route.service drops a dynamic config into the traefik stack's watched file-providers volume routing oc.ci.commoninternet.net -> http://172.18.0.1:8080, where nginx (reachable only on docker_gwbridge) enforces the basic auth and logs 401s with the real client IP (real_ip from X-Forwarded-For). The fail2ban nginx jail bans in DOCKER-USER, since that traffic is forwarded, not INPUT. 8443 and the ACME-host variant are gone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:49:53 +00:00
notplantsandClaude Fable 5.1	8f7f125fc0	cc-ci host: no tailscale; fail2ban; opencode UI public on 8443 with the LE wildcard Operator 2026-09-07: the combined host stays off the tailnet. ssh is key-only (password + keyboard-interactive auth off), the opencode UI is served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's *.ci.commoninternet.net ACME cert (new orchestrator-host option opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail). tailscaled ordering dropped from the orchestrator units. README updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:45:59 +00:00
notplantsandClaude Fable 5.1	e5b0177c93	journal + reboots: 2026-09-07 consolidation sessions; agents.toml: orchestrator on claude/fable-5-1 (operator 2026-09-07) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:33:49 +00:00
notplantsandClaude Fable 5.1	a895bd3e3e	plan-cc-ci-combined-host: steps 1-5 done, host up, waiting on tailscale key + DNS Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:28:50 +00:00
autonomic-bot	86f15ae2c3	Merge pull request 'nix: one Hetzner host for the cc-ci CI server + the orchestrator (#cc-ci)' (#19 ) from feat/combined-cc-ci-host into main nix: one Hetzner host for the cc-ci CI server + the orchestrator (#19)	2026-09-07 20:26:25 +00:00
notplantsandClaude Fable 5.1	4bc483326b	orchestrator-host: symlink the opencode installer's binary into ~/.local/bin; README: nginx file perms, first-rebuild wrapper, pre-cutover checks Found on the first activation of #cc-ci on 195.201.88.249: the opencode installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and nginx's pre-start config check runs as the nginx user, so the staged cert/htpasswd must be root:nginx 0640, not root:root 0600. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:24:47 +00:00
notplantsandClaude Fable 5.1	f105808bed	nix/hosts/cc-ci: real hardware + networking from the box; document the infect fixes hardware.nix / networking.nix are now the nixos-infect output of 195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that matters: defaultGateway as an attrset with interface = "eth0". README §2a records what it actually took to get NixOS running on the Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's unit files shadowed every NixOS service), and the gateway fix applied via a chroot rebuild with the nix sandbox off. flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned to 1000 so workspace rsyncs by uid line up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 20:15:16 +00:00
notplantsandClaude Fable 5.1	31af820079	nix: one Hetzner host for the CI server + the orchestrator (#cc-ci) The orchestrator's flake now builds the machine it shares with the cc-ci CI server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server (new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix — the host contract those units always assumed (loops user, claude/opencode CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns 80/443, nix-ld, tool set, `ssh cc-ci` → loopback). nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old server's layout so the flake evaluates; they get replaced by the nixos-infect output of 195.201.88.249. README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake → staging → data restore → cutover). archive/ holds the retired Incus/Hetzner orchestrator host configs, the old terraform and the migration plans; references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working plan for the move. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz	2026-09-07 19:58:33 +00:00
autonomic-bot	5bebd13e49	docs: record weekly upgrade report Completed DeepSeek Flash run and published report verified.	2026-08-31 20:47:10 +00:00
autonomic-bot	82d3127c2a	docs: record weekly upgrade report	2026-08-31 20:46:55 +00:00
autonomic-bot	1b75d986b2	fix: default testme Gitea URL Verified with bash -n; public default only.	2026-08-31 19:29:59 +00:00