recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cc-ci host: the orchestrator is an opencode agent; no Claude on the box; /secrets convention

Browse Source 
Operator 2026-09-07 (evening): Claude sessions stay on notplants-orchestrator;
on the cc-ci host the orchestrator is an opencode agent (agents.toml:
opencode/glm-5.2) steered from https://oc.ci.commoninternet.net, next to the
upgrader/report sessions. claude-install and CLAUDE_BIN are gone from the
modules; launch-upgrader/report print the real UI URL; README "Operating the
orchestrator" rewritten and a "Weekly upgrade run" section added.

Secrets: only cc-ci's, under /secrets/files with runtime paths symlinked
(README §4 table), nothing from other projects — tangled key and the
tailscale line dropped from the host.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
This commit is contained in:
notplants
2026-09-07 21:27:23 +00:00
co-authored by Claude Fable 5.1
parent 01caa90e76
commit a72a816b48
7 changed files with 66 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+3 -2
View File
@@ -31,8 +31,9 @@ the orchestrator watches from outside.
Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) — the operator reconnects to it (that's
why the startup notification matters). Since 2026-09 the orchestrator runs on the **same Hetzner
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) as an **opencode agent** the operator
steers from https://oc.ci.commoninternet.net (no Claude on the cc-ci host; Claude sessions run on
the notplants-orchestrator box and reach cc-ci over ssh). Since 2026-09 the orchestrator runs on the **same Hetzner
host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
README.md
+37 -26
View File
@@ -9,7 +9,7 @@ module from the [cc-ci](https://git.autonomic.zone/recipe-maintainers/cc-ci) rep
| | where |
|---|---|
| Orchestrator loops, timers (weekly upgrader, hourly supervisor) | `nix/modules/cc-ci.nix` → `nixosModules.cc-ci-orchestrator` |
| The host contract those need (loops user, claude/opencode CLIs, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
| The host contract those need (loops user, opencode CLI, opencode web UI) | `nix/modules/orchestrator-host.nix` → `nixosModules.orchestrator-host` |
| The CI server (swarm, traefik, drone, runner, `!testme` bridge, dashboard, reports, acme-dns) | cc-ci repo `nix/modules/` → `nixosModules.cc-ci-server` (flake input `cc-ci`) |
| The machine: hardware, networking, root keys, firewall + fail2ban | `nix/hosts/cc-ci/` → `nixosConfigurations.cc-ci` |
| Plans, launch tooling, loop prompts, journal | `cc-ci-plan/` (see `AGENTS.md` for roles) |
@@ -205,22 +205,24 @@ git clone https://git.autonomic.zone/recipe-maintainers/cc-ci.git cc-ci-adv #
mkdir -p .cc-ci-logs .sops
```
Then the files below (`install -m600 -o loops -g users`):
Secrets live in **`/secrets/files/`** (owned by loops, mode 700 on the directory), the same
convention as the notplants-orchestrator box, with the runtime paths symlinked to them so one
`ls /secrets/files` is the complete inventory. **Only cc-ci's secrets go there** — nothing from
other projects on the box they were copied from (no tailscale, lichen, tangled, b1, borg, …):
secrets do not get spread around. `/secrets/README.txt` lists each file and its purpose.
| file | what | source |
| runtime path → `/secrets/files/…` | what | source |
|---|---|---|
| `/srv/cc-ci/.testenv` | `TS_AUTH_KEY`, `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys | old host `/secrets/files/cc-ci.testenv`; fresh: create each credential |
| `/srv/cc-ci/upgrader.env` | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run (no secrets) | old host, or copy the example in `AGENTS.md` |
| `/srv/cc-ci/.sops/master-age.txt` | the same master age key as 4b (skills that re-key secrets use it) | old host |
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) | pushes recipe branches / PRs as `autonomic-bot` | old host; fresh: new key added to the bot's Gitea account |
| `~loops/.ssh/tangled-ed25519` | optional, tangled.org mirrors | old host |
| `~loops/.claude/` | Claude Code auth + settings + the orchestrator session history | old host (`rsync -a`); fresh: `claude auth login` as loops (device code, interactive) |
| `~loops/.local/share/opencode/auth.json`, `~loops/.config/opencode/` | opencode provider auth (the weekly upgrader runs on opencode) | old host; fresh: `opencode auth login` |
| `/etc/nginx/oc-htpasswd` | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
| `/srv/cc-ci/.testenv` → `cc-ci.testenv` | `GITEA_PASSWORD` (autonomic-bot), `DOCKERHUB_USERNAME/TOKEN`, model API keys used by opencode providers. No `TS_AUTH_KEY` (no tailscale here). | old host's copy minus the tailscale line; fresh: create each credential |
| `/srv/cc-ci/upgrader.env` (not a secret, lives in the checkout) | `LOOP_TIER`, `LOOP_MODEL`, `REPORT_MODEL` for the weekly run | old host, or copy the example in `AGENTS.md` |
| `/srv/cc-ci/.sops/master-age.txt` → `master-age.txt` | the same master age key as 4b (skills that re-key secrets use it); root's `/var/lib/sops-nix/key.txt` is a root-only copy of it | old host |
| `~loops/.ssh/cc-ci-root-ed25519` (+`.pub`) → same names | `ssh cc-ci` as root — to loopback on this host | old host; fresh: `ssh-keygen -t ed25519` and add the pub to `nix/hosts/cc-ci/ssh-keys` |
| `~loops/.ssh/autonomic-bot-gitea-ed25519` (+`.pub`) → same names | pushes recipe branches / PRs as `autonomic-bot`; root's `/root/.ssh/` copy clones the private cc-ci-secrets submodule | old host; fresh: new key added to the bot's Gitea account |
| `~loops/.local/share/opencode/auth.json` → `opencode-auth.json`; `~loops/.config/opencode/opencode.jsonc` (config, not secret) | opencode provider auth + config — the orchestrator AND the weekly upgrader are opencode agents; there is no Claude on this host | old host; fresh: `opencode auth login` as loops |
| `/etc/nginx/oc-htpasswd` (root:nginx copy; plaintext in `/secrets/files/oc-basic-auth.txt`) | basic auth for the opencode UI (`https://oc.ci.commoninternet.net`, via traefik); **nginx refuses to start without it**, and its config check runs as the `nginx` user, so `root:nginx 0640` (the `nginx` group exists after the first rebuild — fix ownership then and `systemctl restart nginx`) | old host (`/secrets/files/oc-basic-auth.txt` has the plaintext), or a new `oc:<bcrypt>` line via `mkpasswd -m bcrypt` |
`~loops/.ssh/config` is written by the activation script on first rebuild (`Host cc-ci` →
`127.0.0.1`, `git.autonomic.zone`, `tangled.org`); it is not overwritten if present.
`127.0.0.1`, `git.autonomic.zone`); it is not overwritten if present.
## 5. Build and activate
@@ -340,29 +342,38 @@ stopped on the new host while its directory is copied:
---
# Operating the orchestrator session
# Operating the orchestrator
The steering session is a long-lived interactive Claude Code session under tmux with
`--remote-control`, so it can be watched and steered from **claude.ai/code** (or the mobile app).
`cc-ci-orchestrator.service` relaunches it on boot via `cc-ci-plan/agents.py up orchestrator`
(backend + model in `cc-ci-plan/agents.toml`).
The cc-ci orchestrator on this host is an **opencode agent** (`cc-ci-plan/agents.toml`:
backend `opencode`, model `opencode/glm-5.2`), launched on boot by `cc-ci-orchestrator.service`
via `cc-ci-plan/agents.py up orchestrator` into tmux session `cc-ci-orchestrator`, attached to
the shared opencode web server. Steer it from the web UI, **https://oc.ci.commoninternet.net**
(basic auth, user `oc`), where the weekly upgrader's and report's sessions show up too.
There is no Claude Code on this host: Claude sessions run on the notplants-orchestrator box and
operate the CI from there over ssh (`Host cc-ci` in that box's loops ssh config → this host).
```bash
# attach on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator
# or resume the conversation by hand in a fresh tmux pane
claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
# already inside a live session and just want the web surface? /remote-control
# on the host
sudo -iu loops tmux attach -t cc-ci-orchestrator # the agent's terminal
systemctl restart cc-ci-orchestrator # relaunch it (fresh opencode session)
sudo -iu loops tmux ls # cc-ci-orchestrator, cc-ci-upgrader (+watchdog) during the weekly run
```
`--resume <name|id>` selects the *conversation* to restore; the `--remote-control "<name>"` value is
only the web display label. Don't pass `--fork-session` unless you mean to branch.
## Weekly upgrade run
`cc-ci-upgrade-all.timer` (Thu 22:00 America/New_York) starts `launch-upgrader.py`, which runs
`/upgrade-all` as an opencode agent (`upgrader.env`: `LOOP_MODEL`/`REPORT_MODEL`) in tmux
`cc-ci-upgrader` with a usage-limit watchdog; the hourly `cc-ci-upgrade-supervisor.timer` drives a
stalled run to completion. Start one by hand with `systemctl start cc-ci-upgrade-all`; follow
`/srv/cc-ci/.cc-ci-logs/cc-ci-upgrader.log`; it ends with `UPGRADE RUN COMPLETE`, a summary in
`.cc-ci-logs/upgrades/upgrade-all-<date>.md`, and a report published to
`https://report.ci.commoninternet.net/week-<date>.html`.
## Kick off / supervise the loops
```bash
cd /srv/cc-ci/cc-ci-plan
./launch.sh start # Builder + Adversary loops (interactive --remote-control in tmux) + watchdog
./launch.sh start # Builder + Adversary loops + watchdog (all phases are DONE; this just confirms)
./launch.sh status # session + DONE state
./launch.sh logs builder|adversary|watchdog
./launch.sh stop
cc-ci-plan/agents.toml
+3 -2
View File
@@ -53,8 +53,9 @@ log_dir = "/srv/cc-ci/.cc-ci-logs"
[[agent]]
name = "orchestrator" # tmux session: cc-ci-orchestrator
kind = "persistent"
backend = "claude" # operator 2026-09-07: Claude Code under remote-control so they can talk to it
model = "claude-fable-5-1" # Fable 5.1
backend = "opencode" # operator 2026-09-07 (evening): NO Claude on the cc-ci host — the
model = "opencode/glm-5.2" # orchestrator is an opencode agent, steered via the opencode web UI
# (https://oc.ci.commoninternet.net). Claude sessions live on notplants-orchestrator.
resume = false # opencode backend has no --resume equivalent (fresh session each launch)
watch = "heal" # restart if dead / FATAL / backend-mismatch; never stall-reboot
wake = { interval = 3600, prompt_file = "ai-progress-monitor-prompt.txt" }
cc-ci-plan/launch-report.py
+1 -1
View File
@@ -124,7 +124,7 @@ def start(mode, date):
_lu._pin_new_session(_prev_ids)
if BACKEND == "opencode":
if OPENCODE_SHARE:
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net +public --share link")
log(f" attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net +public --share link")
# Watchdog for the report too: it runs on glm-5.2 sharing the same opencode-go budget the
# upgrade run just drained, so a 429 stall is likely. Reuse launch-upgrader.py's watchdog,
# pointed at THIS (cc-ci-report) session with a report-specific marker + resume prompt. It
cc-ci-plan/launch-upgrader.py
+3 -3
View File
@@ -67,7 +67,7 @@ REMOTE_CONTROL = os.environ.get("REMOTE_CONTROL", "1") == "1"
OPENCODE_BIN = os.environ.get("OPENCODE_BIN", "/home/loops/.local/bin/opencode")
OPENCODE_SERVER = os.environ.get("OPENCODE_SERVER", "http://127.0.0.1:4096")
# Web visibility for the opencode backend: attach the session to the shared opencode
# web server (viewable at http://oc.commoninternet.net, tailnet-only) AND optionally
# web server (viewable at https://oc.ci.commoninternet.net (basic auth)) AND optionally
# create a public opencode.ai --share link. Default both on so the run is monitorable.
OPENCODE_SHARE = os.environ.get("OPENCODE_SHARE", "1") == "1"
@@ -220,7 +220,7 @@ def start(mode="use-or-create"):
f"{OPENCODE_BIN} run {model_flag} {share_flag} --attach '{OPENCODE_SERVER}' "
f"--title '{SESSION}' \"$(cat '{kf}')\""
)
log(f" attached to {OPENCODE_SERVER} → http://oc.commoninternet.net (tailnet only)"
log(f" attached to {OPENCODE_SERVER} → https://oc.ci.commoninternet.net"
+ (" +public --share link (printed in the session)" if OPENCODE_SHARE else ""))
else:
die(f"unknown LOOP_BACKEND '{BACKEND}' — use 'claude' or 'opencode'")
@@ -655,7 +655,7 @@ Model: {MODEL} (LOOP_MODEL or UPGRADER_MODEL env var)
Args: {UPGRADER_ARGS or '<none>'} (UPGRADER_ARGS env var, passed to /upgrade-all)
claude: viewable at claude.ai/code
opencode: viewable at http://oc.commoninternet.net server={OPENCODE_SERVER}
opencode: viewable at https://oc.ci.commoninternet.net server={OPENCODE_SERVER}
""")
nix/modules/cc-ci.nix
+12 -12
View File
@@ -17,7 +17,7 @@
systemd.services.cc-ci-loops = {
description = "cc-ci Builder/Adversary loops + watchdog (launch.sh start)";
wantedBy = [ "multi-user.target" ]; # enabled after workspace staged (Hetzner cutover)
after = [ "network-online.target" "claude-install.service" ];
after = [ "network-online.target" "opencode-install.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
# KillMode=process: this unit only LAUNCHES the tmux server, it does not own it. With the
@@ -32,14 +32,14 @@
# Append one line to REBOOTS.md per genuine reboot (boot_id-gated; not on manual restart).
ExecStartPre = "${pkgs.bash}/bin/bash /srv/cc-ci/cc-ci-plan/reboot-log.sh";
};
# CLAUDE_BIN points at the standalone CLI installed by claude-install.service; the loops
# backend defaults to claude (persisted in .loop-backend). Without this, launch.py's preflight
# `which(claude)` fails because the systemd `path` below has no /home/loops/.local/bin.
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
# The loops backend is persisted in .cc-ci-logs/.loop-backend (opencode on the cc-ci host —
# no Claude there since 2026-09-07). The build phases are all DONE; on boot launch.sh just
# re-confirms that and exits.
environment = { RESUME_PHASE = "1"; HOME = "/home/loops"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
# Put the standalone claude/opencode binaries on PATH. On a cold boot this is the env the
# tmux server (and thus every agent session) inherits, so bare `claude` resolves everywhere.
# Put the standalone opencode binary on PATH. On a cold boot this is the env the tmux
# server (and thus every agent session) inherits.
export PATH="/home/loops/.local/bin:$PATH"
[ -x /srv/cc-ci/cc-ci-plan/launch.sh ] && /srv/cc-ci/cc-ci-plan/launch.sh start || \
echo "workspace not staged yet — skipping loop start"
@@ -50,9 +50,9 @@
# lichen-orchestrator / project-orchestrator above: this unit only LAUNCHES the orchestrator's
# tmux session via the agent-orchestrator harness (cc-ci-plan/agents.py); it does not own the
# session or the tmux server. The orchestrator agent is declared in cc-ci-plan/agents.toml
# (backend/model chosen there — Claude Code under Remote Control since 2026-09-07; before that
# opencode/glm-5.2 attached to the shared opencode web server, opencode-web.service in
# orchestrator-host.nix, which the upgrader still uses). The harness watchdog (started by
# (backend/model chosen there: an opencode agent, opencode/glm-5.2, attached to the shared
# opencode web server — opencode-web.service in orchestrator-host.nix — and steered from its
# web UI; no Claude on this host, operator 2026-09-07). The harness watchdog (started by
# `agents.py up`) keeps it alive: heal-only (no stall reboots — a persistent supervisor must not
# be killed just for idling). Added 2026-08-03 for reboot-resilience.
systemd.services.cc-ci-orchestrator = {
@@ -85,7 +85,7 @@
# wantedBy multi-user.target) so it never runs on boot/activation — only on the schedule.
systemd.services.cc-ci-upgrade-all = {
description = "cc-ci weekly /upgrade-all run (recipe upgrade survey + PRs, never merges)";
after = [ "network-online.target" "claude-install.service" ];
after = [ "network-online.target" "opencode-install.service" ];
wants = [ "network-online.target" ];
serviceConfig = {
Type = "oneshot"; # launch-upgrader.py spawns the cc-ci-upgrader tmux session and returns
@@ -103,7 +103,7 @@
# timer fire. Holds no secrets (the tinfoil API key lives in the opencode config / auth.json).
EnvironmentFile = "-/srv/cc-ci/upgrader.env";
};
environment = { HOME = "/home/loops"; CLAUDE_BIN = "/home/loops/.local/bin/claude"; };
environment = { HOME = "/home/loops"; };
path = [ pkgs.bash pkgs.tmux pkgs.git pkgs.python3 pkgs.openssh pkgs.nettools ];
script = ''
export PATH="/home/loops/.local/bin:$PATH"
nix/modules/orchestrator-host.nix
+7 -24
View File
@@ -1,6 +1,6 @@
# orchestrator-host.nix — the host contract that nix/modules/cc-ci.nix (the orchestrator's
# loops/timers) silently assumes, made explicit and reusable: the `loops` user the agents run as,
# the standalone claude/opencode CLIs, the shared opencode web server and its tailnet-only UI,
# the standalone opencode CLI, the shared opencode web server and its basic-auth web UI,
# nix-ld so foreign binaries run on NixOS, and the tool set agents reach for.
#
# Exported from flake.nix as `nixosModules.orchestrator-host`. A host imports this together with
@@ -79,12 +79,12 @@ in
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
# /home/loops/.local/bin holds the standalone claude + opencode binaries; it must be first on
# every PATH (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
# /home/loops/.local/bin holds the standalone opencode binary; it must be first on every PATH
# (interactive shells, tmux, the systemd units in cc-ci.nix prepend it too).
environment.variables.PATH = lib.mkForce
"/home/loops/.local/bin:/run/current-system/sw/bin:/run/wrappers/bin:/usr/bin:/bin";
# ---- nix-ld: the standalone Claude Code / opencode CLIs are foreign dynamic ELF binaries ---
# ---- nix-ld: the standalone opencode CLI is a foreign dynamic ELF binary --------------------
programs.nix-ld.enable = true;
programs.nix-ld.libraries = with pkgs; [ stdenv.cc.cc.lib zlib openssl curl glibc ];
@@ -123,10 +123,6 @@ Host git.autonomic.zone
User git
IdentityFile /home/loops/.ssh/autonomic-bot-gitea-ed25519
IdentitiesOnly yes
Host tangled.org
IdentityFile /home/loops/.ssh/tangled-ed25519
IdentitiesOnly yes
SSHCFG
chmod 600 /home/loops/.ssh/config
chown loops:users /home/loops/.ssh/config
@@ -134,22 +130,9 @@ SSHCFG
'';
# ---- standalone CLIs (idempotent installers; re-run on every activation, no-op if present) --
systemd.services.claude-install = {
description = "Install Claude Code CLI for loops user (idempotent)";
wantedBy = [ "multi-user.target" ];
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
serviceConfig = { Type = "oneshot"; RemainAfterExit = true; User = "loops"; Group = "users"; };
environment = { HOME = "/home/loops"; };
path = [ pkgs.curl pkgs.bash pkgs.coreutils pkgs.gnutar pkgs.gzip ];
script = ''
if [ ! -x "$HOME/.local/bin/claude" ]; then
echo "installing Claude Code CLI for loops user..."
curl -fsSL https://claude.ai/install.sh | bash || echo "install failed — retry on next activation"
fi
'';
};
# No Claude Code on this host (operator 2026-09-07): the orchestrator and the weekly upgrader
# are opencode agents; Claude sessions run on the notplants-orchestrator box and reach this
# host over ssh.
systemd.services.opencode-install = {
description = "Install opencode CLI for loops user (idempotent)";
wantedBy = [ "multi-user.target" ];