One tmux server hosts every agent session on this box and it lives in
cc-ci-loops.service's cgroup, so any rebuild touching that unit killed ALL sessions
(cc-ci, lichen, PO, pi). With KillMode=process systemd kills only the already-exited
launcher, not the cgroup — added to cc-ci-loops, lichen-orchestrator and
project-orchestrator. Trade-off: `systemctl stop` no longer tears agents down; do it
by hand.
Carries an in-flight nginx basicAuth block from the cc-ci agent that was already
uncommitted in this file and is now deployed with the 26.05 switch — committed here so
it is not lost, not authored by me.
Co-Authored-By: Claude <noreply@anthropic.com>
Upstream module dropped host-specific comments for its public repo, so the
context that matters here — gateway fronting, why this vhost stays off the
tailscale address group where opencode lives, and the exposure check script —
moves into the copy's header where it belongs.
Functional changes from upstream: the container now runs as uid 10001 and the
unit chowns the bind-mounted cache before start (without it, cache files left
by the earlier root container are unreadable and every cached actor 500s).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module. The atproto vhost listens on 0.0.0.0 only, so it
never shares an address group with oc.commoninternet.net (which is the default
server on 100.84.190.30:80 and has no auth in front of it). A new
public-default-reject vhost owns the public addresses — return 444 on 80,
ssl_reject_handshake on 443 — so unmatched, absent or spoofed Host headers from
the internet get nothing, now and after any future vhost is added.
The tailnet is untouched: oc.commoninternet.net still serves opencode there.
Verified with tools/check-exposure.sh in the project repo — 9/9 ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module in the project repo. forceSSL would have bounced
the gateway's plain-HTTP hop back to the gateway; and the oc vhost's explicit
listen on 100.84.190.30:80 was shadowing this vhost on the tailnet, serving the
opencode UI for Host: atproto.commoninternet.net.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Imports the notplants-atproto module: Docker daemon (this host had none), a
systemd unit running the project's docker-compose stack, and an nginx vhost
proxying to it on 127.0.0.1:8731 with ACME and websocket support.
Opens 80/443 publicly. Until now only 22 was open and everything else was
tailnet-only, so this is a real change in exposure — nginx is now reachable
from the internet. The existing oc.commoninternet.net vhost is untouched and
stays bound to the tailscale IP.
nix/atproto-likes.nix is a COPY; the canonical file lives in the project repo
at /srv/project-orchestrator/projects/notplants-atproto/nix/. Pure evaluation
forbids importing an absolute path outside the flake tree, so it has to be
duplicated here — and a new file must be git-added or nix silently ignores it.
ACME currently FAILS: *.commoninternet.net is a wildcard pointing at
143.244.213.108, so the HTTP-01 challenge is answered by that host (500).
nginx serves a self-signed placeholder and starts fine. Fix is an explicit
A record atproto.commoninternet.net -> 168.119.126.100, then
`systemctl start acme-atproto.commoninternet.net.service`.
Applied with nixos-rebuild switch; container healthy, TLS proxy and wss
verified end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
The PO is the operator's fleet-management Remote Control point; until now nothing
brought it back after a reboot. Mirrors lichen-orchestrator: oneshot + RemainAfterExit
running `agents.py up` as loops, which starts the PO agent and its watchdog. The PO's
agents.toml declares no `wake`, so the watchdog only heals a dead session — it never
sends periodic prompts.
Also lands the /mnt/data hardware.nix mount (by-uuid + nofail) that was sitting
uncommitted; fstab output is unchanged from the running system.
Deployed 2026-08-01: build delta was 5 derivations (the new unit + etc/system-units);
every other unit file byte-identical, dry-activate showed only a dbus reload. The
in-flight pi-coding-agent / launch.py work is parked on wip/pi-orchestrator-2026-08-01.
Co-Authored-By: Claude <noreply@anthropic.com>
Mirrors cc-ci-loops.service for the SEPARATE p-lichen-orchestrator (lichen.page testing). On boot,
resumes its Remote Control session + watchdog + pipeline via engine/agents.py up. Added after the
2026-07-08 reboot (Hetzner rollback of a bad staging nixos-rebuild) left this orchestrator down while
cc-ci-loops auto-recovered. Points at the /home path for now; re-point to /srv at that migration.
Verified: diff-closures shows only this unit added; sshd/dhcpcd/tailscaled/network unchanged.
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.
- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
(unused AND older than a week, so this week's likely-reused images stay) before
each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
deterministic gate — no-ops (zero tokens) when the run is complete or
progressing; only when a run stalled/died before completing does it launch a
short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
cc-ci-upgrade-all now reads an optional EnvironmentFile so the weekly run can
switch backend/model (e.g. LOOP_BACKEND=opencode LOOP_MODEL=opencode-go/glm-5.2)
without a rebuild. Absent file → claude/sonnet (unchanged). Built+switched on
cc-ci-orchestrator-hetzner, host verified healthy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per operator: drop the hourly cc-ci-reap-dev-deploys systemd timer; instead run the
dev-* reaper at the START (Step 0, alongside the orphan sweep) and END (new step 4b)
of each /upgrade-all run, with THRESHOLD=0 (the run is quiescent then, so clear all
dev-* unconditionally). The reaper keeps its safe default (4h) for ad-hoc use.
Step-2b mandatory teardown is unchanged (primary mechanism); this is the backstop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- /recipe-upgrade step 2b: teardown is now MANDATORY on every exit path (finally),
with a verify-no-leak check; tear down even on failure before reporting.
- reap-dev-deploys.sh: safe, age-gated backstop that removes only idle dev-* stacks
(never CI per-run stacks, warm-*, infra; an active dev loop stays fresh).
- orchestrator: hourly cc-ci-reap-dev-deploys systemd timer runs it against cc-ci,
bounding any leaked dev deploy from a crashed/abandoned loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- SSH config: replace REPLACE_WITH_CC_CI_HETZNER_TAILNET_IP placeholder with
the real tailnet IP 100.95.31.88 (so a fresh re-provision is correct).
- nix/configuration.nix + nix/README.md: mark HISTORICAL/dead (old Incus VM,
superseded by the Hetzner host) to prevent a wrong-host deploy.
- nginx oc.commoninternet.net vhost: note it's PARKED alongside opencode-web
(kept for one-step re-enable, not deleted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep the unit definition in the flake for easy re-enable; just stop it
auto-starting. Restore wantedBy = [ "multi-user.target" ] to bring it back.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the boot-fragile busybox-crond-in-tmux (phase 5 §4) with a
systemd service+timer. Service is timer-triggered only (not wantedBy
multi-user.target) so it never runs on boot/activation; mirrors the
cc-ci-loops env fix (CLAUDE_BIN + /home/loops/.local/bin on PATH).
Timer fires Sundays 02:00 UTC, Persistent=true so a missed run (box
down) fires once on next boot. Runs launch-upgrader.py start ->
cc-ci-upgrader agent -> /upgrade-all DEFAULT (opens recipe PRs, never
merges). Activate via nixos-rebuild + retire the old Monday crond after
the phase-5 T0-fire verification completes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The service path lacked /home/loops/.local/bin, so launch.py preflight's
which(claude) failed on every boot and the loops never auto-started
(they were restarted by hand). Set CLAUDE_BIN to the standalone CLI's
absolute path and prepend the dir to PATH so the tmux server every agent
session inherits resolves bare claude.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
configuration.nix:
- systemd.services.opencode-web: one shared opencode server on 127.0.0.1:4096,
EnvironmentFile=/srv/cc-ci/.testenv (TINFOIL_API_KEY), ExecStartPre clears
stale /tmp/opencode so restarts never fail on the EEXIST race.
- services.nginx: reverse-proxy oc.commoninternet.net → localhost:4096,
bound to tailscale IP 100.84.190.30 (tailnet-only, plain HTTP).
DNS: A record oc.commoninternet.net → 100.84.190.30 (operator step).
launch.sh + launch-upgrader.sh:
- Drop per-session ports / OPENCODE_HOST; add OPENCODE_SERVER=http://127.0.0.1:4096.
- opencode backend: agents use `opencode run --attach $OPENCODE_SERVER --title $session`
so each shows up as a named session in the web UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Now the workspace is staged on the Hetzner cpx22 (server 134487234, public
91.98.47.73, tailnet cc-ci-orchestrator-1 @ 100.84.190.30):
- configuration.nix: enable cc-ci-loops.service (wantedBy multi-user.target) so the
loops + watchdog auto-resume on boot; wire reboot-log.sh as ExecStartPre so reboots
auto-log to REBOOTS.md (boot_id-gated).
- plan-orchestrator-hetzner-migration.md: full migration record.
- REBOOTS.md / AGENTS.md: point the orchestrator host at Hetzner; first auto-logged
reboot line.
- launch-orchestrator.sh: default session id -> the Hetzner orchestrator session.
- flake.lock: pin inputs.
Verified: nixos-rebuild switch applied; systemctl is-enabled cc-ci-loops.service =
enabled; ExecStartPre logged this boot to REBOOTS.md; loops healthy on phase 2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nixos-rebuild removed the infect-provisioned authorized_keys — declare it
explicitly so rebuilds don't lock out root access.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds terraform/ to provision a Hetzner cpx11 (2 vCPU / 2 GB dedicated AMD / 40 GB NVMe)
for the loops runtime, and a flake + NixOS host config to converge it — replacing the slow
b1 Incus VM. Mirrors the cc-ci server terraform (same nixos-infect pin, same pattern).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Merging recipe-maintainers/cc-ci-orchestrator (the VM NixOS config repo)
into this repo as nix/ — the next step toward consolidating the two
orchestrator repos into a single cc-ci-orchestrator.
The source repo will be renamed to archived-cc-ci-orchestrator on Gitea.
This repo will be renamed cc-ci-orchestrator.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
