Operator decision: main driving sessions stay glm-5.2; every task-tool subagent
runs opencode/deepseek-v4-pro (~3-5x cheaper, near-free cache hits) — the weekly
upgrade run burns most of its budget in subagents. Report generator stays glm-5.2
(launch-report.py default). Project-scoped config.
The 2026-08-03 cc-ci outage was recovered with ad-hoc tooling living in /tmp
(leftover from a PREVIOUS incident, half-evaporated). Promoted to the repo:
- scripts/recovery/hetzner.py — Hetzner API helper (status/actions/reboot/reset/
power/rescue-on|off/console), knows cc-ci=134485294 + orchestrator=134487234 by
name; token from HCLOUD_TOKEN or /srv/cc-ci/.hcloud-token (0600, never in git).
- scripts/recovery/hetzner-console.sh — shell-only VGA console: fresh console
session -> websocat bridge -> vncdotool (venv auto-bootstrapped in ~/.cache).
screenshot / key / type subcommands; encodes the reset-invalidates-session and
single-connection-bridge gotchas.
- scripts/recovery/README.md — the condensed 10-minute unreachable-server drill,
incl. the GRUB submenu 1>N ids + clear-grubenv-after-switch rule.
- hetzner-server-recovery skill: console/API sections now point at the repo tools
instead of describing /tmp rebuilds.
Smoke-tested: hetzner.py cc-ci status OK.
Operator decision: no policy difference between cc-ci and recipe-maintainer. On
inspection ARM already agrees (recipe-upgrade-cron-all: 'PRs are reviewed and merged
manually by a human afterwards... never merges anything'; 'no human review in the
middle' = skip the mid-run plan confirmation only). Wrappers previously framed this
as a cc-ci override over ARM auto-merge flows — wrong reading; now stated as ONE
unified rule. /help conventions updated to match.
The repo already vendored ARM as the references/recipe-maintainer submodule (old
repo name recipe-maintainers/recipe-maintainer, pinned 460eba0). Rather than carry
two copies, drop the just-added vendor/ duplicate and:
- retarget references/recipe-maintainer to
ssh://git.autonomic.zone/recipe-maintainers/autonomic-recipe-maintainer.git
(same lineage — 460eba0 is an ancestor) and bump to latest acd5cfb, which also
refreshes the parity-test SOURCE reference the tests cite.
- gen-cctest-skills.py + all 30 cctest-* wrappers + /help now point at
references/recipe-maintainer.
- JOURNAL.md: pending session entries (server-update policy addendum, tests-update,
orchestrator-update, upgrade-run notes).
One operator interface for both toolkits (operator decision 2026-08-04):
- vendor/autonomic-recipe-maintainer: ARM pinned as a submodule at acd5cfb (latest).
- scripts/gen-cctest-skills.py: generates a cctest-<name> wrapper pair
(.opencode canonical + .claude thin) for every ARM skill — frontmatter carries ARM's
own description tagged [recipe-maintainer/cctest]; body points at the canonical
submodule SKILL.md, sets cwd/sandbox context, and states the policy overrides
(auto-merge-style ARM flows need per-run operator opt-in; never touch cc-ci infra
from an ARM skill; submodule is read-only here). Re-run after every submodule bump.
- 30 cctest-* wrappers generated.
- /help: cctest family section + situation-table rows + the cc-ci-vs-cctest rule of thumb.
cctest = the recipe-maintainer test server; these skills run against it + the ARM
sandbox, never against the cc-ci CI server/swarm.
Seven check areas: weekly-run recency+outcome, report publishing, stale recipes/tests,
open recipe PRs (CVE-carrying PRs open >14d flagged high-priority, ready-to-merge PRs
listed as normal), server+orchestrator flake-update recency vs channel tip, host/service
health incl. the bridge !testme path (silent-401 stale-secret check from the 2026-08-03
finding), and maintained-set consistency. Ends with ALL HEALTHY or a prioritized findings
list, each mapped to the skill to invoke. Strictly read-only.
Codifies the full path walked for the 2026-08-03 wordpress enrollment (cc-ci PR #14):
survey -> mirror create+sync from coopcloud -> author test suite (health floor +
non-vacuous recipe-specific tests incl. sec4.3 create-an-object round-trip, recipe-local
setup helper, PARITY.md) -> bridge POLL_REPOS -> used-recipes.md weekly row +
upstream/<recipe>.md registry -> full-suite-green verification with the new tests ->
bridge deploy via test-before-switch -> merge-on-green + report (PR-visibility policy).
Includes the traps hit live: creds injection over stdin for cc-ci-side helpers, fresh-
deploy wizard state in HEALTH_OK, repo-dev-shell ruff, swarm serialization, stale bridge
secret 401s silently dropping !testme.
- cc-ci-server-update / cc-ci-tests-update / cc-ci-update: operator policy change —
open PRs for visibility/historical record and merge directly once verified
(invocation = authorization); reports must list merged-PR links + change summaries.
(tests-update still never merges the paired recipe upgrade PRs.)
- cc-ci-server-update: mandatory nixos-rebuild test step (5d) before switch, with
detached-activation + transient-unit notes from the 2026-08-03 26.05 deploy.
- AGENTS.md: test-before-switch policy for orchestrator host rebuilds.
- NEW skill cc-ci-orchestrator-update (+ thin .claude wrapper): the /cc-ci-server-update
analogue for this host (flake /srv/cc-ci-orch, .#cc-ci-orchestrator-hetzner, Hetzner
server 134487234), with self-update caveats.
- JOURNAL.md: 2026-08-03 server-update incident/recovery handoff entry.
Reverse the skill location: the full definitions now live in
.opencode/skills/<name>/SKILL.md (opencode's native project-skill
location, auto-scanned), and .claude/skills/<name>/SKILL.md are thin
pointer wrappers that keep the frontmatter for discoverability and
redirect to the .opencode/ canonical file for the procedure.
Also drops the 'operator-authorized mid-run' gating from
/cc-ci-server-update: the operator's choice to invoke the skill IS the
authorization for the live nixos-rebuild switch — no second check to
wait for. The quiescent-moment preference stays (don't disrupt an
in-flight !testme), but it's not a gate.
Three new skills for keeping cc-ci itself current (manual triggers for
what should ideally be automatic):
- /cc-ci-server-update — bump the cc-ci SERVER host's nixpkgs/sops-nix
flake inputs (hard rev pin, moves deliberately), open a cc-ci PR, then
deploy to the live host via nixos-rebuild switch (operator-authorized,
no-CI window) + health gate (no failed units, infra up, endpoints 200).
Manual trigger for periodic nixpkgs bumps.
- /cc-ci-tests-update — sweep all maintained recipes for stale cc-ci
tests (carry-over from /upgrade-all's commented stale tests + a fresh
/ci-test-review sweep), author the minimal test update per stale
recipe, open a cc-ci test PR, verify via the standard PR + !testme ->
CI flow. Closes the loop the weekly /upgrade-all deliberately defers
(it comments-only; this fixes the tests). Never weakens a test.
- /cc-ci-update — umbrella: run server-update then tests-update, in
that order, gated on the server health check (don't sweep tests on a
sick/rolled-back host). Server-first because the harness toolchain
rides on the host nixpkgs.
All three: create + verify, NEVER merge. Reuse the existing ci-test-review
helpers (open-cc-ci-pr.sh, run-all-recipes.sh, verify-pr.sh) and the
recipe-upgrade --with-tests discipline.
Switch the cc-ci-orchestrator agent (cc-ci-plan/agents.toml) from the claude
backend to the opencode backend so it attaches to the shared opencode web
server (opencode-web.service, 127.0.0.1:4096) and is reachable for Remote
Control at https://oc.commoninternet.net under the /srv/cc-ci-orch project.
Model = opencode/glm-5.2 (same provider family as the weekly upgrader /
supervisor). resume=false — the opencode backend has no --resume equivalent
(a fresh session per launch, matching the existing launch-opencode.sh shape).
Add a cc-ci-orchestrator.service systemd unit so on boot it runs
'agents.py up orchestrator' after opencode-web.service is up — the same
reboot-resilience shape lichen-orchestrator / project-orchestrator already
have. KillMode=process so a rebuild that merely touches this unit does not
tear down the shared tmux server and every agent session with it.
No secrets: the diff is only config + comments (paths, model names, unit
definition).
Add a 443 listener on the tailscale IP for the opencode vhost, alongside the
existing port 80. The name resolves to a CGNAT tailscale IP so Let's Encrypt
HTTP-01 can't validate it and there's no DNS-01 provider here; the vhost is
tailnet-only, so a self-signed cert (out-of-band at /etc/nginx/oc-selfsigned.*,
like oc-htpasswd) is acceptable. addSSL=true is required so the NixOS nginx
module actually renders ssl_certificate — without it nginx -t fails and takes
the atproto vhost down too. Rationale + regen command documented inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
One tmux server hosts every agent session on this box and it lives in
cc-ci-loops.service's cgroup, so any rebuild touching that unit killed ALL sessions
(cc-ci, lichen, PO, pi). With KillMode=process systemd kills only the already-exited
launcher, not the cgroup — added to cc-ci-loops, lichen-orchestrator and
project-orchestrator. Trade-off: `systemctl stop` no longer tears agents down; do it
by hand.
Carries an in-flight nginx basicAuth block from the cc-ci agent that was already
uncommitted in this file and is now deployed with the 26.05 switch — committed here so
it is not lost, not authored by me.
Co-Authored-By: Claude <noreply@anthropic.com>
nixpkgs un-pinned from a frozen rev (50ab7937, 24.11, 2025-06-30) to the nixos-26.05
channel; sops-nix un-pinned to follow it. The old pins claimed to match the cc-ci
server "for ecosystem consistency", but this host runs agents/tmux/nginx/docker, not
recipe CI, so it need not track that server — and a frozen rev only accrues unpatched
CVEs. Built + activated: generation nixos-26.05.20260731.5b4f72e, kernel 6.6.94 ->
6.18.40 (reboot pending for the kernel). Pre-update files: /tmp/flake.{nix,lock}.pre-update.
Co-Authored-By: Claude <noreply@anthropic.com>
Upstream module dropped host-specific comments for its public repo, so the
context that matters here — gateway fronting, why this vhost stays off the
tailscale address group where opencode lives, and the exposure check script —
moves into the copy's header where it belongs.
Functional changes from upstream: the container now runs as uid 10001 and the
unit chowns the bind-mounted cache before start (without it, cache files left
by the earlier root container are unreadable and every cached actor 500s).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module. The atproto vhost listens on 0.0.0.0 only, so it
never shares an address group with oc.commoninternet.net (which is the default
server on 100.84.190.30:80 and has no auth in front of it). A new
public-default-reject vhost owns the public addresses — return 444 on 80,
ssl_reject_handshake on 443 — so unmatched, absent or spoofed Host headers from
the internet get nothing, now and after any future vhost is added.
The tailnet is untouched: oc.commoninternet.net still serves opencode there.
Verified with tools/check-exposure.sh in the project repo — 9/9 ok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Sync of the canonical module in the project repo. forceSSL would have bounced
the gateway's plain-HTTP hop back to the gateway; and the oc vhost's explicit
listen on 100.84.190.30:80 was shadowing this vhost on the tailnet, serving the
opencode UI for Host: atproto.commoninternet.net.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
Imports the notplants-atproto module: Docker daemon (this host had none), a
systemd unit running the project's docker-compose stack, and an nginx vhost
proxying to it on 127.0.0.1:8731 with ACME and websocket support.
Opens 80/443 publicly. Until now only 22 was open and everything else was
tailnet-only, so this is a real change in exposure — nginx is now reachable
from the internet. The existing oc.commoninternet.net vhost is untouched and
stays bound to the tailscale IP.
nix/atproto-likes.nix is a COPY; the canonical file lives in the project repo
at /srv/project-orchestrator/projects/notplants-atproto/nix/. Pure evaluation
forbids importing an absolute path outside the flake tree, so it has to be
duplicated here — and a new file must be git-added or nix silently ignores it.
ACME currently FAILS: *.commoninternet.net is a wildcard pointing at
143.244.213.108, so the HTTP-01 challenge is answered by that host (500).
nginx serves a self-signed placeholder and starts fine. Fix is an explicit
A record atproto.commoninternet.net -> 168.119.126.100, then
`systemctl start acme-atproto.commoninternet.net.service`.
Applied with nixos-rebuild switch; container healthy, TLS proxy and wss
verified end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmEK2voMnBa23495aLk1Ce
The PO is the operator's fleet-management Remote Control point; until now nothing
brought it back after a reboot. Mirrors lichen-orchestrator: oneshot + RemainAfterExit
running `agents.py up` as loops, which starts the PO agent and its watchdog. The PO's
agents.toml declares no `wake`, so the watchdog only heals a dead session — it never
sends periodic prompts.
Also lands the /mnt/data hardware.nix mount (by-uuid + nofail) that was sitting
uncommitted; fstab output is unchanged from the running system.
Deployed 2026-08-01: build delta was 5 derivations (the new unit + etc/system-units);
every other unit file byte-identical, dry-activate showed only a dbus reload. The
in-flight pi-coding-agent / launch.py work is parked on wip/pi-orchestrator-2026-08-01.
Co-Authored-By: Claude <noreply@anthropic.com>
GO subscription hit a monthly usage limit (resets in 9 days). Add LOOP_TIER
env (default 'zen') to launch-upgrader.py + launch-report.py so the opencode
backend can use either subscription: zen→opencode/glm-5.2, go→opencode-go/glm-5.2.
The tier selects the default model, the usage-limit probe endpoint, and the
auth.json key. The systemd timer reads LOOP_TIER from /srv/cc-ci/upgrader.env.
Mirrors cc-ci-loops.service for the SEPARATE p-lichen-orchestrator (lichen.page testing). On boot,
resumes its Remote Control session + watchdog + pipeline via engine/agents.py up. Added after the
2026-07-08 reboot (Hetzner rollback of a bad staging nixos-rebuild) left this orchestrator down while
cc-ci-loops auto-recovered. Points at the /home path for now; re-point to /srv at that migration.
Verified: diff-closures shows only this unit added; sshd/dhcpcd/tailscaled/network unchanged.
A run can finish the recipe work + summary + report yet never print a clean
sign-off marker (it may wedge afterwards, as on 2026-07-03). Without a second
completion signal the supervisor would try to 'resume' the already-finished run
every hour until the 96h window closed. Now the gate also treats the run as done
when a weekly summary file (upgrade-all-<date>.md, the Step-5 output) exists with
an mtime after the session began.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
The 2026-07-03 finish run wedged because _completed() returned True while the run
was still mid-work — so the watchdog exited early and nothing recovered the wedge.
Cause: it scanned part.get('text') across ALL message parts, so DONE_MARKER inside
a TOOL part (a subagent `task` prompt / bash command that referenced 'print
UPGRADE RUN COMPLETE') matched. Now: require the marker in the LAST assistant
TEXT (prose) message — the genuine sign-off — ignoring tool-call args and any
mid-run echo of the instruction (work after the echo disqualifies it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
More fixes from live-running the finish of the 2026-07-03 run:
- _session_idle_min(): measure staleness across the whole session TREE (top-level
run + all descendant subagents), via the opencode server time.updated, NOT the
tmux log mtime (which freezes when a headless run doesn't stream to the pane).
A per-recipe subagent deploy runs 20-40min during which the PARENT session's
updated time is stale — reading the parent alone looked 'idle' and would false-
resume, killing the productive run. Renamed from _log_idle_min (kept as alias).
- watchdog(): wrap each poll in try/except so a transient server blip/race can
never kill the watchdog (a dead watchdog silently abandons the run).
- watchdog + supervisor now read the tree-aware session idle instead of log mtime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Live-testing the resume path surfaced two gaps: (1) an `opencode run` proc
EXITS when the model ends its turn, so a long /upgrade-all run's process dies
repeatedly before the whole run completes — and the log mtime freezes on death,
so the watchdog's log-idle>15min signal is both too slow and unreliable. (2) A
resumed run had no watchdog, so nothing re-continued it.
- watchdog(): detect PROC-DEATH (no live `opencode run` proc for the session +
not completed) and resume promptly, in addition to log-idle. Guarded by
MAX_RESUMES (default 20) so a no-progress loop (e.g. disk-full) eventually hands
off to the supervisor/operator instead of spinning forever.
- resume(): auto-spawn a watchdog if none is alive (skips when the watchdog itself
called resume — it lives in {SESSION}-watchdog — so no duplicate).
- launch-supervisor.py gate: defer while the per-run watchdog is alive (it is the
single writer for prompt-recovery). The supervisor only takes over once the
watchdog gives up (MAX_RESUMES) — i.e. a wedge a bare resume can't fix. Removes
the supervisor/watchdog double-resume race.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
Root-cause fix for the 2026-07-03 run stalling: the cc-ci host disk filled to
100% (ENOSPC) mid-run (Wave 6, lasuite-drive), the agent stopped to reclaim
space, and nothing resumed it — the log-idle/429 watchdog only covers opencode-go
usage-limit stalls, not an environmental wedge.
- launch-upgrader.py: step-0 prereclaim_cc_ci() prunes STALE cc-ci docker images
(unused AND older than a week, so this week's likely-reused images stay) before
each weekly run. Best-effort; env-tunable (UPGRADER_PRERECLAIM*).
- launch-supervisor.py (new): hourly glm-5.2 orchestrator wake-up. Cheap
deterministic gate — no-ops (zero tokens) when the run is complete or
progressing; only when a run stalled/died before completing does it launch a
short-lived glm-5.2 agent to diagnose + drive it to a clean DONE. Progress is
judged by live run-proc + log mtime (session_busy() is claude-tuned and misreads
a headless opencode run as idle).
- configuration.nix: cc-ci-upgrade-supervisor service + hourly timer (:07).
- upgrade-all SKILL §0: note the stale-image reclaim for manual runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WxbpH3DquKzoSTSwGvGuET
autonomic-bot
autonomic-bot
notplantsandClaude
cc-ci-bot
cc-ci
cc-ci-bot