The operator's AUTONOMIC_OPENCODE_KEY is a Go subscription key. Put in the
`opencode` (ZEN) slot it authenticates but every request fails "Insufficient
balance", because ZEN is pay-as-you-go credit; on the `opencode-go` endpoint
the same key answers fine. So the orchestrator agent moves to
opencode-go/glm-5.2, and README records that this host is a Go host and that
its opencode config must carry no inline apiKey.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
`vi` and `vim` were both absent on the cc-ci host (operator hit it over ssh);
the vim package ships a `vi` binary, so one entry covers both names, and
EDITOR=vim makes git/systemctl/visudo open something that exists.
sqlite earns its place beyond convenience: Drone's build and step logs live in
its sqlite volume, and the runbook plus /cc-ci-status tell you to read them
there — until now every such check needed an ad-hoc `nix-shell -p sqlite`.
Installed via environment.systemPackages, so they are on PATH for BOTH root and
loops through /run/current-system/sw/bin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday
03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone
builds → skip), `nix flake update` → build → switch-to-configuration test →
cc-ci health checks (sshd, 0 failed units, core units, every swarm service at
replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401)
→ profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci
fast-forwarded; revert + lock restore on failure; one-line state file for
/cc-ci-status.
Skills (.opencode canonical, .claude pointers' descriptions synced):
- cc-ci-orchestrator-update: THE host update — drives the auto-update unit by
hand; --cc-ci-only for a cc-ci-main-only move; failure playbook.
- cc-ci-server-update: delegates to it and explains why the old procedure
(rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host.
- cc-ci-update: chains orchestrator-update then tests-update.
- cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot
generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator
agent session, secrets inventory; verdict updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The 2026-09-07 weekly run's step-0 orphan sweep removed the `opencode-ui`
swarm stack (the traefik route to the opencode web UI) because it was not
on sweep-orphans.sh's keep-list. Renamed to ccci-opencode-ui alongside the
other control-plane stacks and added to KEEP_RE.
Journal: the run itself (lasuite-docs #8, n8n #7 GREEN on the new Drone,
report week-2026-09-07 published) ran entirely on the new host.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator: no ssh keys or non-essential secrets from the old orchestrator
box on the cc-ci host. `ssh cc-ci` uses cc-ci-local-ed25519 (generated on
the host, pub in nix/hosts/cc-ci/ssh-keys); pushes to Gitea use
autonomic-bot-cc-ci-ed25519 (generated on the host, registered on the bot
account); root reuses that file for the cc-ci-secrets submodule. README §4:
the /secrets/files inventory shrinks to exactly what cc-ci needs, and §4b
now says to make the host a sops recipient and take the master key away.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator 2026-09-07 (evening): Claude sessions stay on notplants-orchestrator;
on the cc-ci host the orchestrator is an opencode agent (agents.toml:
opencode/glm-5.2) steered from https://oc.ci.commoninternet.net, next to the
upgrader/report sessions. claude-install and CLAUDE_BIN are gone from the
modules; launch-upgrader/report print the real UI URL; README "Operating the
orchestrator" rewritten and a "Weekly upgrade run" section added.
Secrets: only cc-ci's, under /secrets/files with runtime paths symlinked
(README §4 table), nothing from other projects — tangled key and the
tailscale line dropped from the host.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
During the DNS cutover the Hetzner resolvers nixos-infect had left in
networking.nix still served the old address (3h TTL) while 1.1.1.1/8.8.8.8
had the new one, so every other lookup of drone.ci.commoninternet.net from
the box itself went to the old server: runner "cannot request stage 404",
bridge "drone trigger failed 404", traefik probes alternating 200/404.
Drop those resolvers; pin ci/drone/report/traefik/warm-keycloak/oc names to
this host in networking.hosts. README §7 records it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Enabling FILE_PROVIDER_DIRECTORY_ENABLED in the coop-cloud traefik recipe
REPLACES its single provider file (wildcard cert + `security` middleware),
which took every cc-ci front door down for two minutes on 2026-09-07.
Reverted. The route is now what every recipe does: a one-container swarm
stack (`opencode-ui`, alpine/socat relay to nginx on the docker bridge)
carrying the traefik router labels, deployed by opencode-ui-route.service.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator: standard 443, routed by domain. Traefik already owns 443 with the
*.ci.commoninternet.net cert, so opencode-ui-route.service drops a dynamic
config into the traefik stack's watched file-providers volume routing
oc.ci.commoninternet.net -> http://172.18.0.1:8080, where nginx (reachable
only on docker_gwbridge) enforces the basic auth and logs 401s with the
real client IP (real_ip from X-Forwarded-For). The fail2ban nginx jail bans
in DOCKER-USER, since that traffic is forwarded, not INPUT. 8443 and the
ACME-host variant are gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator 2026-09-07: the combined host stays off the tailnet. ssh is
key-only (password + keyboard-interactive auth off), the opencode UI is
served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's
*.ci.commoninternet.net ACME cert (new orchestrator-host option
opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic
auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail).
tailscaled ordering dropped from the orchestrator units. README updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Found on the first activation of #cc-ci on 195.201.88.249: the opencode
installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and
nginx's pre-start config check runs as the nginx user, so the staged
cert/htpasswd must be root:nginx 0640, not root:root 0600.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
hardware.nix / networking.nix are now the nixos-infect output of
195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that
matters: defaultGateway as an attrset with interface = "eth0".
README §2a records what it actually took to get NixOS running on the
Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue
mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's
unit files shadowed every NixOS service), and the gateway fix applied via a
chroot rebuild with the nix sandbox off.
flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned
to 1000 so workspace rsyncs by uid line up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).
nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.
README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The Gitea host was stored in the credentials env file, so a literal
secret scanner treated it as a secret and refused two READMEs in another
repo that link to git.autonomic.zone.
It was never secret: advisory-scan.py, pr-survey.py and recipe-report.py
already default it to git.autonomic.zone, and it is this repo's own
remote host. It is a bare hostname with no scheme, userinfo or path.
Removed from the env file and defaulted here the same way the Python
already does. GITEA_USERNAME and GITEA_PASSWORD keep their :? guard --
those are real credentials and must still fail loudly if unset.
The orchestrator host is a general agent box that also serves several unrelated
projects (atproto-likes, the lichen and project orchestrators, the shared
opencode web server). Everything that exists FOR cc-ci — the loops, the cc-ci
orchestrator session, the weekly recipe-upgrade run and its hourly supervisor —
now lives in nix/modules/cc-ci.nix and is exported as `nixosModules.cc-ci`, so
the two can evolve and be reviewed independently and another host can consume
just the cc-ci half.
No behaviour change. Verified by building .#cc-ci-orchestrator-hetzner after the
split: the unit set is identical to the running system, and all six cc-ci units
are byte-identical modulo nix store hashes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016L6nYYwkCWnrEFKTnKAfet
Move plausible from weekly to external in used-recipes.md. cc-ci still
deploys/tests it (the recipe + tests stay enrolled, !testme bridge
unchanged), but it is no longer surveyed/upgraded by the weekly /upgrade-all
run — maintained elsewhere, like uptime-kuma.
The open plausible PR #5 (clickhouse 23.4.2.11 -> 23.4.6.25) from this week's
run stays open for the operator; no future weekly PRs will be opened for it.
The weekly /upgrade-all parent session and the /recipe-report session now
run on tinfoil/deepseek-v4-pro (LOOP_MODEL + REPORT_MODEL in upgrader.env).
The hourly supervisor stays on opencode-go/glm-5.2 (SUPERVISOR_MODEL default
in launch-supervisor.py, not overridden). Subagents already bind deepseek via
the cc-ci repo's opencode config (fix from 2026-08-10, verified this week:
all 16 subagents across both waves ran deepseek-v4-pro).
LOOP_TIER=zen is kept so the tier check passes; the watchdog's usage-limit
probe sends the deepseek model name to the zen endpoint, which returns 200
(not 429) → resume immediately — correct, since tinfoil has no rolling usage
limit to wait out. Verified the probe behaviour with a direct curl.
Root cause: the 2026-08-14 run stalled mid-recipe on 'Insufficient balance'
(opencode zen workspace balance exhausted), then sat unfinished for 40h while
the supervisor cron spun hourly unable to recover it. Deepseek (pay-per-use
API key) has no rolling balance limit, so this can't recur.
Also documents the session recovery in JOURNAL.md (the stalled run was
completed via a fresh scoped upgrader — the original 2.58M-token session was
unresumable: the inference endpoint silently drops the oversized request).
notplantsandClaude Opus 5
auto-update
autonomic-bot
autonomic-bot