The operator's AUTONOMIC_OPENCODE_KEY is a Go subscription key. Put in the
`opencode` (ZEN) slot it authenticates but every request fails "Insufficient
balance", because ZEN is pay-as-you-go credit; on the `opencode-go` endpoint
the same key answers fine. So the orchestrator agent moves to
opencode-go/glm-5.2, and README records that this host is a Go host and that
its opencode config must carry no inline apiKey.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
nix/modules/auto-update.nix (own module, no notplants-nix dependency): Tuesday
03:00 UTC, busy-gated (CI run, weekly upgrader, report, sweep, running Drone
builds → skip), `nix flake update` → build → switch-to-configuration test →
cc-ci health checks (sshd, 0 failed units, core units, every swarm service at
replica count, sops decrypted, dashboard/reports/drone 200, opencode UI 401)
→ profile + bootloader → flake.lock committed and pushed to main → /etc/cc-ci
fast-forwarded; revert + lock restore on failure; one-line state file for
/cc-ci-status.
Skills (.opencode canonical, .claude pointers' descriptions synced):
- cc-ci-orchestrator-update: THE host update — drives the auto-update unit by
hand; --cc-ci-only for a cc-ci-main-only move; failure playbook.
- cc-ci-server-update: delegates to it and explains why the old procedure
(rebuilding the cc-ci repo's standalone #cc-ci) must not be run on this host.
- cc-ci-update: chains orchestrator-update then tests-update.
- cc-ci-status: §5/§6 for one host — auto-update state, generation vs boot
generation, front doors (oc.ci = 401), sops, fail2ban, timers, orchestrator
agent session, secrets inventory; verdict updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The 2026-09-07 weekly run's step-0 orphan sweep removed the `opencode-ui`
swarm stack (the traefik route to the opencode web UI) because it was not
on sweep-orphans.sh's keep-list. Renamed to ccci-opencode-ui alongside the
other control-plane stacks and added to KEEP_RE.
Journal: the run itself (lasuite-docs #8, n8n #7 GREEN on the new Drone,
report week-2026-09-07 published) ran entirely on the new host.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator: no ssh keys or non-essential secrets from the old orchestrator
box on the cc-ci host. `ssh cc-ci` uses cc-ci-local-ed25519 (generated on
the host, pub in nix/hosts/cc-ci/ssh-keys); pushes to Gitea use
autonomic-bot-cc-ci-ed25519 (generated on the host, registered on the bot
account); root reuses that file for the cc-ci-secrets submodule. README §4:
the /secrets/files inventory shrinks to exactly what cc-ci needs, and §4b
now says to make the host a sops recipient and take the master key away.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator 2026-09-07 (evening): Claude sessions stay on notplants-orchestrator;
on the cc-ci host the orchestrator is an opencode agent (agents.toml:
opencode/glm-5.2) steered from https://oc.ci.commoninternet.net, next to the
upgrader/report sessions. claude-install and CLAUDE_BIN are gone from the
modules; launch-upgrader/report print the real UI URL; README "Operating the
orchestrator" rewritten and a "Weekly upgrade run" section added.
Secrets: only cc-ci's, under /secrets/files with runtime paths symlinked
(README §4 table), nothing from other projects — tangled key and the
tailscale line dropped from the host.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
During the DNS cutover the Hetzner resolvers nixos-infect had left in
networking.nix still served the old address (3h TTL) while 1.1.1.1/8.8.8.8
had the new one, so every other lookup of drone.ci.commoninternet.net from
the box itself went to the old server: runner "cannot request stage 404",
bridge "drone trigger failed 404", traefik probes alternating 200/404.
Drop those resolvers; pin ci/drone/report/traefik/warm-keycloak/oc names to
this host in networking.hosts. README §7 records it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Enabling FILE_PROVIDER_DIRECTORY_ENABLED in the coop-cloud traefik recipe
REPLACES its single provider file (wildcard cert + `security` middleware),
which took every cc-ci front door down for two minutes on 2026-09-07.
Reverted. The route is now what every recipe does: a one-container swarm
stack (`opencode-ui`, alpine/socat relay to nginx on the docker bridge)
carrying the traefik router labels, deployed by opencode-ui-route.service.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator: standard 443, routed by domain. Traefik already owns 443 with the
*.ci.commoninternet.net cert, so opencode-ui-route.service drops a dynamic
config into the traefik stack's watched file-providers volume routing
oc.ci.commoninternet.net -> http://172.18.0.1:8080, where nginx (reachable
only on docker_gwbridge) enforces the basic auth and logs 401s with the
real client IP (real_ip from X-Forwarded-For). The fail2ban nginx jail bans
in DOCKER-USER, since that traffic is forwarded, not INPUT. 8443 and the
ACME-host variant are gone.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Operator 2026-09-07: the combined host stays off the tailnet. ssh is
key-only (password + keyboard-interactive auth off), the opencode UI is
served as oc.ci.commoninternet.net on 8443 — TLS from the CI server's
*.ci.commoninternet.net ACME cert (new orchestrator-host option
opencodeUiAcmeHost; nginx joins the acme group) behind the existing basic
auth — and fail2ban watches sshd plus nginx's 401s (nginx-http-auth jail).
tailscaled ordering dropped from the orchestrator units. README updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Found on the first activation of #cc-ci on 195.201.88.249: the opencode
installer lands in ~/.opencode/bin (opencode-web then failed at EXEC), and
nginx's pre-start config check runs as the nginx user, so the staged
cert/htpasswd must be root:nginx 0640, not root:root 0600.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
hardware.nix / networking.nix are now the nixos-infect output of
195.201.88.249 (ESP E079-7D41, eth0 MAC-pinned), with the one edit that
matters: defaultGateway as an attrset with interface = "eth0".
README §2a records what it actually took to get NixOS running on the
Debian 13 image: NO_SWAP=true (tmpfs /tmp), a manual lustration from rescue
mode because 26.05's systemd initrd did not honour NIXOS_LUSTRATE (Debian's
unit files shadowed every NixOS service), and the gateway fix applied via a
chroot rebuild with the nix sandbox off.
flake: cc-ci input back on main (PR #32 merged as f6dbfa3); loops uid pinned
to 1000 so workspace rsyncs by uid line up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
The orchestrator's flake now builds the machine it shares with the cc-ci CI
server: `nixosConfigurations.cc-ci` composes cc-ci's nixosModules.cc-ci-server
(new flake input, nixpkgs + sops-nix follow ours), this repo's orchestrator
module (nix/modules/cc-ci.nix, exported as cc-ci-orchestrator, `cc-ci` kept
as an alias for notplants-nix) and the new nix/modules/orchestrator-host.nix
— the host contract those units always assumed (loops user, claude/opencode
CLIs, opencode web server + tailnet-only UI on 8443 since traefik owns
80/443, nix-ld, tool set, `ssh cc-ci` → loopback).
nix/hosts/cc-ci/{hardware,networking}.nix are PROVISIONAL copies of the old
server's layout so the flake evaluates; they get replaced by the
nixos-infect output of 195.201.88.249.
README.md is the deploy guide (Hetzner Debian → nixos-infect → this flake →
staging → data restore → cutover). archive/ holds the retired Incus/Hetzner
orchestrator host configs, the old terraform and the migration plans;
references updated. cc-ci-plan/plan-cc-ci-combined-host.md is the working
plan for the move.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FqkQq3CDmFWcQ7u1LzoyRz
Gitea repos renamed:
cc-ci-autonomous-orchestrator → cc-ci-orchestrator
cc-ci-orchestrator → archived-cc-ci-orchestrator
Updated in this workspace:
- README.md, AGENTS.md: repo title
- cc-ci-plan/plan-orchestrator-migration.md: cc-ci-autonomous-orchestrator refs
- cc-ci-plan/plan-repo-consolidation.md: marked complete + Pi remote-update notice
- cc-ci-plan/launch-orchestrator.sh, launch.sh: session naming comment cleanup
NOTE: Pi clone still has the old origin URL. On the Pi, run:
git remote set-url origin https://git.autonomic.zone/recipe-maintainers/cc-ci-orchestrator.git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Records the exact sequence to keep the orchestrator alive in tmux and resume it
with remote control (survives disconnects/laptop close), reconnect commands, and
pointers to launch/supervision docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
