cc-ci-orchestrator/AGENTS.md

cc-ci-orchestrator — AGENTS.md

This folder is the orchestrator workspace for building the cc-ci Co-op Cloud recipe CI server. It holds the plan, the launch/supervision tooling, and the two loop prompts. The actual CI project (NixOS config, test runner, recipe tests) lives in a separate repo the loops create at git.autonomic.zone/recipe-maintainers/cc-ci — do not confuse the two.

The agent map lives in cc-ci-plan/orchestration.md

That doc is the root structure: every agent (Builder, Adversary, Orchestrator, Assistant, Upgrader) → its prompt + plan, plus the watchdog and the git coordination protocol. Read it first.

In short: the Orchestrator (this session/role) supervises and keeps everyone on track — it is separate from the loops and is the only role that should power-cycle/recreate the host. The Builder and Adversary loops coordinate only through the cc-ci git repo (plan.md §6.1); the orchestrator watches from outside.

On startup: read the journal, announce yourself, report reboots

Every time you (the orchestrator) start or resume:

1. Read cc-ci-plan/JOURNAL.md — the most recent ## Session entry is where the previous session left off. This is the persistent handoff record; read it before anything else.
2. Read cc-ci-plan/REBOOTS.md (count entries) and run cc-ci-plan/launch.sh status (current phase + whether loops/watchdog are running).
3. PushNotification (proactive): "cc-ci orchestrator online — phase X, loops+watchdog running; N reboots logged (last )."
4. If loops are down, relaunch: RESUME_PHASE=1 cc-ci-plan/launch.sh start.
5. On handoff / end of session: append a ## Session block to JOURNAL.md summarising what happened, current state, and open items (see format in that file).

Reboot resilience is handled by cc-ci-loops.service (system unit): on boot it logs the reboot to REBOOTS.md (boot_id-gated) and runs launch.sh start with RESUME_PHASE=1, so the loops + watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by cc-ci-orchestrator.service (agents.py up orchestrator) — the operator reconnects to it (that's why the startup notification matters). Since 2026-09 the orchestrator runs on the same Hetzner host as the cc-ci CI server (cc-ci, public 195.201.88.249, tailnet cc-ci), declared by nixosConfigurations.cc-ci in this repo's flake.nix, which imports the CI server from the cc-ci repo's nixosModules.cc-ci-server. ssh cc-ci from the loops user therefore goes to loopback. The full provisioning + deploy guide is README.md; the move is recorded in cc-ci-plan/plan-cc-ci-combined-host.md; the previous hosts (Pi → Incus VM → Hetzner cpx22 shared with notplants) are in archive/. Rebuild this host with nixos-rebuild switch --flake .#cc-ci from /srv/cc-ci-orch — but always nixos-rebuild test the same flake target first and verify the host is still healthy/reachable before the switch (general policy for nix deploys to this host: test leaves the bootloader and system profile untouched, so a reboot always recovers to the last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see .cc-ci-logs/server-update-2026-08-03.md).

Keep the orchestrator open, under remote-control

Run this session as a long-lived interactive session with --remote-control so the operator can check in on the loops and steer/restart things from claude.ai/code (or the Claude mobile app) without being at the terminal.

• Already in the session? Just run /remote-control — it attaches claude.ai/code to the live conversation (no exit, no resume needed).
• Starting fresh: claude --remote-control 'autonomous-orchestrator' --dangerously-skip-permissions
• Resuming this orchestrator later (history preserved):

  claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
  

  Note the two names are different: --resume <name|id> restores this conversation (the name set via -n/--name, shown in the /resume picker); the --remote-control [name] value is only the web display label and resumes nothing. The conversation persists on disk across exits; remote control itself only stays "connected" while the local process is alive (resume + re-enable to get it back after a full exit).

Use it to: tail loop logs (cc-ci-plan/launch.sh logs builder|adversary|watchdog), inspect STATUS.md/REVIEW.md in the cc-ci repo, edit the plan or prompts, restart a stuck loop, or power-cycle/recreate the cc-ci VM (see cc-ci-plan/kickoff.md → "Fallback: restart/recreate the cc-ci VM"). The orchestrator is the human's steering wheel; the loops are the engine.

Loop file-location rule (machine-docs/)

In the cc-ci loops repo, ALL coordination / loop-state files live under machine-docs/, NEVER the repo root: phase-namespaced STATUS-*.md / BACKLOG-*.md / REVIEW-*.md / JOURNAL-*.md, shared DECISIONS.md / DEFERRED.md, and the ADVERSARY-INBOX.md / BUILDER-INBOX.md side-channels. This is enforced in the kickoff (launch.py build_kickoff), the loop prompts, and the loops' own AGENTS.md. If the orchestrator ever sees such a file at the root, move it into machine-docs/ and check the kickoff/prompts still mandate it. (The watchdog's resolve_state and INBOX handoff already read machine-docs/ first, so this is the single canonical home.)

Launch & supervise the loops

• Source of truth for the loops: cc-ci-plan/plan.md (mission, Definition of Done, §1.5 credential map, §6 two-agent protocol, §7 loop discipline).
• Launch/supervision guide: cc-ci-plan/kickoff.md.
• cc-ci-plan/launch.sh start → both loops (interactive --remote-control in tmux) + a watchdog. tmux is installed; launch.sh defaults now point at /srv/cc-ci/....

Access & credentials (pointers only — values are gitignored)

• .testenv (NOT committed): Tailscale auth key + Gitea bot creds. Load with set -a; . .testenv; set +a (never echo the values).
• cc-ci: ssh cc-ci (root) directly — the orchestrator VM is a direct tailnet peer (100.90.116.4). No proxy. Key: ~/.ssh/cc-ci-root-ed25519. If unreachable, check tailscale status.
• Incus/VM fallback: mTLS certs at /srv/incus-terraform-nix-vm-creator/terraform-secrets/; b1 is on the same tailnet (reach via the same proxy). See kickoff "Fallback".
• Full credential map + how to use each: plan.md §1.5.

Hard rule

Never commit secret values. .testenv, *.tfstate, *.key/*.pem, and the loop runtime/clone dirs are gitignored. Reference secret locations, never their contents (plan.md §9).

Agent memory lives in memory/ (in this repo)

The orchestrator's persistent agent memory is the memory/ directory of this repo — one file per fact with frontmatter, indexed by memory/MEMORY.md. The Claude auto-memory path (~/.claude/projects/-srv-cc-ci-orch/memory) is a symlink to it, so memories written the normal way land in the repo automatically. Future memories must also go there: after writing or updating a memory file (and its MEMORY.md index line), commit it here and push, like any other intentional repo change. Never put secret values in a memory file (see Hard rule).

Commit discipline

When the orchestrator, Builder, or assistant makes intentional repository changes here, commit them promptly and push them to git.autonomic.zone in append-only fashion (never force-push). Match the existing commit author and message style in this repo. Do not bundle unrelated worktree changes you did not make; stage only the intended files.

Ship as PRs, merge them yourself, operator reviews retrospectively

This applies to the two INFRASTRUCTURE repos — recipe-maintainers/cc-ci-orchestrator (here) and recipe-maintainers/cc-ci (the CI product). For work in either:

1. Branch, don't commit straight to main.
2. Open a PR with a description written to be read after the fact: what changed, why, and what evidence says it works (test output, a verified run, a before/after number). The PR is the review artifact and the historical record.
3. Merge it yourself once it is verified — do not wait for review. The invocation is the authorization; blocking on review would stall the pipeline these repos exist to run.
4. The operator reviews retrospectively, from the PR.

So the PR is not a gate — it is how the work stays legible. A PR that merely says "fix scanner" has failed at its only job.

This does NOT extend to RECIPE repos

Recipe PRs — any coop-cloud/<recipe> or its recipe-maintainers/<recipe> mirror — are created and verified but NEVER merged by an agent. Those change what deploys on other people's infrastructure, so a human merges them. The split is deliberate: agents own the tooling, the operator owns the recipes.

If work has already landed on main without a PR, do not rewrite published history to fix it. Create a branch pinned at the pre-work commit and open the PR against that, so the diff is still reviewable and merging only advances the pointer (see PRs #2-#5, 2026-08-11).