cc-ci-orchestrator/AGENTS.md

# cc-ci-orchestrator — AGENTS.md

This folder is the **orchestrator** workspace for building the **cc-ci** Co-op Cloud recipe CI
server. It holds the plan, the launch/supervision tooling, and the two loop prompts. The actual CI
project (NixOS config, test runner, recipe tests) lives in a **separate** repo the loops create at
`git.autonomic.zone/recipe-maintainers/cc-ci` — do not confuse the two.

## The agent map lives in `cc-ci-plan/orchestration.md`

That doc is the **root structure**: every agent (Builder, Adversary, Orchestrator, Assistant,
Upgrader) → its prompt + plan, plus the watchdog and the git coordination protocol. Read it first.

In short: the **Orchestrator** (*this* session/role) supervises and keeps everyone on track — it is
**separate** from the loops and is the only role that should power-cycle/recreate the host. The
**Builder** and **Adversary** loops coordinate **only** through the cc-ci git repo (`plan.md` §6.1);
the orchestrator watches from outside.

## On startup: read the journal, announce yourself, report reboots

**Every time you (the orchestrator) start or resume:**
1. **Read `cc-ci-plan/JOURNAL.md`** — the most recent `## Session` entry is where the previous
   session left off. This is the persistent handoff record; read it before anything else.
2. Read `cc-ci-plan/REBOOTS.md` (count entries) and run `cc-ci-plan/launch.sh status`
   (current phase + whether loops/watchdog are running).
3. **`PushNotification`** (proactive): *"cc-ci orchestrator online — phase X, loops+watchdog
   running; N reboots logged (last <date>)."*
4. If loops are down, relaunch: `RESUME_PHASE=1 cc-ci-plan/launch.sh start`.
5. **On handoff / end of session:** append a `## Session` block to `JOURNAL.md` summarising
   what happened, current state, and open items (see format in that file).

Reboot resilience is handled by **`cc-ci-loops.service`** (system unit): on boot it logs the reboot
to `REBOOTS.md` (boot_id-gated) and runs `launch.sh start` with `RESUME_PHASE=1`, so the loops +
watchdog auto-resume the saved phase. The orchestrator session itself is relaunched by
`cc-ci-orchestrator.service` (`agents.py up orchestrator`) as an **opencode agent** the operator
steers from https://oc.ci.commoninternet.net (no Claude on the cc-ci host; Claude sessions run on
the notplants-orchestrator box and reach cc-ci over ssh). Since 2026-09 the orchestrator runs on the **same Hetzner
host as the cc-ci CI server** (`cc-ci`, public `195.201.88.249`, tailnet `cc-ci`), declared by
`nixosConfigurations.cc-ci` in this repo's `flake.nix`, which imports the CI server from the cc-ci
repo's `nixosModules.cc-ci-server`. `ssh cc-ci` from the loops user therefore goes to loopback.
The full provisioning + deploy guide is `README.md`; the move is recorded in
`cc-ci-plan/plan-cc-ci-combined-host.md`; the previous hosts (Pi → Incus VM → Hetzner `cpx22`
shared with notplants) are in `archive/`. Rebuild this host with
`nixos-rebuild switch --flake .#cc-ci` from `/srv/cc-ci-orch` — but **always
`nixos-rebuild test` the same flake target first and verify the host is still healthy/reachable
before the `switch`** (general policy for nix deploys to this host: `test`
leaves the bootloader and system profile untouched, so a reboot always recovers to the
last-known-good generation; the 2026-08-03 cc-ci 26.05 bump outage is the cautionary tale, see
`.cc-ci-logs/server-update-2026-08-03.md`).

## Keep the orchestrator open, under remote-control

Run this session as a long-lived **interactive** session with `--remote-control` so the operator can
check in on the loops and steer/restart things from **claude.ai/code** (or the Claude mobile app)
without being at the terminal.

- **Already in the session?** Just run `/remote-control` — it attaches claude.ai/code to the live
  conversation (no exit, no resume needed).
- **Starting fresh:** `claude --remote-control 'autonomous-orchestrator' --dangerously-skip-permissions`
- **Resuming this orchestrator later (history preserved):**
  ```bash
  claude --resume autonomous-orchestrator --remote-control "autonomous-orchestrator" --dangerously-skip-permissions
  ```
  Note the two names are different: `--resume <name|id>` restores *this conversation* (the name set
  via `-n/--name`, shown in the `/resume` picker); the `--remote-control [name]` value is only the
  web display label and resumes nothing. The conversation persists on disk across exits; remote
  control itself only stays "connected" while the local process is alive (resume + re-enable to get
  it back after a full exit).

Use it to: tail loop logs (`cc-ci-plan/launch.sh logs builder|adversary|watchdog`), inspect
`STATUS.md`/`REVIEW.md` in the cc-ci repo, edit the plan or prompts, restart a stuck loop, or
power-cycle/recreate the cc-ci VM (see `cc-ci-plan/kickoff.md` → "Fallback: restart/recreate the
cc-ci VM"). The orchestrator is the human's steering wheel; the loops are the engine.

## Loop file-location rule (machine-docs/)

In the **cc-ci loops repo**, ALL coordination / loop-state files live under `machine-docs/`, NEVER
the repo root: phase-namespaced `STATUS-*.md` / `BACKLOG-*.md` / `REVIEW-*.md` / `JOURNAL-*.md`,
shared `DECISIONS.md` / `DEFERRED.md`, and the `ADVERSARY-INBOX.md` / `BUILDER-INBOX.md`
side-channels. This is enforced in the kickoff (`launch.py build_kickoff`), the loop prompts, and
the loops' own `AGENTS.md`. If the orchestrator ever sees such a file at the root, move it into
`machine-docs/` and check the kickoff/prompts still mandate it. (The watchdog's `resolve_state` and
INBOX handoff already read `machine-docs/` first, so this is the single canonical home.)

## Launch & supervise the loops

- **Source of truth for the loops:** `cc-ci-plan/plan.md` (mission, Definition of Done, §1.5
  credential map, §6 two-agent protocol, §7 loop discipline).
- **Launch/supervision guide:** `cc-ci-plan/kickoff.md`.
- `cc-ci-plan/launch.sh start` → both loops (interactive `--remote-control` in tmux) + a watchdog.
  tmux is installed; `launch.sh` defaults now point at `/srv/cc-ci/...`.

## Access & credentials (pointers only — values are gitignored)

- `.testenv` (**NOT committed**): Tailscale auth key + Gitea bot creds. Load with
  `set -a; . .testenv; set +a` (never echo the values).
- **cc-ci:** `ssh cc-ci` (root) directly — the orchestrator VM is a direct tailnet peer (`100.90.116.4`).
  No proxy. Key: `~/.ssh/cc-ci-root-ed25519`. If unreachable, check `tailscale status`.
- **Incus/VM fallback:** mTLS certs at `/srv/incus-terraform-nix-vm-creator/terraform-secrets/`;
  b1 is on the same tailnet (reach via the same proxy). See kickoff "Fallback".
- **Full credential map + how to use each:** `plan.md` §1.5.

## Hard rule

Never commit secret values. `.testenv`, `*.tfstate`, `*.key`/`*.pem`, and the loop runtime/clone
dirs are gitignored. Reference secret *locations*, never their contents (`plan.md` §9).

## Agent memory lives in `memory/` (in this repo)

The orchestrator's persistent agent memory is the **`memory/`** directory of this repo — one file
per fact with frontmatter, indexed by `memory/MEMORY.md`. The Claude auto-memory path
(`~/.claude/projects/-srv-cc-ci-orch/memory`) is a **symlink** to it, so memories written the normal
way land in the repo automatically. **Future memories must also go there**: after writing or
updating a memory file (and its `MEMORY.md` index line), commit it here and push, like any other
intentional repo change. Never put secret values in a memory file (see Hard rule).

## Commit discipline

When the orchestrator, Builder, or assistant makes intentional repository changes here, commit them
promptly and push them to `git.autonomic.zone` in append-only fashion (never force-push). Match the
existing commit author and message style in this repo. Do not bundle unrelated worktree changes you
did not make; stage only the intended files.

## Ship as PRs, merge them yourself, operator reviews retrospectively

**This applies to the two INFRASTRUCTURE repos — `recipe-maintainers/cc-ci-orchestrator` (here) and
`recipe-maintainers/cc-ci` (the CI product).** For work in either:

1. Branch, don't commit straight to `main`.
2. Open a PR with a description written to be read **after** the fact: what changed, why, and what
   evidence says it works (test output, a verified run, a before/after number). The PR *is* the
   review artifact and the historical record.
3. **Merge it yourself once it is verified** — do not wait for review. The invocation is the
   authorization; blocking on review would stall the pipeline these repos exist to run.
4. The operator reviews **retrospectively**, from the PR.

So the PR is not a gate — it is how the work stays legible. A PR that merely says "fix scanner" has
failed at its only job.

> ### This does NOT extend to RECIPE repos
> Recipe PRs — any `coop-cloud/<recipe>` or its `recipe-maintainers/<recipe>` mirror — are
> **created and verified but NEVER merged by an agent**. Those change what deploys on other people's
> infrastructure, so a human merges them. The split is deliberate: agents own the tooling, the
> operator owns the recipes.

If work has already landed on `main` without a PR, do not rewrite published history to fix it.
Create a branch pinned at the pre-work commit and open the PR against that, so the diff is still
reviewable and merging only advances the pointer (see PRs #2-#5, 2026-08-11).