recipe-maintainers/cc-ci-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

decommission Pi: update all docs for VM-only setup

Browse Source 
The orchestrator Pi is retired (2026-05-31). All agents now run on the
cc-ci-orchestrator VM (NixOS, loops user, /srv/cc-ci). The VM is a
direct tailnet peer to cc-ci — no SOCKS proxy, no userspace tailscaled,
no ProxyCommand. Updated across all affected files:

AGENTS.md
  - Remove Pi from reboot description; migration complete (not "parked")
  - cc-ci access: direct ssh, not via proxy

kickoff.md
  - Prerequisites: direct tailnet peer, not proxy
  - Host deps: NixOS (not apt)
  - Fallback/Incus: b1 reachable directly, no --proxy curl flag

plan.md §1 + §1.5
  - §1 bootstrap: direct SSH, check tailscale status (not restart proxy)
  - §1.5 intro: "VM" not "sandbox host"; no proxy
  - Credentials table: remove TS_AUTH_KEY row; update cc-ci SSH row
  - Replace "Tailscale connection (proxy)" subsection with direct-peer description

plan-orchestrator-migration.md
  - Mark COMPLETE (2026-05-31); historical record only

plan-phase1c-full-reproducibility.md
  - Incus access: direct, not via SOCKS proxy

prompts/builder.md + prompts/adversary.md
  - cc-ci access language only: direct ssh, no proxy restart instructions
  - adversary: *.ci.commoninternet.net via plain curl, no proxy flag

REBOOTS.md
  - Retitle for VM; note Pi retired; Pi entries marked historical

systemd/cc-ci-loops.service
  - User/Group/HOME/PATH: notplants → loops
  - Remove cc-ci-tailscaled.service dependency (no proxy on VM)
  - Add note about nix/configuration.nix as the authoritative VM declaration

test-e2e-testme-acceptance.md
  - tailscale status: no --socket flag
  - ssh to throwaway: no ProxyCommand

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-05-31 00:16:37 +00:00
co-authored by Claude Sonnet 4.6
parent db375bcc07
commit 01874821f2
10 changed files with 58 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cc-ci-plan/plan.md
+21 -31
View File
@@ -33,9 +33,9 @@ Do these in order. Each step is idempotent; re-running is safe.
1. **Verify access.** (Full credential map + how each is used is in **§1.5** — read it first.)
- `ssh cc-ci 'hostname && whoami'` — you log in as **root** on cc-ci (NixOS), so there is no
separate sudo step. `ssh cc-ci` is preconfigured to tunnel through the userspace-tailscaled
SOCKS proxy (§1.5); if it fails, the proxy/daemon is probably down — restart it (§1.5) before
declaring blocked.
separate sudo step. `ssh cc-ci` reaches cc-ci directly (the orchestrator VM is a direct tailnet
peer — no proxy; key `~/.ssh/cc-ci-root-ed25519`). If it fails, check `tailscale status`
before declaring blocked.
- `ssh cc-ci 'nixos-version'` — confirm NixOS.
- Confirm you can reach the Gitea API with the bot creds from `.testenv` (§1.5):
`curl -s https://$GITEA_URL/api/v1/version`. The bot authenticates with
@@ -72,45 +72,35 @@ Do these in order. Each step is idempotent; re-running is safe.
## 1.5 Credentials & access — where everything lives and how to use it
The loops run **on the sandbox host** (not on cc-ci) and reach cc-ci over Tailscale. This section
is the authoritative map of what credentials exist, where, and how to use them. **Never copy any
secret value into the repo, a commit, a log, or the dashboard** (§9) — reference locations only.
The loops run **on the cc-ci-orchestrator VM** (`100.116.55.106`, NixOS, `loops` user) and reach
cc-ci directly over Tailscale (direct tailnet peer — no proxy). This section is the authoritative
map of what credentials exist, where, and how to use them. **Never copy any secret value into the
repo, a commit, a log, or the dashboard** (§9) — reference locations only.
### Provided credentials (already in place)
| What | Where | How to use |
|---|---|---|
| **Tailscale auth key** (joins cc-ci's tailnet `taila4a0bf.ts.net`) | `/srv/cc-ci/.testenv` → `TS_AUTH_KEY` (Tailscale SaaS key, keyID ends `CNTRL`) | Used to bring up the userspace tailscaled (below). It's reusable; re-run `tailscale up` with it if the node drops. |
| **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; config `Host cc-ci` in `~/.ssh/config` | Just run `ssh cc-ci` (logs in as **root**). The pubkey is already in cc-ci's `/root/.ssh/authorized_keys`. |
| **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; `Host cc-ci` in `~/.ssh/config` (HostName `100.90.116.4`, no ProxyCommand) | Just run `ssh cc-ci` (logs in as **root**). The orchestrator VM is a direct tailnet peer — direct route, no proxy. Pubkey already in cc-ci's `/root/.ssh/authorized_keys`. |
| **Gitea bot account** | `/srv/cc-ci/.testenv` → `GITEA_USERNAME` (`autonomic-bot`), `GITEA_PASSWORD`, `GITEA_URL` (`git.autonomic.zone`) | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to push the `cc-ci` project repo, read recipe repos, comment on PRs, and poll for `!testme` (read-level; the bot does not register webhooks). |
Load them in a shell with: `set -a; . /srv/cc-ci/.testenv; set +a` (don't echo the values).
### The Tailscale connection (how `ssh cc-ci` and the proxy work)
### The Tailscale connection (how `ssh cc-ci` works)
cc-ci (`cc-nix-test`, **100.90.116.4**) is on a *different* tailnet than the sandbox host's default
one, so it is reached via a **second, userspace tailscaled** — this keeps the host's own tailnet
untouched. State lives in `~/.cc-ci-ts/`; it exposes a **SOCKS5/HTTP proxy on `127.0.0.1:1055`**,
which is the only route to that tailnet (userspace networking ⇒ the host OS can't route the tailnet
IPs directly).
cc-ci (`cc-nix-test`, **100.90.116.4`) is on the same tailnet as the orchestrator VM
(`taila4a0bf.ts.net`), so it is reached **directly** — no SOCKS proxy, no userspace tailscaled.
The VM's system tailscaled is on that tailnet; `ssh cc-ci` routes straight to cc-ci.
It runs as a **persistent systemd service** (`cc-ci-tailscaled.service`, enabled, `Restart=always`,
starts on boot; unit at `/etc/systemd/system/cc-ci-tailscaled.service`, runs as user `notplants`).
It reuses the already-authenticated state in `~/.cc-ci-ts/`, so it reconnects across reboots/crashes
without the auth key.
- `ssh cc-ci` works out of the box (its `ProxyCommand` uses the proxy; logs in as root).
- For HTTP(S) to cc-ci / `*.ci.commoninternet.net` from the sandbox, go through the proxy, e.g.
`curl --proxy socks5h://localhost:1055 https://<app>.ci.commoninternet.net`.
- **If connectivity is down:** `sudo systemctl restart cc-ci-tailscaled` (diagnose with
`systemctl status cc-ci-tailscaled` / `journalctl -u cc-ci-tailscaled`). A dead proxy is an access
failure to recover, not a `## Blocked`-and-stop condition — *unless* the auth key itself is
rejected (then re-auth with `tailscale --socket=$HOME/.cc-ci-ts/tailscaled.sock up
--auth-key="$TS_AUTH_KEY" --hostname=cc-ci-claude-sandbox --accept-routes --accept-dns=false`, and
if that fails the key is a class-A1 blocker).
- **DNS gotcha:** this host's `/etc/resolv.conf` lists only Tailscale resolvers, so direct
`dig @1.1.1.1 …` queries get no answer and look falsely empty. Use `getent hosts <name>` to
resolve from the sandbox. `commoninternet.net` itself is a normal public zone hosted at **Gandi**.
- `ssh cc-ci` works out of the box (`~/.ssh/config` has `Host cc-ci` pinned to `100.90.116.4`,
no ProxyCommand; key `~/.ssh/cc-ci-root-ed25519`; logs in as root).
- For HTTP(S) to cc-ci / `*.ci.commoninternet.net` from the VM, use plain `curl` — no proxy flag
needed. The VM uses public DNS resolvers (`1.1.1.1`/`8.8.8.8`) so `*.ci.commoninternet.net`
resolves normally.
- **If `ssh cc-ci` fails:** run `tailscale status` (as loops or root) to confirm the VM is still
on the tailnet and cc-ci is listed; check `systemctl status tailscaled`. A connectivity failure
is recoverable, not an immediate `## Blocked`-and-stop, unless the VM has lost tailnet membership
entirely (then that IS a class-A1 blocker).
### Credentials the loop GENERATES itself (do not wait on a human for these)