Planning + launch + setup material for the cc-ci Co-op Cloud recipe CI server:
plan.md (single source of truth), kickoff/launch supervision, and the
Builder/Adversary loop prompts. Secrets (.testenv) and runtime dirs are gitignored.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Set the org in `plan.md` (`git.autonomic.zone/recipe-maintainers/cc-ci`) and lock the six proof recipes (§8).
- Ensure the launching shell has: SSH+sudo to `cc-ci`, the Gitea token, `git.autonomic.zone` access.
- Preconfigure test-app DNS + TLS (plan §4.0): point a wildcard `*.ci.commoninternet.net` record at a gateway that TLS-passthroughs to cc-ci, and **pre-issue the wildcard cert** (`*.ci.commoninternet.net` + `ci.commoninternet.net`, via Gandi DNS-01) into `/var/lib/ci-certs/live/` on cc-ci. The agent handles everything else on cc-ci (Traefik file provider → that cert, swarm, routing) and does **no ACME**; renewal (~90 days) is an out-of-band operator task, so the DNS token never goes to the agent.
-`export CC_CI_REPO=https://git.autonomic.zone/recipe-maintainers/cc-ci.git` so the watchdog can detect `## DONE`.
## What "done" means
The loops stop only when all of `plan.md` §2 (D1–D10) hold **and** the Adversary has independently
re-verified each within 24h. The watchdog then tears the loops down automatically.
I want you to work in an autonomous loop over the next few days until the CI server is fully functional, polished and documented
on any PR on git.autonomic.zone it should be invokable by writing !testme as a comment
this should invoke the set of CI tests to be run for the recipe code at that PR
the CI tests should be run via drone
the tests run for a recipe should be written in python. e2e testing via playwright should be used whe necessary to confirm functionality
there should be tests which test
- new install
- upgrade
- backups (including restore)
all the tests should be fully e2e, with a real deployed recipe
the CI runner should be deployed on a server called cc-ci which is running nixos
cc-ci git repo should also live on git.autonomic.zone which contains all the nix configuration for the server, as well as the code for the CI test runner
the CI test runner should have its own folder of tests, with one folder for each recipe, with each of those folders containg a set of tests as python files which get invoked for that recipe
secrets should also be handled in a reasonable and repeatable way
additionally, if a recipe repo itself contains a tests folder in the recipe, the CI runner should also invoke those tests as part of the CI run for those tests
the results of the test run should be easily viewable, with trackable logs, and a final result, very similar in style to the way the yunohost CI runner looks and feels
you will have ssh access to cc-ci server, as well as sudo access there
you will also have access to create and modify repos on git.autonomic.zone
WATCH_DIR="${WATCH_DIR:-/srv/cc-ci/.cc-ci-watch}"# tiny clone the watchdog reads STATUS.md from
LOG_DIR="${LOG_DIR:-/srv/cc-ci/.cc-ci-logs}"
CC_CI_REPO="${CC_CI_REPO:-https://git.autonomic.zone/recipe-maintainers/cc-ci.git}"# CI project repo (DONE detection); harmless until the Builder creates it
CC_CI_BRANCH="${CC_CI_BRANCH:-main}"
WATCH_INTERVAL="${WATCH_INTERVAL:-300}"# seconds between watchdog checks
**Code/config home:**`git.autonomic.zone/recipe-maintainers/cc-ci` (the CI project repo — distinct from this
`/srv/cc-ci/cc-ci-plan/` planning+launch folder)
**Last updated:** keep current via `STATUS.md` (see §7)
---
## 0. How to read this document
This plan is written to be handed to an **autonomous Claude agent running in a sandbox over
several days**, driving itself in a loop until the CI server is "done" per §2. A second agent
(the **Adversary**) independently tries to disprove every "done" claim. Neither agent is
trusted to mark its own work complete.
If you are an agent waking up into this loop for the first time, go straight to **§1 Bootstrap**.
On every subsequent wake, go to **§7 The Loop Protocol** and continue from `STATUS.md`.
The rest of the document (§3–§6) is the technical design. Treat it as the default architecture,
but you are allowed to revise it when reality disagrees — record any deviation in `DECISIONS.md`
with a one-line rationale.
---
## 1. Bootstrap (first wake only)
Do these in order. Each step is idempotent; re-running is safe.
1.**Verify access.** (Full credential map + how each is used is in **§1.5** — read it first.)
-`ssh cc-ci 'hostname && whoami'` — you log in as **root** on cc-ci (NixOS), so there is no
separate sudo step. `ssh cc-ci` is preconfigured to tunnel through the userspace-tailscaled
SOCKS proxy (§1.5); if it fails, the proxy/daemon is probably down — restart it (§1.5) before
declaring blocked.
-`ssh cc-ci 'nixos-version'` — confirm NixOS.
- Confirm you can reach the Gitea API with the bot creds from `.testenv` (§1.5):
`curl -s https://$GITEA_URL/api/v1/version`. The bot authenticates with
`GITEA_USERNAME`/`GITEA_PASSWORD` (basic auth) or a token you mint from them via
`POST /api/v1/users/<user>/tokens` — do **not** expect a ready-made `$GITEA_TOKEN`.
- Confirm the **preconfigured** test-app DNS (§4.0/§4.4): a random subdomain under the wildcard
resolves, e.g. `getent hosts probe-$RANDOM.ci.commoninternet.net` returns the **gateway's** IP
(not cc-ci's — the gateway TLS-passthroughs to cc-ci, so do not expect cc-ci's address; and use
`getent`, not `dig`, since this host's resolver is Tailscale-only — see §1.5).
Traefik is *not* up yet — you configure it (file provider → the pre-issued cert at
`/var/lib/ci-certs/live/`, **no ACME**); the DNS record + gateway passthrough + cert are the
preconditions, and full end-to-end HTTPS reachability is proven at M1, not now.
If the wildcard does not resolve at all, that's a `## Blocked` item (operator fixes DNS/gateway).
- If any check fails, write the failure to `STATUS.md` under `## Blocked` and stop — a human must fix access. Do **not** try to work around missing access.
2.**Create the `cc-ci` repo** on git.autonomic.zone if it does not exist. Push an initial
skeleton (see §3 layout). The Builder clones to `/srv/cc-ci/cc-ci`; the Adversary loop keeps
its **own independent clone** at `/srv/cc-ci/cc-ci-adv`. The repo is the only channel between
the two loops (§6.1) — loop state lives inside it (`STATUS.md`, `BACKLOG.md`, etc.).
3.**Snapshot the starting environment** into `cc-ci/docs/baseline.md`: current NixOS config on
the server (`/etc/nixos` or existing flake), installed packages, whether Docker/Swarm/abra
already exist, DNS that already points at the box. This is the rollback reference.
4.**Seed the loop state files** (§7) if absent: `STATUS.md`, `BACKLOG.md`, `REVIEW.md`,
`JOURNAL.md`, `DECISIONS.md`. Give `BACKLOG.md` two H2 sections — `## Build backlog`
(populated from §5 milestones) and `## Adversary findings` (empty) — per the single-writer
rule in §6.1.
5. Commit ("chore: bootstrap cc-ci loop state") and begin the loop at §7.
---
## 1.5 Credentials & access — where everything lives and how to use it
The loops run **on the sandbox host** (not on cc-ci) and reach cc-ci over Tailscale. This section
is the authoritative map of what credentials exist, where, and how to use them. **Never copy any
secret value into the repo, a commit, a log, or the dashboard** (§9) — reference locations only.
### Provided credentials (already in place)
| What | Where | How to use |
|---|---|---|
| **Tailscale auth key** (joins cc-ci's tailnet `taila4a0bf.ts.net`) | `/srv/cc-ci/.testenv` → `TS_AUTH_KEY` (Tailscale SaaS key, keyID ends `CNTRL`) | Used to bring up the userspace tailscaled (below). It's reusable; re-run `tailscale up` with it if the node drops. |
| **cc-ci SSH (root)** | private key `~/.ssh/cc-ci-root-ed25519`; config `Host cc-ci` in `~/.ssh/config` | Just run `ssh cc-ci` (logs in as **root**). The pubkey is already in cc-ci's `/root/.ssh/authorized_keys`. |
| **Gitea bot account** | `/srv/cc-ci/.testenv` → `GITEA_USERNAME` (`autonomic-bot`), `GITEA_PASSWORD`, `GITEA_URL` (`git.autonomic.zone`) | Basic-auth to the Gitea API, or mint a scoped token: `POST https://$GITEA_URL/api/v1/users/$GITEA_USERNAME/tokens`. Used to create/push the `cc-ci` repo, read recipe repos, comment on PRs, and register `!testme` webhooks. |
Load them in a shell with: `set -a; . /srv/cc-ci/.testenv; set +a` (don't echo the values).
### The Tailscale connection (how `ssh cc-ci` and the proxy work)
cc-ci (`cc-nix-test`, **100.90.116.4**) is on a *different* tailnet than the sandbox host's default
one, so it is reached via a **second, userspace tailscaled** — this keeps the host's own tailnet
untouched. State lives in `~/.cc-ci-ts/`; it exposes a **SOCKS5/HTTP proxy on `127.0.0.1:1055`**,
which is the only route to that tailnet (userspace networking ⇒ the host OS can't route the tailnet
IPs directly).
It runs as a **persistent systemd service** (`cc-ci-tailscaled.service`, enabled, `Restart=always`,
starts on boot; unit at `/etc/systemd/system/cc-ci-tailscaled.service`, runs as user `notplants`).
It reuses the already-authenticated state in `~/.cc-ci-ts/`, so it reconnects across reboots/crashes
without the auth key.
-`ssh cc-ci` works out of the box (its `ProxyCommand` uses the proxy; logs in as root).
- For HTTP(S) to cc-ci / `*.ci.commoninternet.net` from the sandbox, go through the proxy, e.g.
You are the Adversary agent for cc-ci — one of two independent loops. Your job is to DISBELIEVE the Builder. Read /srv/cc-ci/cc-ci-plan/plan.md in full, especially §2, §6, §6.1, and §9.
Start a self-paced loop now: invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. Pace yourself: poll short (~4m) while watching a CLAIMED gate or a running build; sleep 20–30m when idle. Keep running independent break-it probes even when no gate is pending. Stop only when STATUS.md says ## DONE and you have logged a fresh PASS for every D1–D10.
Credentials/access: §1.5 is the authoritative map. Provided creds are in /srv/cc-ci/.testenv and ~/.ssh; reach cc-ci with `ssh cc-ci` (root, via the userspace-tailscaled SOCKS proxy on 127.0.0.1:1055), and hit the dashboard / *.ci.commoninternet.net through that proxy (`curl --proxy socks5h://localhost:1055 ...`). If the proxy is down, restart it per §1.5. Verify from a COLD START but you may rely on this shared access path.
You run as a SEPARATE process and coordinate ONLY through the git repo per §6.1:
- Keep your OWN clone at /srv/cc-ci/cc-ci-adv. If the repo doesn't exist yet, wait and retry on your next wake — the Builder creates it during §1 Bootstrap.
- git pull --rebase before every edit; commit; push; never --force.
- Write ONLY your files: REVIEW.md and the "## Adversary findings" section of BACKLOG.md. Everything else (code, STATUS.md, JOURNAL.md, "## Build backlog") is read-only to you.
Each wake:
1. Pull. Read STATUS.md for any "Gate: <Mn> CLAIMED, awaiting Adversary".
2. Verify claims from a COLD START (fresh shell, your own clone, no cached state). Re-run the milestone/D-gate acceptance check yourself; do not trust the Builder's word.
3. Actively try to break things: !testmexyz must NOT trigger; non-collaborator comments rejected; a failing PR must report RED; killing an app mid-run still leaves clean teardown; published logs AND the dashboard contain no secrets (incl. generated app passwords); two concurrent !testme runs don't collide on domain/volume/secrets; the SAME generated app secrets persist across install → upgrade → backup/restore.
4. Record verdicts in REVIEW.md ("<Mn>: PASS @<ts>" + evidence, or FAIL). File each defect as a "## Adversary findings" item tagged [adversary] with repro steps. Only YOU close those, after re-test. You hold veto power: write "## VETO <reason>" to REVIEW.md to forbid DONE until cleared.
5. Push. Schedule the next wake.
Begin: read /srv/cc-ci/cc-ci-plan/plan.md, then enter the self-paced loop (start by cloning the repo to /srv/cc-ci/cc-ci-adv if it exists yet).
You are the Builder agent for the cc-ci project — one of two independent loops. Your job is to build a Co-op Cloud recipe CI server, working autonomously over multiple days.
Single source of truth: /srv/cc-ci/cc-ci-plan/plan.md. Read it in full now, then begin at §1 Bootstrap. The original brief /srv/cc-ci/cc-ci-plan/brief.md is context only — do not edit it.
Start a self-paced loop now: invoke `/loop` with no interval so you re-wake yourself via ScheduleWakeup. Each iteration = one unit of work (see §7). Pace per §7: poll ~4m while a build/deploy/rebuild is in flight to stay cache-warm; sleep 20–30m when genuinely idle or parked at a gate. Do NOT spin on a build that takes minutes. Stop the loop only when STATUS.md says ## DONE.
You run as a SEPARATE process from the Adversary loop and coordinate ONLY through the git repo per §6.1:
- git pull --rebase before every edit; make the smallest change; commit; git push. Never --force.
- Write ONLY your files: source/config, STATUS.md, JOURNAL.md, DECISIONS.md, and the "## Build backlog" section of BACKLOG.md. Treat REVIEW.md and "## Adversary findings" as read-only — the Adversary owns them.
- At each milestone gate, set "Gate: <Mn> CLAIMED, awaiting Adversary" in STATUS.md and work other unblocked items; do NOT advance past the gate until REVIEW.md shows its PASS.
- Write "## DONE" only when REVIEW.md shows a PASS dated <24h for every D1–D10 and there is no standing "## VETO".
Overriding rules:
- "Done" is defined ONLY by §2 (D1–D10), Adversary-verified. No self-certifying.
- Verify every change against the real server/Drone/Gitea; paste command + output into JOURNAL.md. No "should work."
- Never weaken, skip, or delete a test to make a run pass. A red test is information.
- Only cc-ci is yours to reconfigure. Never push code to recipe repos; never touch production servers/domains. Keep server state Nix-declared and reversible.
- 3rd identical failure → stop, record dead-end in DECISIONS.md, change approach or mark blocked.
- Credentials: §1.5 is the authoritative map. Provided creds are in /srv/cc-ci/.testenv (TS_AUTH_KEY, GITEA_USERNAME/PASSWORD/URL) and ~/.ssh (cc-ci-root-ed25519). Reach cc-ci with `ssh cc-ci` (root, via the userspace-tailscaled SOCKS proxy on 127.0.0.1:1055); if it fails, restart the proxy per §1.5 before declaring blocked. There is NO ready-made $GITEA_TOKEN — mint one from the bot creds if you want a token.
- Secret classes (§4.4), handled differently:
• Class A1 EXTERNAL infra inputs (cc-ci SSH/root access, TS auth key, Gitea bot creds, the pre-issued wildcard TLS cert at /var/lib/ci-certs/live/, registry creds; plus the preconfigured DNS/gateway facts): if missing/invalid → STATUS.md ## Blocked and stop. Do NOT improvise/invent. NEVER attempt ACME/DNS-01 for commoninternet.net — the cert is pre-provided and renewed out-of-band; point Traefik's file provider at /var/lib/ci-certs/live/{fullchain.pem,privkey.pem}.
• Class A2 INTERNAL infra secrets (Drone RPC, webhook HMAC, Gitea OAuth app, host age key): you GENERATE these yourself — never block on them.
• Class B RECIPE APP secrets: NOT a blocker. The harness generates them (abra app secret generate + chosen fixtures), persists them per-run so the SAME values survive install → upgrade → backup/restore, and destroys them at teardown.
Begin: read /srv/cc-ci/cc-ci-plan/plan.md, then execute §1 Bootstrap, then enter the self-paced loop.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Claude Opus 4.7