cc-ci/STATUS.md

STATUS — cc-ci Builder

Phase: M0/M1/M2/M4/M5 PASS; M3 PASS (Adversary-verified); M6 CLAIMED (awaiting Adversary). Bridge→Drone→harness integration DONE (recipe-ci pipeline). M6.5 underway: keycloak full 3-stage GREEN through Drone (build #39). Next: enroll recipes 3–6 (remaining D10 categories), M7, M8. In-flight: M6.5 gate CLAIMED — all 6 D10 recipes full 3-stage green (host + canonical Drone): custom-html, keycloak(#39), cryptpad(#46), matrix-synapse(#51), lasuite-docs(#57), n8n(#63 in flight). bluesky-pds (TLS-passthrough) swapped → n8n per DECISIONS (caddy self-ACME vs no-ACME design). M6.5 PASS + M7/D6 PASS (Adversary). M8/D7 CLAIMED — dashboard overview+badges LIVE + PR-comment outcome reflection (bridge edits comment to ✅/❌; verified). Remaining for DONE: M9 docs/reproducibility (D8 from-scratch rebuild + D9 docs) and M10/D10 — the six recipes green via real !testme PRs (currently proven via API-trigger; the Adversary-flagged gap). M10 = enroll recipes in the bridge POLL_REPOS + open recipe-mirror PRs + !testme each. Last updated: 2026-05-27 (M6.5 CLAIMED — 6/6 recipes 3-stage green across all D10 categories)

Near-complete (2026-05-27)

Feature-complete except the 6th D10 recipe. Verified/claimed: M0–M6 PASS, M6.5 PASS, M7/D6 PASS, M8/D7 CLAIMED. M9/D9 docs complete (architecture+runbook added). M10: 5/6 recipes green via real !testme (custom-html/keycloak/matrix-synapse/n8n/cryptpad). DONE is gated on: (1) operator Docker Hub registry creds → lasuite-docs 6th green (A1 blocker, notified; retries halted); (2) Adversary verification of M8/M9 + D8 from-scratch rebuild + the D10 runs. No unblocked Builder implementation remains — awaiting operator creds + Adversary. On each wake: check .testenv/sops for creds + rate-limit reset → if available, wire creds (or quota-retry) + run lasuite; else idle.

Gate: M6.5 — CLAIMED, awaiting Adversary (2026-05-27)

All 6 D10 recipes have a full install/upgrade/backup green run, each verified on host AND via the canonical Drone recipe-ci pipeline (build #s above), each with clean teardown (0 orphans). Categories: custom-html=simple, keycloak=SSO/identity+DB, cryptpad=stateful/no-DB, matrix-synapse=DB+media/ large-volume, lasuite-docs=multi-service+S3/MinIO/object-storage, n8n=workflow automation. D5 held: each recipe enrolled via tests/<recipe>/ + recipe_meta.py (EXTRA_ENV for cryptpad SANDBOX_DOMAIN / lasuite TIMEOUT) only — no shared runner/harness changes per recipe. Repro: trigger a custom Drone build with RECIPE= (or cc-ci-run runner/run_recipe_ci.py with RECIPE/STAGES on host).

Gates

• Gate: M0 — CLAIMED, awaiting Adversary (2026-05-26). Evidence: flake rebuilds cc-ci from repo (switch --flake /root/cc-ci#cc-ci, gen healthy, no failed units); sops-nix decrypts /run/secrets/test_secret (0400 root, value = generated cc-ci-m0-…). Repro: clone repo, sync to host, nixos-rebuild switch --flake .#cc-ci, then systemctl is-system-running + check the secret. Per §6.1 I will NOT advance past this gate to M2; M1 work proceeds as independent unblocked work. → M0 PASS logged by Adversary in REVIEW.md @2026-05-26T21:35Z (cold verify, leak probe clean).
• Gate: M1 — CLAIMED, awaiting Adversary (2026-05-26). Evidence: Docker single-node swarm + proxy overlay; real coop-cloud/traefik via abra (wildcard/file-provider, no ACME); custom-html deployed by hand → HTTP 200 over HTTPS via gateway at cchtml1.ci.commoninternet.net with the wildcard cert; torn down clean (services/volumes/secrets/containers all 0). Repro: scripts/deploy-proxy.sh + abra app new/deploy/undeploy. Starting M2 as independent work; will not flip M2's gate until M1 shows PASS. → M1 PASS @2026-05-26T22:20Z.
• Gate: M2 — CLAIMED, awaiting Adversary (2026-05-26). Evidence: Drone server (coop-cloud recipe, reconcile oneshot, Gitea SSO) healthz 200 via gateway; exec runner polling (capacity=2). cc-ci repo activated (push webhook). Pushing .drone.yml triggered build #1 → success (clone + hello exec steps, exit 0; ran abra/docker on the host). Repro: nixos-rebuild switch + one-time scripts/bootstrap-drone-oauth.sh. Starting M3 as independent work; won't flip M3 gate until M2 PASS.
• Gate: M3 — CLAIMED, awaiting Adversary (2026-05-27). Trigger redesigned per orchestrator (plan §4.1): polling is PRIMARY (outbound, read-only, ≤30s), webhook optional/admin-registered; commenter auth via org membership (GET /orgs/{owner}/members/{user} 204, read-level) + optional allowlist — NOT the admin-requiring /collaborators/{user}/permission. Evidence: posted !testme on PR #1 (by bot, an org member) → poller fired in 6s → Drone build #26 for head d397720a → bridge posted the run-link comment back. Auth endpoint verified read-level: bot/trav/ notplants → 204, non-member → 404. The old webhook-delivery blocker is moot (polling doesn't need the Gitea ALLOWED_HOST_LIST whitelist). Won't advance past this gate until REVIEW shows PASS; doing the bridge→Drone integration as independent work meanwhile.

Resource safety (plan §4.2/§4.3 — orchestrator change 2026-05-27)

• MAX_TESTS = DRONE_RUNNER_CAPACITY = 1 (modules/drone-runner.nix): ≤1 build at once, Drone auto-queues the rest natively. Verified DRONE_RUNNER_CAPACITY=1 on the runner.
• Per-build timeout = 60m (modules/drone.nix, reconciled best-effort, non-fatal): a hung build is cancelled → frees its slot. Verified Drone repo timeout: 60.
• Janitor backstop for SIGKILL'd builds (reaps orphaned run apps at run-start). At capacity=1 the recipe-CI pipeline will set CCCI_JANITOR_MAX_AGE=0 (safe — no concurrent runs). See DECISIONS.

Blocked

• Docker Hub anonymous pull rate limit — registry pull creds needed (A1, operator). During the D10 real-!testme breadth runs, lasuite-docs (heaviest: 9 images) hit toomanyrequests: unauthenticated pull rate limit on its upgrade stage (redis:8.2.6 task Rejected "No such image" → couldn't pull). Confirmed: docker pull redis:8.2.6 on the node → rate-limited. This is the plan's flagged A1 input (§1.5/§4.4: "registry pull creds … rate-limit failure traced to this is a finding, then request creds"). Operator action: provide Docker Hub pull creds (store sops-encrypted in secrets/, wire into the docker daemon / swarm). NOT globally blocking: 5/6 recipes already green via real !testme (custom-html/keycloak/matrix-synapse/ n8n/cryptpad); lasuite-docs install+backup green too — only its upgrade (most pulls) is gated. Contributing factor: my mid-breadth docker image prune -af evicted cached images → forced re-pulls → tipped the limit (see DECISIONS). The anonymous limit resets in ~hours, so a retry may also pass without creds, but creds are the durable fix. Working M9 (docs) meanwhile.
• (M3 webhook blocker previously here — cleared by the polling-primary redesign; polling is read-only/outbound and needs no Gitea ALLOWED_HOST_LIST whitelist.)

Tracking (adversary findings I must address)

• [adversary] A4 — concurrent same-recipe runs collide on shared ~/.abra/recipes/<recipe>. Root cause the finding names ("no Drone concurrency cap — runner capacity=2") is now eliminated: MAX_TESTS = DRONE_RUNNER_CAPACITY = 1 (resource-safety change). With ≤1 build at a time there is no concurrent run on this single node, so the shared-recipe-dir race cannot occur. Builder side addressed via the concurrency cap (per plan §4.2 "concurrency cap 1–2"); Adversary to re-test/close. (Per-run ABRA_DIR/HOME isolation would be belt-and-suspenders but is unnecessary at capacity=1.)
• [adversary] A2 — janitor -pr filter dead. Already fixed in code: lifecycle.RUN_APP_RE = ^[a-z0-9]{1,4}-[0-9a-f]{6}\.ci\.commoninternet\.net$ (the hashed scheme), plus a stack-name regex for .env-less orphans, gated on age. Awaiting Adversary kill-probe re-test.
• [adversary] A3 — teardown unverified; .env removed before confirmed undeploy. Already fixed: lifecycle.teardown_app undeploys → docker stack rm fallback if services remain → removes volumes/secrets while .env exists → drops .env LAST → then _residual() check raises TeardownError if anything is left. Awaiting Adversary kill-mid-run re-test.
• [adversary] A1 — no-ACME hazard for test apps. Acknowledged (valid). The harness (M4) MUST force LETS_ENCRYPT_ENV="" on every test-app deploy (already done in scripts/deploy-proxy.sh and the M1 manual custom-html deploy; scripts/deploy-drone.sh will too). Considering a structural belt-and-suspenders (drop the unused certificatesResolvers from cc-ci's traefik) — deferred, needs a recipe-config override. Will make the harness enforcement the primary fix; Adversary re-tests + closes after M4. → Now enforced: harness.lifecycle.deploy_app sets LETS_ENCRYPT_ENV="" on every test-app deploy (verified in the M4 custom-html run). Adversary can re-test + close A1.

Notes

• Disk RESOLVED: operator grew the VM 8.9→28 GiB (22 GiB free) on 2026-05-26. Inodes 1.78M total / 1.21M free (was ~6k free — old 8.9 GiB fs had only 586k inodes, which the flake's nixpkgs fetch exhausted). Both byte + inode pressure gone.
• M0 base config: flake at repo root pins nixpkgs to the exact rev cc-ci ran (50ab793) → first rebuild is no-op-then-base. Deployed via nixos-rebuild switch --flake /root/cc-ci#cc-ci run as a detached transient systemd unit (survives ssh-over-tailscale drops). Gen 3 current, healthy.
• Open warning: incus module enables systemd.network while we set networking.useDHCP=true (scripted dhcpcd) — Nix warns both may manage interfaces. Inherited from baseline, networking is up; clean up later (pick networkd OR scripting). Tracked, non-blocking.