autonomic-bot
b5f2b104e6
The M2 keycloak enrollment made the canonical collision-free at the DOMAIN layer
(warm-canon-keycloak vs warm-keycloak) but warm STATE stayed keyed by bare recipe:
warmsnap.app_dir("keycloak") resolved both the live-warm reconciler's last_good and
the data-warm canonical's canonical.json + snapshot/ into /var/lib/ci-warm/keycloak/.
snapshot() atomically REPLACES that slot, so the two deployments destroyed each
other's known-good; restore() then raised SnapshotError (fails closed, no cross-stack
data write). Worst case: a sweep promote landing inside the reconciler's
snapshot->wait_healthy window makes its rollback restore() raise after
abra.undeploy(live), leaving the shared OIDC provider undeployed.
Fix: canonical.canonical_ns() is now the single namespace from which BOTH the
canonical's domain and its warm-state slot derive, so they cannot drift apart. A
live-warm provider gets ns "canon-<recipe>": domain warm-canon-keycloak (unchanged)
and slot /var/lib/ci-warm/canon-keycloak/. Every other recipe keeps ns "<recipe>" —
zero on-disk change for the 15 existing canonicals, and no migration on cc-ci
(keycloak's canonical was never seeded: its dir holds only last_good).
- warmsnap: functions take a SLOT, not a recipe; add live_slot(); meta records "slot".
- warmsnap: _assert_slot_not_foreign() refuses to snapshot/restore a slot recorded
against a different domain -- defence in depth, naming-scheme-independent, fails
before the destructive swap rather than at the next restore.
- canonical: registry_path/seed_canonical/prune_stale go through canonical_slot().
- prune_stale: the "reconciler dirs are never pruned" invariant is now STRUCTURAL --
<recipe>/ never gains a canonical.json, so de-enrolling keycloak can no longer
rmtree the reconciler's last_good (consequence 4).
- warm_reconcile: last_good + snapshot/restore go through warmsnap.live_slot().
- run_recipe_ci: canonical rollback restores from canonical_slot(recipe).
- Correct the two comments that claimed the deployments "can never touch each other".
Tests: 10 new (slot disjointness for every WARM_DOMAINS recipe, slot<->stack 1:1,
registry not in the reconciler dir, prune spares last_good, foreign-slot refusal in
both snapshot and restore). Unit suite 315 -> 325, no regressions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX
cc-ci — Co-op Cloud recipe CI server
Comment !testme on a PR in an enrolled Co-op Cloud recipe repo and cc-ci deploys the recipe
at that commit onto a real single-node Docker Swarm, runs install / upgrade / backup-restore tests
(Python + Playwright) end-to-end, and reports a live, tail-able run with pass/fail back to the PR.
This repo declares the entire server as a NixOS flake and holds the test harness, the per-recipe test trees, and the docs to enroll a recipe or rebuild the box from scratch.
Status: under active autonomous construction. See
machine-docs/STATUS.mdfor the live phase andplan.md-driven milestones inmachine-docs/BACKLOG.md. Definition of Done is D1–D10 (see the build plan).
Layout
flake.nix NixOS entry point + devshells (`#cc-ci` = live Hetzner host, `#cc-ci-incus` = legacy Incus host)
nix/hosts/cc-ci/ legacy Incus VM host config (fallback / historical)
nix/hosts/cc-ci-hetzner/ live Hetzner host config
nix/modules/ drone, comment-bridge, swarm, dashboard, secrets (Nix modules)
secrets/ sops-encrypted infra secrets (cc-ci-secrets submodule)
bridge/ !testme webhook listener source
runner/ run_recipe_ci.py + shared pytest harness
dashboard/ results overview generator
tests/<recipe>/ per-recipe install/upgrade/backup tests + custom/
docs/ install, enroll-recipe, secrets, architecture, runbook, baseline
All .nix code lives under nix/; flake.nix/flake.lock stay at the repo root. Host targets are:
#cc-ci= canonical live Hetzner server#cc-ci-hetzner= explicit alias for the same live Hetzner server#cc-ci-incus= legacy Incus VM definition only; do not use on Hetzner
Docs
docs/install.md— rebuild the server from scratch (D8)docs/testing.md— test architecture: generic lifecycle suite + layered recipe overlays (override/extend, discovery precedence, custom install-steps hook)docs/enroll-recipe.md— add a recipe under CI (D5)docs/secrets.md— secret model + rotation (D6)docs/architecture.md,docs/runbook.md— design + debugging failed runsdocs/baseline.md— bootstrap snapshot / rollback reference
Linting & formatting
The codebase is kept formatted + lint-clean by a single entrypoint, run from the pinned lint
devshell so local and CI use identical tool versions:
nix develop .#lint --command bash scripts/lint.sh # check-only (what CI runs)
nix develop .#lint --command bash scripts/lint.sh --fix # auto-format + apply fixes
Covers Nix (nixpkgs-fmt · statix · deadnix), Python (ruff lint+format), Shell
(shellcheck · shfmt), and YAML (yamllint). Config lives in ruff.toml / .yamllint.yaml;
tool/strictness choices are in machine-docs/DECISIONS.md. CI enforces it: the lint step in the
.drone.yml push pipeline runs the same command and fails the build on any unclean file, so
keep commits clean (--fix before pushing).
Loop state (autonomous build)
The multi-agent loop state lives under machine-docs/: STATUS.md (phase/blockers),
BACKLOG.md (work + adversary findings), REVIEW.md (independent verification), JOURNAL.md
(build log), DECISIONS.md (architecture choices) — plus the phase-namespaced *-1b.md / *-1c.md
variants. See the build plan for the two-loop Builder/Adversary protocol.