autonomic-bot
fc68e5ef6e
The canonical is last-GREEN, not CURRENT. Its promotion (WC5) deploys a warm-<recipe> app and health-checks it; when that fails the canonical stays pinned at an old release indefinitely — and the failure is invisible to the upgrade tier, which keeps basing on it. Live evidence (2026-08-10): gitea's canonical was 3.5.3+1.24.2-rootless from 2026-06-17 because warm-gitea crash-loops on a read-only app.ini (open fix: recipe PR #4). Four other recipes are in the same state — the 2026-08-09 sweep logged GREEN-BUT-PROMOTE-FAILED for bluesky-pds, drone, gitea, lasuite-drive and lasuite-meet. Consequence: the upgrade tier tested 3.5.3 -> head, a transition no deployment performs, and MISSED the one they do. 3.5.3 -> head crosses an APP_INI_VERSION change (v21 -> v22) so Swarm creates a fresh config object; the real 3.6.1 -> 3.6.2 upgrade keeps v22 and aborts on Swarm's immutable-config rule ('only updates to Labels are allowed'). That shipped in 3.6.2 and broke every existing install taking the 1.27.1 security release. Now: when the newest published release older than head is NEWER than the canonical, use the release and say so in the log (naming the stale canonical so the promote failure gets noticed). Resolution stays dynamic; UPGRADE_BASE_FLOOR still filters structurally-invalid candidates; the canonical remains the base whenever it is current. Unit-verified on the real gitea tag set: base moves 3.5.3+1.24.2-rootless -> 3.6.1+1.26.2-rootless, exactly the transition that reproduces the break.
cc-ci — Co-op Cloud recipe CI server
Comment !testme on a PR in an enrolled Co-op Cloud recipe repo and cc-ci deploys the recipe
at that commit onto a real single-node Docker Swarm, runs install / upgrade / backup-restore tests
(Python + Playwright) end-to-end, and reports a live, tail-able run with pass/fail back to the PR.
This repo declares the entire server as a NixOS flake and holds the test harness, the per-recipe test trees, and the docs to enroll a recipe or rebuild the box from scratch.
Status: under active autonomous construction. See
machine-docs/STATUS.mdfor the live phase andplan.md-driven milestones inmachine-docs/BACKLOG.md. Definition of Done is D1–D10 (see the build plan).
Layout
flake.nix NixOS entry point + devshells (`#cc-ci` = live Hetzner host, `#cc-ci-incus` = legacy Incus host)
nix/hosts/cc-ci/ legacy Incus VM host config (fallback / historical)
nix/hosts/cc-ci-hetzner/ live Hetzner host config
nix/modules/ drone, comment-bridge, swarm, dashboard, secrets (Nix modules)
secrets/ sops-encrypted infra secrets (cc-ci-secrets submodule)
bridge/ !testme webhook listener source
runner/ run_recipe_ci.py + shared pytest harness
dashboard/ results overview generator
tests/<recipe>/ per-recipe install/upgrade/backup tests + custom/
docs/ install, enroll-recipe, secrets, architecture, runbook, baseline
All .nix code lives under nix/; flake.nix/flake.lock stay at the repo root. Host targets are:
#cc-ci= canonical live Hetzner server#cc-ci-hetzner= explicit alias for the same live Hetzner server#cc-ci-incus= legacy Incus VM definition only; do not use on Hetzner
Docs
docs/install.md— rebuild the server from scratch (D8)docs/testing.md— test architecture: generic lifecycle suite + layered recipe overlays (override/extend, discovery precedence, custom install-steps hook)docs/enroll-recipe.md— add a recipe under CI (D5)docs/secrets.md— secret model + rotation (D6)docs/architecture.md,docs/runbook.md— design + debugging failed runsdocs/baseline.md— bootstrap snapshot / rollback reference
Linting & formatting
The codebase is kept formatted + lint-clean by a single entrypoint, run from the pinned lint
devshell so local and CI use identical tool versions:
nix develop .#lint --command bash scripts/lint.sh # check-only (what CI runs)
nix develop .#lint --command bash scripts/lint.sh --fix # auto-format + apply fixes
Covers Nix (nixpkgs-fmt · statix · deadnix), Python (ruff lint+format), Shell
(shellcheck · shfmt), and YAML (yamllint). Config lives in ruff.toml / .yamllint.yaml;
tool/strictness choices are in machine-docs/DECISIONS.md. CI enforces it: the lint step in the
.drone.yml push pipeline runs the same command and fails the build on any unclean file, so
keep commits clean (--fix before pushing).
Loop state (autonomous build)
The multi-agent loop state lives under machine-docs/: STATUS.md (phase/blockers),
BACKLOG.md (work + adversary findings), REVIEW.md (independent verification), JOURNAL.md
(build log), DECISIONS.md (architecture choices) — plus the phase-namespaced *-1b.md / *-1c.md
variants. See the build plan for the two-loop Builder/Adversary protocol.