cc-ci-orchestrator/cc-ci-plan/plan-phase-canon-canonical-sweep.md

Phase canon — make the canonical sweep actually work (the real "nightly sweep") + verify it

Mission (operator-specified 2026-06-17): the "nightly sweep" was specified in theory but was never actually doing anything — confirmed live: nightly-sweep.timer is deployed and fires green (nightly_sweep.py, last run 2026-06-17 03:09 UTC exit 0), but only custom-html is WARM_CANONICAL -enrolled and ZERO canonical.json records exist — i.e. the machinery has never actually promoted a canonical end-to-end. This phase makes it real and proven, as the substitute for that hollow nightly sweep, with two additions the operator wants:

1. Sync each recipe mirror's main on git.autonomic.zone/recipe-maintainers/<recipe> to its upstream (git.coopcloud.tech/coop-cloud/<recipe>) first, so the sweep tests true upstream latest.
2. Skip a recipe whose main is unchanged vs its current canonical (no rerun needed).

…then run CI cold-on-main for each recipe and actually promote the canonical for any that pass — and prove the whole thing works. The deliverable is correctness, verified end-to-end — and the operator specifically wants confidence it plays nicely with the samever upgrade-base work (§2 "Plays-nice-with-samever"). Operator decisions (2026-06-17): all recipes enrolled (§2.B), and the cadence is weekly (change the existing daily timer to weekly — a one-line OnCalendar tune; exact day/time is not critical). This REPLACES the hollow nightly sweep; it is not a parallel job.

State files: STATUS-canon.md, BACKLOG-canon.md, REVIEW-canon.md, JOURNAL-canon.md. DECISIONS.md shared.

1. Verified starting state (2026-06-17)

• nightly-sweep.timer enabled + active (next ~03:00 UTC); nightly_sweep.py runs and exits 0. The timer/service plumbing already works — reuse it, don't rebuild it.
• Only custom-html sets WARM_CANONICAL = True. The sweep iterates canonical.enrolled_recipes() → essentially one recipe → near-no-op across the fleet.
• No canonical.json exists on the host → the promote path (should_promote_canonical → promote_canonical → write_registry) has never successfully produced a canonical, even for custom-html. This is the crux of "theory, not actually doing it."
• The sweep does not reconcile mirrors to upstream, and does not skip-when-unchanged.

2. The work

A. Prove + fix the promote path FIRST (the core). On custom-html (already enrolled), make a green cold-on-latest run actually write canonical.json (recipe/version/commit/status) AND prove a subsequent --quick warm-reattach uses it (deploy_canonical reattaches the retained volume). If it doesn't happen today, find and fix why (this is the real defect behind the hollow sweep). A canonical must demonstrably exist and be reusable before anything else is meaningful.

B. Enroll ALL recipes (operator decision 2026-06-17). Set WARM_CANONICAL = True for every recipe cc-ci tracks (the used-recipes.md set) — the sweep promotes a canonical for each that passes, not just custom-html.

• Watch the warm-volume disk budget: ~21 recipes each retaining a data volume on the single node is real disk. Verify headroom, lean on the existing WC8 disk-hygiene / ci-docker-prune, and if disk becomes the binding limit, raise it rather than silently dropping recipes (a fallback if needed: decouple the cheap last-green version record — kept for all — from the expensive retained volume). Default remains all-enrolled.
• If a specific recipe genuinely cannot be enrolled (e.g. unbounded data, no stable health), record the exception + reason in DECISIONS — don't silently skip it.

C. Add the upstream mirror-sync step. Before the per-recipe CI, reconcile each mirror's main + tags to coopcloud upstream — reuse recipe-upgrade's open-recipe-pr.sh <recipe> --reconcile-only (handles go-git private-mirror auth, fetches coopcloud via an upstream remote, closes already-merged-upstream PRs, leaves unrelated PRs). This is a faithful mirror sync, not a push of our own changes.

D. Add skip-when-unchanged. After sync, if the recipe's main commit == its canonical record's commit (no change since the last promotion) → skip (log SKIP unchanged). This is the operator's efficiency ask and is also the determinism property (see M2 run-twice proof).

E. Keep it deterministic + AI-free at runtime (it already is — a script + timer). The additions must stay pure code: no AI calls during the run. AI (the loops) only authors + verifies.

F. Make the timer weekly (operator preference): change the existing daily OnCalendar to weekly. The exact day/time is not critical — pick a low-traffic slot; it's a one-line tune. Persistent = true to catch up a missed run. This is the only schedule work; do not over-invest in it.

Plays-nice-with-samever (operator wants this CONFIRMED, not assumed). In the sweep, two distinct guards keep the upgrade tier from a vacuous same-version run — and they use deliberately different keys, so verify them together:

• skip-when-unchanged uses COMMIT equality (main commit == canonical commit) → if literally nothing changed, the recipe is skipped before the upgrade tier runs. This is the primary same-version avoider in the sweep.
• samever uses VERSION equality → for the case where main changed (new commit) but the version LABEL still equals the canonical's version (a non-version-bump recipe change), the upgrade tier's base would equal the head version, and samever steps back to the previous published version so a real delta is still tested. These are complementary: commit-equality (skip) is the coarse filter; version-equality (samever) is the backstop for "changed commit, same version label." Together the sweep must never run a vacuous vX → vX upgrade and never wrongly skip a real change. M2 must prove all three sweep paths explicitly (see Gates).

3. Gates

M1 — machinery works locally, each piece proven. (A) a real canonical.json is produced by a green cold run on ≥1 recipe and reused by a warm reattach — demonstrated, not assumed. (C) mirror-sync and (D) skip-when-unchanged implemented, reusing the existing reconcile + sweep code, with unit tests (skip = commit-equality; sync invoked per recipe; promote still gated on green+cold+latest+enrolled). (B) enrollment scope decided + recorded (≥several recipes enrolled, or the decouple decision). Adversary cold-verifies: a canonical actually exists + reattaches; skip-logic correct; sync is faithful-mirror-only; a RED recipe does NOT promote (prior known-good intact); no AI at runtime.

M2 — proven end-to-end in real CI (the heart of this phase). A full sweep run across the enrolled set on cc-ci: mirrors synced to upstream, canonicals actually promoted for the green recipes (records exist with correct version+commit), red recipes left intact, unchanged recipes skipped — with a per-recipe results log. Determinism proof: run the sweep a SECOND time immediately → it SKIPS every recipe (all main == the canonicals just promoted) = a clean no-op, no CI rerun. Confirm the deployed timer fires the real (non-hollow) job — after a fire, canonicals have advanced (evidence), not exit-0 on an empty set.

samever interaction proven (operator-required). Demonstrate, with evidence, all three sweep paths: (1) main commit == canonical commit → recipe SKIPPED (no upgrade tier run); (2) main changed + version bumped → upgrade tier runs canonical(older) → head(new), a real delta; (3) main changed but version label == canonical version → samever steps back to the previous published version (base version < head version), NOT a vacuous vX→vX and NOT skipped. Confirm the boundary explicitly: a same-version-label-but-different-commit recipe is not skipped (commits differ) and does hit the samever step-back. Construct the scenarios if the natural recipe set doesn't cover all three.

No AI in the loop. Fresh Adversary PASS on both milestones → ## DONE.

4. Guardrails

• Correctness over cadence. The bar is the machinery demonstrably promotes canonicals, syncs mirrors, skips unchanged, and plays nicely with samever. The cadence is decided (weekly) — set it in one OnCalendar line and move on; don't agonize over the exact slot.
• No AI at runtime — pure script + systemd timer; AI only builds/verifies.
• Single-node safety: serial; skip the whole run if a Drone/test build is in flight (reuse the existing nightly guard); tear down every deploy; bound total runtime; mind the warm-volume disk budget.
• Never force-promote / never weaken: promote only on green-cold-latest-enrolled; a red recipe keeps its prior known-good. Never weaken a test to make a recipe promote.
• Faithful mirror sync only: force-sync main/tags to coopcloud upstream; never push our own changes to mirror main; never merge/disturb unrelated PRs.
• Nix/host changes (enrollment is recipe-meta; any timer/module tweak is a nixos-rebuild): loops may deploy if clean and verify host health after; else file for the orchestrator. Commit author autonomic-bot <autonomic-bot@noreply.git.autonomic.zone>; push every commit; abra over a pseudo-TTY.

5. Definition of Done

The canonical sweep actually works and is proven: a green cold-on-latest run produces a real, reusable canonical.json; the sweep reconciles each recipe mirror's main to upstream, skips recipes whose main is unchanged vs canonical, runs CI on the rest, and promotes the canonical for any that pass — across a real multi-recipe set, demonstrated end-to-end in CI, including the run-twice no-op determinism proof and a real (non-hollow) timer fire. Enrollment scope + the warm-volume budget decided/recorded; the runtime job is AI-free; it is the substitute for the hollow nightly sweep (not a parallel job). M1 + M2 fresh Adversary PASSes in REVIEW-canon.md.