cc-ci-orchestrator/cc-ci-plan/plan-phase-canon-canonical-sweep.md

# Phase `canon` — make the canonical sweep actually work (the real "nightly sweep") + verify it

**Mission (operator-specified 2026-06-17):** the "nightly sweep" was specified in theory but **was never
actually doing anything** — confirmed live: `nightly-sweep.timer` is deployed and fires green
(`nightly_sweep.py`, last run 2026-06-17 03:09 UTC exit 0), but **only `custom-html` is `WARM_CANONICAL`
-enrolled and ZERO `canonical.json` records exist** — i.e. the machinery has **never actually promoted a
canonical end-to-end**. This phase makes it **real and proven**, as the **substitute for** that hollow
nightly sweep, with two additions the operator wants:

1. **Sync each recipe mirror's `main`** on `git.autonomic.zone/recipe-maintainers/<recipe>` to its
   **upstream** (`git.coopcloud.tech/coop-cloud/<recipe>`) first, so the sweep tests true upstream latest.
2. **Skip a recipe whose `main` is unchanged** vs its current canonical (no rerun needed).

…then **run CI cold-on-`main` for each recipe and actually promote the canonical for any that pass** —
and **prove the whole thing works**. **The deliverable is correctness, verified end-to-end** — and the
operator specifically wants confidence it **plays nicely with the `samever` upgrade-base work** (§2
"Plays-nice-with-samever"). Operator decisions (2026-06-17): **all recipes enrolled** (§2.B), and the
**cadence is weekly** (change the existing daily timer to weekly — a one-line `OnCalendar` tune; exact
day/time is not critical). This REPLACES the hollow nightly sweep; it is not a parallel job.

State files: `STATUS-canon.md`, `BACKLOG-canon.md`, `REVIEW-canon.md`, `JOURNAL-canon.md`. DECISIONS.md shared.

## 1. Verified starting state (2026-06-17)

- `nightly-sweep.timer` enabled + active (next ~03:00 UTC); `nightly_sweep.py` runs and exits 0. The
  timer/service plumbing already works — **reuse it, don't rebuild it.**
- **Only `custom-html` sets `WARM_CANONICAL = True`.** The sweep iterates `canonical.enrolled_recipes()`
  → essentially one recipe → near-no-op across the fleet.
- **No `canonical.json` exists** on the host → the promote path (`should_promote_canonical` →
  `promote_canonical` → `write_registry`) has **never successfully produced a canonical**, even for
  custom-html. This is the crux of "theory, not actually doing it."
- The sweep does **not** reconcile mirrors to upstream, and does **not** skip-when-unchanged.

## 2. The work

**A. Prove + fix the promote path FIRST (the core).** On `custom-html` (already enrolled), make a green
cold-on-latest run **actually write `canonical.json`** (recipe/version/commit/status) AND prove a
subsequent `--quick` warm-reattach uses it (`deploy_canonical` reattaches the retained volume). If it
doesn't happen today, find and fix why (this is the real defect behind the hollow sweep). A canonical
must demonstrably exist and be reusable before anything else is meaningful.

**B. Enroll ALL recipes (operator decision 2026-06-17).** Set `WARM_CANONICAL = True` for **every** recipe
cc-ci tracks (the `used-recipes.md` set) — the sweep promotes a canonical for each that passes, not just
custom-html.
  - **Watch the warm-volume disk budget:** ~21 recipes each retaining a data volume on the single node is
    real disk. Verify headroom, lean on the existing WC8 disk-hygiene / `ci-docker-prune`, and if disk
    becomes the binding limit, **raise it** rather than silently dropping recipes (a fallback if needed:
    decouple the cheap last-green *version record* — kept for all — from the expensive retained *volume*).
    Default remains all-enrolled.
  - If a specific recipe genuinely cannot be enrolled (e.g. unbounded data, no stable health), record the
    exception + reason in DECISIONS — don't silently skip it.

**C. Add the upstream mirror-sync step.** Before the per-recipe CI, reconcile each mirror's `main` + tags
to coopcloud upstream — reuse `recipe-upgrade`'s `open-recipe-pr.sh <recipe> --reconcile-only` (handles
go-git private-mirror auth, fetches coopcloud via an `upstream` remote, closes already-merged-upstream
PRs, leaves unrelated PRs). This is a **faithful mirror sync, not a push of our own changes.**

**D. Add skip-when-unchanged.** After sync, if the recipe's `main` commit == its canonical record's commit
(no change since the last promotion) → **skip** (log `SKIP unchanged`). This is the operator's efficiency
ask and is also the determinism property (see M2 run-twice proof).

**E. Keep it deterministic + AI-free at runtime** (it already is — a script + timer). The additions must
stay pure code: no AI calls during the run. AI (the loops) only authors + verifies.

**F. Make the timer weekly** (operator preference): change the existing daily `OnCalendar` to weekly. The
exact day/time is not critical — pick a low-traffic slot; it's a one-line tune. `Persistent = true` to
catch up a missed run. This is the only schedule work; do not over-invest in it.

**Plays-nice-with-`samever` (operator wants this CONFIRMED, not assumed).** In the sweep, two distinct
guards keep the upgrade tier from a vacuous same-version run — and they use **deliberately different
keys**, so verify them together:
  - **skip-when-unchanged uses COMMIT equality** (`main` commit == canonical commit) → if literally
    nothing changed, the recipe is skipped *before* the upgrade tier runs. This is the primary
    same-version avoider in the sweep.
  - **`samever` uses VERSION equality** → for the case where `main` *changed* (new commit) but the version
    LABEL still equals the canonical's version (a non-version-bump recipe change), the upgrade tier's base
    would equal the head version, and `samever` steps back to the previous published version so a real
    delta is still tested.
  These are complementary: commit-equality (skip) is the coarse filter; version-equality (`samever`) is
  the backstop for "changed commit, same version label." Together the sweep must **never** run a vacuous
  `vX → vX` upgrade **and never** wrongly skip a real change. M2 must prove all three sweep paths
  explicitly (see Gates).

## 3. Gates

**M1 — machinery works locally, each piece proven.** (A) a real `canonical.json` is produced by a green
cold run on ≥1 recipe and reused by a warm reattach — **demonstrated, not assumed**. (C) mirror-sync and
(D) skip-when-unchanged implemented, reusing the existing reconcile + sweep code, with unit tests
(skip = commit-equality; sync invoked per recipe; promote still gated on green+cold+latest+enrolled).
(B) enrollment scope decided + recorded (≥several recipes enrolled, or the decouple decision). Adversary
cold-verifies: a canonical actually exists + reattaches; skip-logic correct; sync is faithful-mirror-only;
a RED recipe does NOT promote (prior known-good intact); no AI at runtime.

**M2 — proven end-to-end in real CI (the heart of this phase).** A full sweep run across the enrolled set
on cc-ci: mirrors synced to upstream, **canonicals actually promoted for the green recipes** (records
exist with correct version+commit), red recipes left intact, unchanged recipes skipped — with a
per-recipe results log. **Determinism proof: run the sweep a SECOND time immediately → it SKIPS every
recipe** (all `main` == the canonicals just promoted) = a clean no-op, no CI rerun. Confirm the **deployed
timer fires the real (non-hollow) job** — after a fire, canonicals have advanced (evidence), not exit-0
on an empty set.

**`samever` interaction proven (operator-required).** Demonstrate, with evidence, all three sweep paths:
(1) `main` commit == canonical commit → recipe SKIPPED (no upgrade tier run); (2) `main` changed +
version bumped → upgrade tier runs `canonical(older) → head(new)`, a real delta; (3) `main` changed but
version label == canonical version → `samever` steps back to the previous published version (base version
< head version), NOT a vacuous `vX→vX` and NOT skipped. Confirm the boundary explicitly: a
same-version-label-but-different-commit recipe is **not** skipped (commits differ) and **does** hit the
`samever` step-back. Construct the scenarios if the natural recipe set doesn't cover all three.

No AI in the loop. Fresh Adversary PASS on both milestones → `## DONE`.

## 4. Guardrails

- **Correctness over cadence.** The bar is the machinery *demonstrably promotes canonicals, syncs mirrors,
  skips unchanged, and plays nicely with `samever`.* The cadence is decided (**weekly**) — set it in one
  `OnCalendar` line and move on; don't agonize over the exact slot.
- **No AI at runtime** — pure script + systemd timer; AI only builds/verifies.
- **Single-node safety:** serial; skip the whole run if a Drone/test build is in flight (reuse the
  existing nightly guard); tear down every deploy; bound total runtime; mind the warm-volume disk budget.
- **Never force-promote / never weaken:** promote only on green-cold-latest-enrolled; a red recipe keeps
  its prior known-good. Never weaken a test to make a recipe promote.
- **Faithful mirror sync only:** force-sync `main`/tags to coopcloud upstream; never push our own changes
  to mirror `main`; never merge/disturb unrelated PRs.
- **Nix/host changes** (enrollment is recipe-meta; any timer/module tweak is a nixos-rebuild): loops may
  deploy if clean and **verify host health after**; else file for the orchestrator. Commit author
  `autonomic-bot <autonomic-bot@noreply.git.autonomic.zone>`; push every commit; abra over a pseudo-TTY.

## 5. Definition of Done

The canonical sweep **actually works and is proven**: a green cold-on-latest run produces a real,
reusable `canonical.json`; the sweep reconciles each recipe mirror's `main` to upstream, skips recipes
whose `main` is unchanged vs canonical, runs CI on the rest, and promotes the canonical for any that pass
— across a real multi-recipe set, demonstrated end-to-end in CI, including the run-twice no-op determinism
proof and a real (non-hollow) timer fire. Enrollment scope + the warm-volume budget decided/recorded; the
runtime job is AI-free; it is the substitute for the hollow nightly sweep (not a parallel job). M1 + M2
fresh Adversary PASSes in REVIEW-canon.md.