cc-ci-orchestrator/cc-ci-plan/plan-phase-canon-canonical-sweep.md

# Phase `canon` — make the canonical sweep actually work (the real "nightly sweep") + verify it

**Mission (operator-specified 2026-06-17):** the "nightly sweep" was specified in theory but **was never
actually doing anything** — confirmed live: `nightly-sweep.timer` is deployed and fires green
(`nightly_sweep.py`, last run 2026-06-17 03:09 UTC exit 0), but **only `custom-html` is `WARM_CANONICAL`
-enrolled and ZERO `canonical.json` records exist** — i.e. the machinery has **never actually promoted a
canonical end-to-end**. This phase makes it **real and proven**, as the **substitute for** that hollow
nightly sweep, with the operator's refinements (2026-06-17):

1. **Sync each recipe mirror's `main`** on `git.autonomic.zone/recipe-maintainers/<recipe>` to its
   **upstream** (`git.coopcloud.tech/coop-cloud/<recipe>`) first, so the sweep sees true upstream
   tags/latest.
2. **Trigger on a new RELEASED VERSION, not a new commit.** Test a recipe only when its latest **release
   tag** on the synced `main` is **newer** than its current canonical version — **skip when there is no
   new version**, even if `main` has new *untagged* commits. The sweep tests releases, not arbitrary commits.
3. **Promote the canonical only to a TAGGED release.** A canonical advances only to a version that has a
   real release tag (a published release) — never to an arbitrary untagged commit.

…then **run CI cold-on-`main` for each recipe and actually promote the canonical for any that pass** —
and **prove the whole thing works**. **The deliverable is correctness, verified end-to-end** — and the
operator specifically wants confidence it **plays nicely with the `samever` upgrade-base work** (§2
"Plays-nice-with-samever"). Operator decisions (2026-06-17): **all recipes enrolled** (§2.B), and the
**cadence is weekly** (change the existing daily timer to weekly — a one-line `OnCalendar` tune; exact
day/time is not critical). This REPLACES the hollow nightly sweep; it is not a parallel job.

State files: `STATUS-canon.md`, `BACKLOG-canon.md`, `REVIEW-canon.md`, `JOURNAL-canon.md`. DECISIONS.md shared.

## 1. Verified starting state (2026-06-17)

- `nightly-sweep.timer` enabled + active (next ~03:00 UTC); `nightly_sweep.py` runs and exits 0. The
  timer/service plumbing already works — **reuse it, don't rebuild it.**
- **Only `custom-html` sets `WARM_CANONICAL = True`.** The sweep iterates `canonical.enrolled_recipes()`
  → essentially one recipe → near-no-op across the fleet.
- **No `canonical.json` exists** on the host → the promote path (`should_promote_canonical` →
  `promote_canonical` → `write_registry`) has **never successfully produced a canonical**, even for
  custom-html. This is the crux of "theory, not actually doing it."
- The sweep does **not** reconcile mirrors to upstream, and does **not** skip-when-unchanged.

## 2. The work

**A. Prove + fix the promote path FIRST (the core).** On `custom-html` (already enrolled), make a green
cold-on-latest run **actually write `canonical.json`** (recipe/version/commit/status) AND prove a
subsequent `--quick` warm-reattach uses it (`deploy_canonical` reattaches the retained volume). If it
doesn't happen today, find and fix why (this is the real defect behind the hollow sweep). A canonical
must demonstrably exist and be reusable before anything else is meaningful.
  - **Promote-gate addition (operator 2026-06-17): only promote to a TAGGED release.** Extend
    `should_promote_canonical` so a promote ALSO requires the tested version to correspond to a published
    **release tag** (`warm_reconcile.recipe_tags`): green + cold + latest + enrolled **+ tagged**. The
    canonical must always be a real release — never an arbitrary untagged `main` commit. An untagged state
    must never be written as a canonical.

**B. Enroll ALL recipes (operator decision 2026-06-17).** Set `WARM_CANONICAL = True` for **every** recipe
cc-ci tracks (the `used-recipes.md` set) — the sweep promotes a canonical for each that passes, not just
custom-html.
  - **Watch the warm-volume disk budget:** ~21 recipes each retaining a data volume on the single node is
    real disk. Verify headroom, lean on the existing WC8 disk-hygiene / `ci-docker-prune`, and if disk
    becomes the binding limit, **raise it** rather than silently dropping recipes (a fallback if needed:
    decouple the cheap last-green *version record* — kept for all — from the expensive retained *volume*).
    Default remains all-enrolled.
  - If a specific recipe genuinely cannot be enrolled (e.g. unbounded data, no stable health), record the
    exception + reason in DECISIONS — don't silently skip it.

**C. Add the upstream mirror-sync step.** Before the per-recipe CI, reconcile each mirror's `main` + tags
to coopcloud upstream — reuse `recipe-upgrade`'s `open-recipe-pr.sh <recipe> --reconcile-only` (handles
go-git private-mirror auth, fetches coopcloud via an `upstream` remote, closes already-merged-upstream
PRs, leaves unrelated PRs). This is a **faithful mirror sync, not a push of our own changes.**

**D. Trigger on a new RELEASED VERSION (skip when no new version).** After sync, compute the recipe's
latest **release tag** version reachable on `main` and compare it to the canonical record's version:
  - latest release tag **== canonical version** → **skip** (`SKIP no-new-version`) — *even if `main` has
    new untagged commits*. The sweep tests releases, not arbitrary commits.
  - latest release tag **newer than** canonical → run CI **cold on that tagged version** → promote on green
    (tagged, per §2.A).
  - no release tag at all (recipe never released) → skip with a recorded reason.
  This is the operator's trigger refinement (version/tag-keyed, **not** commit-keyed) and the determinism
  property (M2 run-twice → everything skips).

**E. Keep it deterministic + AI-free at runtime** (it already is — a script + timer). The additions must
stay pure code: no AI calls during the run. AI (the loops) only authors + verifies.

**F. Make the timer weekly** (operator preference): change the existing daily `OnCalendar` to weekly. The
exact day/time is not critical — pick a low-traffic slot; it's a one-line tune. `Persistent = true` to
catch up a missed run. This is the only schedule work; do not over-invest in it.

**G. Retire `UPGRADE_BASE_VERSION` if plausible no longer needs it (operator 2026-06-17).** Today it is
still used: **`plausible`** sets `UPGRADE_BASE_VERSION = "3.0.1+v2.0.0"` (the old static `[-2]` default
picked `3.0.0`, whose clickhouse entrypoint 404s on amd64 → base never converges; the pin forces the
newest published `3.0.1`); **`bluesky-pds`** only *references* it (in an `EXPECTED_NA` upgrade-skip note as
a future re-enable path). Once this phase enrolls plausible and promotes its canonical to its latest green
release (`3.0.1`), the **dynamic base resolves to `3.0.1` on its own** — the correct base, avoiding the
broken `3.0.0` — so the explicit pin becomes redundant. Therefore:
  - With plausible's canonical established at `3.0.1`, **remove the pin from `tests/plausible/recipe_meta.py`
    and confirm its upgrade tier still resolves the correct base (`3.0.1`) and passes** under the dynamic
    resolver.
  - If that holds, **strip `UPGRADE_BASE_VERSION` entirely**: the meta key (`runner/harness/meta.py` KEYS),
    the override branch in `resolve_upgrade_base` (`run_recipe_ci.py`), the docs (`recipe-customization.md`
    §4/§5, `testing.md`), and the unit tests (`test_meta.py`, `test_upgrade_base.py`); and update
    `bluesky-pds`'s comment so its re-enable path is the dynamic base, not the removed key.
  - **GATE (do not force it):** if plausible genuinely still can't get the right base dynamically (e.g.
    `3.0.1` itself won't cold-deploy green, so no canonical), **KEEP `UPGRADE_BASE_VERSION`** as the
    escape-hatch and record why in DECISIONS — never drop a recipe's upgrade coverage to delete a key.

**Plays-nice-with-`samever` (operator wants this CONFIRMED).** The release-tag trigger (D) makes the
sweep and `samever` **orthogonal** — confirm they don't interfere:
  - In the **sweep**, a recipe runs **only when a new release tag exists**, so the version under test is
    always *newer* than the canonical → the upgrade tier's base (previous canonical/released version) is
    strictly older → **`samever`'s same-version step-back never fires in the sweep** (the tag trigger
    already prevents a `vX→vX` run; no-new-version recipes are skipped outright).
  - `samever` remains the guard for the **PR path** (`!testme`), where a PR can carry the same version
    label as the canonical without cutting a release — that's where the step-back matters, and it's
    owned/proven by the `samever` phase, not here.
  So in the sweep, verify only: (a) no new release tag → recipe SKIPPED (no upgrade-tier run, no promote);
  (b) new release tag → `canonical(older) → new tagged version`, a real delta, promote (tagged). The sweep
  must never promote an untagged version and never run a same-version upgrade.

## 3. Gates

**M1 — machinery works locally, each piece proven.** (A) a real `canonical.json` is produced by a green
cold run on ≥1 recipe and reused by a warm reattach — **demonstrated, not assumed** — and the promote gate
now also requires a **release tag** (untagged → no promote). (C) mirror-sync and (D) the **new-release-tag
trigger** implemented, reusing the existing reconcile + sweep code, with unit tests (trigger = latest
release tag vs canonical version, NOT commit; sync invoked per recipe; promote gated on
green+cold+latest+enrolled+**tagged**). (B) all recipes enrolled. Adversary cold-verifies: a canonical
actually exists + reattaches; an **untagged** state never promotes; the trigger skips no-new-tag recipes
and runs new-tag ones; sync is faithful-mirror-only; a RED recipe does NOT promote; no AI at runtime.

**M2 — proven end-to-end in real CI (the heart of this phase).** A full sweep run across the enrolled set
on cc-ci: mirrors synced to upstream, **canonicals actually promoted for the green recipes** (records
exist with correct version+commit), red recipes left intact, unchanged recipes skipped — with a
per-recipe results log. **Determinism proof: run the sweep a SECOND time immediately → it SKIPS every
recipe** (latest release tag == canonical version for all → skip all) = a clean no-op, no CI rerun.
Confirm the **deployed timer fires the real (non-hollow) job** — after a fire, canonicals have advanced
(evidence), not exit-0 on an empty set. **Tagged-promote proven:** show a green run on an untagged state
does NOT promote, and a green run on a tagged release DOES.

**`samever` orthogonality proven (operator-required).** Demonstrate, with evidence, the two sweep paths:
(1) **no new release tag** (latest tag == canonical version, even with new untagged commits on `main`) →
recipe SKIPPED — no upgrade-tier run, no promote; (2) **new release tag** → cold-test the new tagged
version, upgrade `canonical(older) → new`, a real delta, promote (because tagged). Confirm `samever`'s
step-back **never fires inside the sweep** (the tag trigger prevents same-version runs) — its same-version
behavior is owned/proven by the `samever` phase on the PR path. Construct scenarios if the live recipe set
doesn't cover both.

No AI in the loop. Fresh Adversary PASS on both milestones → `## DONE`.

## 4. Guardrails

- **Correctness over cadence.** The bar is the machinery *demonstrably promotes canonicals, syncs mirrors,
  skips unchanged, and plays nicely with `samever`.* The cadence is decided (**weekly**) — set it in one
  `OnCalendar` line and move on; don't agonize over the exact slot.
- **No AI at runtime** — pure script + systemd timer; AI only builds/verifies.
- **Single-node safety:** serial; skip the whole run if a Drone/test build is in flight (reuse the
  existing nightly guard); tear down every deploy; bound total runtime; mind the warm-volume disk budget.
- **Never force-promote / never weaken:** promote only on green-cold-latest-enrolled; a red recipe keeps
  its prior known-good. Never weaken a test to make a recipe promote.
- **Faithful mirror sync only:** force-sync `main`/tags to coopcloud upstream; never push our own changes
  to mirror `main`; never merge/disturb unrelated PRs.
- **Nix/host changes** (enrollment is recipe-meta; any timer/module tweak is a nixos-rebuild): loops may
  deploy if clean and **verify host health after**; else file for the orchestrator. Commit author
  `autonomic-bot <autonomic-bot@noreply.git.autonomic.zone>`; push every commit; abra over a pseudo-TTY.

## 5. Definition of Done

The canonical sweep **actually works and is proven**: a green cold run on a **tagged release** produces a
real, reusable `canonical.json` (an untagged state never promotes); the sweep reconciles each recipe
mirror's `main` to upstream, **skips recipes with no new release tag** (even if `main` has new untagged
commits), runs CI cold on the new tagged version for the rest, and promotes the canonical only to that
tagged release — across all enrolled recipes, demonstrated end-to-end in CI, including the run-twice no-op
determinism proof, the tagged-promote proof, and a real (non-hollow) timer fire. `samever` confirmed
orthogonal (never fires in the sweep). All recipes enrolled + warm-volume budget recorded. `UPGRADE_BASE_VERSION`
retired (key + resolver branch + docs + tests removed, plausible migrated to the dynamic base) **if**
plausible works without it — else kept with a recorded reason (§2.G). The runtime job is AI-free; it is the
substitute for the hollow nightly sweep (not a parallel job). M1 + M2 fresh Adversary PASSes in REVIEW-canon.md.