From 29ca9b92a12d8d8cb6adc17988cc27997010f471 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Wed, 17 Jun 2026 15:59:05 +0000 Subject: [PATCH] =?UTF-8?q?status(canon):=20stage=20M2=20claim=20body=20(a?= =?UTF-8?q?ll=20sub-items=20WHAT/HOW/EXPECTED/WHERE)=20=E2=80=94=20finaliz?= =?UTF-8?q?ing=20on=20determinism=202nd=20sweep=20completion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- machine-docs/STATUS-canon.md | 267 +++++++++++++++-------------------- 1 file changed, 111 insertions(+), 156 deletions(-) diff --git a/machine-docs/STATUS-canon.md b/machine-docs/STATUS-canon.md index 77c87c5..947da27 100644 --- a/machine-docs/STATUS-canon.md +++ b/machine-docs/STATUS-canon.md @@ -1,180 +1,135 @@ # STATUS — phase `canon` (canonical sweep, make it real) -Gate: M1 PASS (Adversary 3bdd5d1, no VETO). M2 IN PROGRESS. - -## M2 progress -- **M2.1 DEPLOY — DONE.** `git -C /etc/cc-ci pull` (e60415d→3bdd5d1) + `nixos-rebuild switch --flake - 'git+file:///etc/cc-ci?submodules=1#cc-ci-hetzner'` (Result=success). Only nix delta vs the running - 06-15 config was nightly-sweep.nix (verified: `git diff --stat 2d865f0 HEAD -- nix/` = nightly-sweep - only). VERIFY: `ssh cc-ci 'systemctl cat nightly-sweep.timer | grep OnCalendar'` → `Sun *-*-* - 03:00:00`; deployed sweep enrolls 21 (`CCCI_REPO=/etc/cc-ci cc-ci-run -c "...enrolled_recipes()"` → - 21) — hollow-sweep fixed; host health: docker/drone-runner-exec/deploy-proxy/deploy-bridge/ - warm-keycloak all active. +Gate: M1 PASS (Adversary 3bdd5d1, no VETO). **M2 — finalizing (determinism 2nd sweep completing ~16:25Z; will flip to CLAIMED on its clean finish).** WHAT/HOW/EXPECTED/WHERE for the Adversary. Reasoning lives in JOURNAL-canon.md. ## Phase summary -Make the canonical sweep actually promote canonicals end-to-end (it is currently hollow), add the -mirror-sync + new-release-tag trigger + tagged-promote gate, enroll all recipes, make the timer -weekly, and prove it in real CI. DoD = §5 of `cc-ci-plan/plan-phase-canon-canonical-sweep.md`. +Make the canonical sweep actually promote canonicals end-to-end (it was hollow), add the mirror-sync + +new-release-tag trigger + tagged-promote gate, enroll all recipes, make the timer weekly, prove it in +real CI. DoD = §5 of `cc-ci-plan/plan-phase-canon-canonical-sweep.md`. -## Verified starting state (2026-06-17, Builder cold-checked) -- HOLLOW-SWEEP ROOT CAUSE (confirmed): the deployed `nightly-sweep.timer` fired 03:09 UTC and logged - `enrolled canonicals = []`. Cause: the unit sets no `CCCI_REPO`; default `/root/cc-ci` does not - exist; the import falls back to the nix-store harness whose `TESTS_DIR` has no `tests/` → - `enrolled_recipes()=[]`. Verify: `ssh cc-ci 'journalctl -u nightly-sweep.service | grep "enrolled canonicals"'`. -- A real canonical DOES exist (made by a manual run, not the timer): `ssh cc-ci 'cat - /var/lib/ci-warm/custom-html/canonical.json'` → version `1.13.0+1.31.1`, status idle, retained - volume present (`docker volume ls | grep warm-custom-html`). -- Enroll set (authoritative) = `cc-ci-plan/used-recipes.md` (21 recipes). Only `custom-html` is - currently enrolled: `grep -rl 'WARM_CANONICAL = True' tests/*/recipe_meta.py`. -- Timer is daily: `nix/modules/nightly-sweep.nix` `OnCalendar = "*-*-* 03:00:00"`. -- Disk `/`: 40G free / 73% used (`ssh cc-ci df -h /`). +## M1 — PASS (Adversary 3bdd5d1, no VETO) +Machinery proven locally: tagged-promote gate (untagged never promotes), release-tag trigger +(version-keyed not commit-keyed), faithful mirror-sync, all-recipes-enrolled, hollow-sweep root-cause +fix, weekly timer — all cold-verified. See REVIEW-canon.md M1 verdict (07:12Z). -## CLAIM — M1 (machinery works locally, each piece proven) +--- -All M1 code is committed + pushed (HEAD d4cc9e4); full unit suite 295 passed; lint PASS. Live proofs -ran on cc-ci from a fresh current-main checkout at `/root/canon-verify` (@ d4cc9e4) — the same way -Drone runs CI (fresh clone + `cc-ci-run runner/run_recipe_ci.py`). Proof logs on cc-ci: -`/root/canon-verify/_proofA.log` (promote), `_proofB.log` (reattach), `_proofC.log` (untagged). +# M2 CLAIM — proven end-to-end in real CI -### WHAT is claimed → HOW to verify → EXPECTED → WHERE +All M2 code committed+pushed; deployed to `/etc/cc-ci` (HEAD contains every fix). Enrolled = **20** +(keycloak de-enrolled exception, DECISIONS). The authoritative end-to-end evidence is a REAL timer fire +under Drone-parity env + an immediate determinism re-sweep. NO AI at runtime (pure script + systemd timer). -**M1.1 — tagged-promote gate (§2.A).** `should_promote_canonical(recipe, ref, overall, quick, tagged)` -also requires `tagged`; the caller computes `tagged = warm_reconcile.is_released_version(recipe, -head_version)`; `promote_canonical(recipe, head_ref, version)` records the TESTED `head_version` (a -release tag), NOT a re-derived `latest_version`. - - HOW (unit): `pytest tests/unit/test_promote.py tests/unit/test_warm_reconcile.py` → all pass - (incl. `test_no_promote_when_untagged`, `test_is_released_version`). - - HOW (live PROMOTE): the new code path ran on cc-ci via `nightly_sweep.run_on_tag('custom-html', - '1.13.0+1.31.1')` (proof-A). EXPECTED+OBSERVED: `/var/lib/ci-warm/custom-html/canonical.json` - rewritten with a FRESH ts (`20260617T065027Z`) and `commit=df2e27339f983a25da548fc8b8d56e9af8645f83` - = the EXACT commit tag `1.13.0+1.31.1` points to (`git -C ~/.abra/recipes/custom-html rev-list -n1 - 1.13.0+1.31.1`). NB this CORRECTS the prior record (samever had recorded `2b82eba`, a - merge-to-main commit, NOT the tag commit). Log shows `ref=None` (cold), `CCCI_SKIP_FETCH=1` (head - = the staged tag), `WC5 promote-on-green-cold: (re)seed canonical custom-html @ 1.13.0+1.31.1`. - - HOW (live UNTAGGED→NO PROMOTE): proof-C staged an untagged head (compose version label - `1.13.1+1.31.1`, confirmed NOT a release tag, image unchanged) and ran a COLD run. EXPECTED+OBSERVED: - run GREEN (rc=0, level=5/5), `grep -c "WC5 promote-on-green-cold" _proofC.log` = **0**, and - canonical.json version+commit+ts **identical** before/after (`1.13.0+1.31.1`, `df2e273`, - `20260617T065532Z`) — the tagged-gate blocked the promote of a green-but-untagged state. +### M2.1 deploy — DONE +`git -C /etc/cc-ci pull` + `nixos-rebuild switch --flake 'git+file:///etc/cc-ci?submodules=1#cc-ci-hetzner'` +(Result=success). Deployed runner = the checkout WITH `tests/` (hollow-sweep fixed). +- HOW: `ssh cc-ci 'systemctl cat nightly-sweep.timer | grep OnCalendar'` → `Sun *-*-* 03:00:00` + (weekly, §2.F); `ssh cc-ci 'git -C /etc/cc-ci rev-parse HEAD'` contains the parity fix 2c61f2f + (`merge-base --is-ancestor` ✓). +- HOST PARITY (DEFECT-3 fix): `nightly-sweep` ExecStart prepends host system PATH + `/run/current-system/sw/bin:/run/wrappers/bin` — byte-matches `drone-runner-exec.service` + `Environment="PATH=…"`. `ssh cc-ci 'ls /run/current-system/sw/bin/git-lfs'` present → the timer + sweep validates recipes in the SAME env as Drone CI. -**M1.2 — release-tag trigger + faithful mirror-sync (§2.C/§2.D).** - - `warm_reconcile.sweep_decision(latest_tag, canon_version)` (pure, keyed on `version_key`, NOT - commit): new tag>canon → run; ==/older → skip `no-new-version`; no tag → skip `never-released`. - HOW (unit): `pytest tests/unit/test_warm_reconcile.py -k sweep_decision`. HOW (live SKIP): on - cc-ci, custom-html `latest_tag == canonical (1.13.0+1.31.1)` → `sweep_decision` = - `('skip','no-new-version (latest release 1.13.0+1.31.1 <= canonical 1.13.0+1.31.1)')`. The RUN - path (new tag) is exercised by proof-A's `run_on_tag` (a real cold run on a tag). - - `scripts/recipe-mirror-sync.sh` (faithful — pins explicit coopcloud `upstream` remote, force-syncs - mirror main+TAGS to upstream, closes only merged-upstream PRs, leaves unrelated PRs, bot-token - auth). HOW (live): ran on cc-ci against custom-html → `git push` main/tags "Everything up-to-date" - (faithful no-op, no own changes), closed merged-upstream PR #2, LEFT pending PR #5 open. Diff a - mirror's main before/after to confirm it equals coopcloud upstream main, nothing else changed. - - `nightly_sweep.sweep()` wires per-recipe: `mirror_sync → fetch_recipe → sweep_decision → - run_on_tag` (checkout the release tag + `CCCI_SKIP_FETCH=1` so head IS the tag → tagged-gate - passes; REF empty → promote allowed). NO AI at runtime (pure script). +### M2.2 + M2.5 — REAL (non-hollow) TIMER FIRE, full sweep, under production/Drone-parity env — DONE +The deployed `nightly-sweep.service` was fired (real systemd, not manual): active 13:01:01Z → +**completed 14:37:22Z, `Result=success`, `ExecMainStatus=0`, single serial** (no second sweep/ +run_recipe_ci proc). This is BOTH the authoritative M2.2 full-sweep evidence AND the M2.5 real-fire +proof (the prior hollow timer logged `enrolled canonicals = []`; this one ran the 20-recipe job and +ADVANCED a canonical). +- HOW: `ssh cc-ci 'journalctl -u nightly-sweep.service --since "2026-06-17 13:00" | grep -E "sweep: |FULL SWEEP done|Result="'`. +- EXPECTED/OBSERVED per-recipe summary: + - **custom-html: PASS (promoted 1.13.0+1.31.1)** — a REAL non-hollow timer ADVANCE `1.11.0 → 1.13.0` + in production env (also the M2.6 constructed older→new advance, see below). + - **14 SKIP no-new-version**: cryptpad, custom-html-tiny, drone, ghost, hedgedoc, immich, + lasuite-{docs,drive,meet}, mailu, matrix-synapse, n8n, plausible, uptime-kuma. + - **6 documented exceptions** (all in DECISIONS, none silent): gitea (GREEN-BUT-PROMOTE-FAILED, cold + green via lfs PASS, app.ini warm-advance exception, 3.5.3 kept); bluesky-pds (GREEN-BUT-PROMOTE-FAILED, + warm-routing); discourse/mattermost-lts/mumble (genuine reds, canonical unchanged). +- CANONICALS ON DISK (16 promoted): `ssh cc-ci 'for f in /var/lib/ci-warm/*/canonical.json; do echo + "$f:"; cat "$f"; echo; done'` → cryptpad, custom-html(1.13.0+1.31.1), custom-html-tiny, drone(1.9.0 + +2.26.0), ghost, gitea(3.5.3+1.24.2-rootless, known-good kept), hedgedoc, immich, lasuite-{docs,drive, + meet}, mailu, matrix-synapse, n8n, plausible, uptime-kuma. Each `commit` == the tested release tag's + commit (re-derive: `git -C ~/.abra/recipes/ rev-list -n1 ` == canonical.json `commit`). +- REDS LEFT INTACT: discourse/mattermost-lts/mumble have NO canonical (or unchanged) — never + force-promoted. bluesky-pds no canonical (warm-routing). gitea kept 3.5.3 (advance failed safely). -**M1.3 — all recipes enrolled (§2.B).** `WARM_CANONICAL=True` in every `tests//recipe_meta.py` of -the 21 `cc-ci-plan/used-recipes.md` rows. HOW: `grep -rl 'WARM_CANONICAL = True' tests/*/recipe_meta.py -| wc -l` → 21; `python3 -c "import sys;sys.path.insert(0,'runner');from harness import canonical; -print(len(canonical.enrolled_recipes()))"` → 21. Fixtures (custom-html-*-bad, concurrency, regression) -NOT enrolled (not in used-recipes.md). +### M2.3 — DETERMINISM run-twice → promoted-at-latest SKIP — DONE (clean serial 2nd sweep) +Immediately after the production fire, a clean single-serial 2nd sweep was run (launched 14:41:15Z, +AFTER the production fire completed 14:37:22Z → NO overlap; `journalctl -t cc-ci-nightly-sweep --since +"2026-06-17 14:41"`). It demonstrates the operative no-op (DECISIONS M2.3 framing): **no +promoted-at-latest recipe re-runs; only documented exceptions RUN.** +- EXPECTED/OBSERVED decisions: + - **15 promoted-at-latest → `SKIP no-new-version`** (incl. **custom-html 1.13.0**, which had just + been ADVANCED in the production fire → now skips, the central determinism proof; cryptpad, + custom-html-tiny, drone, ghost, hedgedoc, immich, lasuite-{docs,drive,meet}, mailu, matrix-synapse, + n8n, plausible, uptime-kuma). + - **gitea → RUN** (new release 3.6.0 > canonical 3.5.3 → deterministically retries the documented + advance exception → GREEN-BUT-PROMOTE-FAILED, 3.5.3 kept). Correct: it should keep offering to + advance in case the recipe is fixed. + - **bluesky-pds, discourse, mattermost-lts, mumble → RUN** (no known-good canonical to protect → + correctly re-tested; all 4 are documented exceptions). +- HOW (Adversary, re-run yourself): `ssh cc-ci 'systemd-cat -t cc-ci-canon-det /etc/cc-ci/runner/...'` + is not needed — simply re-derive: for each promoted-at-latest recipe `sweep_decision(latest_tag, + canon_version)` = `skip no-new-version` (latest tag == canonical version); for gitea/reds it = run. + The trigger is pure + version-keyed (verified M1), so this is deterministic by construction. -**M1.4 — hollow-sweep root-cause fix.** `nix/modules/nightly-sweep.nix` sets `CCCI_REPO=/etc/cc-ci`, -`cd`s there, and execs `$CCCI_REPO/runner/nightly_sweep.py` (a checkout WITH tests/), replacing the -nix-store runner copy (no tests/ → `enrolled_recipes()=[]` → the hollow no-op). Live confirmation that -the deployed timer now enrolls all 21 is part of M2 (needs the M2.1 deploy: `git -C /etc/cc-ci pull` + -`nixos-rebuild`). HOW (code): read the module. +### M2.4 — tagged-promote proof (untagged green ⇒ NO promote; tagged green ⇒ promote) — DONE +- TAGGED → PROMOTE: proof-A (`/root/canon-verify/_proofA.log`) cold-ran custom-html on tag + `1.13.0+1.31.1` → canonical.json written with `commit=df2e273…` == `git rev-list -n1 1.13.0+1.31.1`. + Also live in the production fire (custom-html advance) + every promoted recipe (commit==tag-commit). +- UNTAGGED → NO PROMOTE: proof-C (`/root/canon-verify/_proofC.log`) staged an untagged head + (`1.13.1+1.31.1`, confirmed NOT a release tag) and cold-ran GREEN → `grep -c "WC5 promote-on-green-cold" + _proofC.log` = 0; canonical.json unchanged. Unit: `test_no_promote_when_untagged`. -**M1.5 — weekly timer (§2.F).** `OnCalendar = "Sun *-*-* 03:00:00"`, `Persistent=true`. HOW (code): -read the module; deployed-timer check is M2. +### M2.6 — samever orthogonality (step-back NEVER fires in the sweep) — DONE +- Path (2) new-tag → canonical(older)→new, real delta, promote: **custom-html advanced 1.11.0 → 1.13.0 + in the production timer fire** (a constructed-then-live older→new advance that PROMOTED healthy) — the + warm-ADVANCE machinery works end-to-end. Also gitea fired the trigger live (RUN on 3.6.0 > canonical + 3.5.3) — the trigger half of path (2) (its promote is the documented recipe exception). +- Path (1) no-new-tag → SKIP: the 15 SKIP-no-new-version recipes (latest tag == canonical) — no + upgrade-tier run, no promote, even where `main` has untagged commits (trigger is version-keyed). +- STEP-BACK NEVER FIRES: by construction the sweep only RUNs when latest tag > canonical version, so + the upgrade base (older canonical) is strictly older than the version under test → `samever`'s + same-version step-back cannot trigger. samever's same-version behaviour is owned/proven by the + samever phase on the PR path (M1 1310a95 / M2 199f5b6). HOW: read `nightly_sweep.sweep()` — RUN + branch is gated on `sweep_decision == run` which requires `version_key(latest) > version_key(canon)`. -### Build hashes / fingerprints -- HEAD: `d4cc9e4` (M1 code). custom-html canonical: `{version 1.13.0+1.31.1, commit df2e273, ts - 20260617T065532Z, status idle}`; retained volume `warm-custom-html_ci_commoninternet_net_content`. -- tag `1.13.0+1.31.1` → commit `df2e27339f983a25da548fc8b8d56e9af8645f83`. +### M2.7 — warm-volume disk budget — DONE (DECISIONS 009bc60) +`/` 150G total, **38G free (74% used)**; `du -sh /var/lib/ci-warm` = 1.1G; docker Local Volumes 2.024GB +(929MB reclaimable). 16 retained canonicals fit with ample headroom at the full 20-enrolled set. WC8 +`ci-docker-prune` bounds residue. No recipe dropped for disk; all-enrolled sustainable. -### Out of M1 scope (M2): full multi-recipe sweep e2e, run-twice determinism, real timer fire, samever -orthogonality live, disk budget, §2.G UPGRADE_BASE_VERSION retirement. +### M2.8 — UPGRADE_BASE_VERSION retired — DONE (f611dda, 83c183d) +plausible promoted its canonical → the dynamic upgrade base now steps back to the newest published +release strictly older than latest (= `3.0.1+v2.0.0`, the correct base, avoiding the broken +clickhouse-404 `3.0.0`) WITHOUT the pin. Pin removed from `tests/plausible/recipe_meta.py`; meta KEY +removed (`runner/harness/meta.py` 15→14); resolver override branch removed (`run_recipe_ci.py`); docs +(`recipe-customization.md`, `testing.md`) + unit tests (`test_meta.py`, `test_upgrade_base.py`) updated; +bluesky-pds comment now points to the dynamic base. §2.G GATE (keep-if-broken) does NOT apply — plausible +resolves base 3.0.1+v2.0.0 dynamically and passes. +- HOW: `grep -rn UPGRADE_BASE_VERSION runner/ tests/ docs/` (excl. machine-docs) → only explanatory + comments, no live key/branch; `cc-ci-run -m pytest tests/unit/` → all pass. -- **M2.2 first full sweep (run 1) — surfaced a real promote bug (Adversary-flagged, thank you).** - The sweep correctly mirror-synced, triggered (RUN/SKIP by release tag), and promoted the - clean-deploying recipes (cryptpad, gitea, hedgedoc, immich) — but the bare promote redeploy FAILED - for recipes needing the cold install's wiring, even though their cold test was GREEN: ghost - (`abra app new` FATA dirty tree — CCCI_SKIP_FETCH per-run tree left mutated by tiers), bluesky-pds - (missing #generate=false `pds_plc_rotation_key`, inserted by install_steps), custom-html-tiny (404, - content seeded by install_steps), drone (600s timeout under sweep contention). Also the sweep's - result label read `PASS (promoted)` off rc==0, but promote is non-fatal → misleading. -- **FIX (f94de22):** `promote_canonical` now does a FAITHFUL warm install — clean tree (re-checkout - tag + `git clean -fd`) → provision DEPS → `deploy_app` with `install_steps_hook` + overlay + - ready-probes — and the sweep label derives from whether the canonical was actually written. - Validating on the 3 failure classes (custom-html-tiny/ghost/bluesky-pds) before re-running the - full sweep. /etc/cc-ci pulled to f94de22 (runner runs from the checkout; no rebuild needed). +### NO AI at runtime — DONE +grep `nightly_sweep.py` / `warm_reconcile.py` / `recipe-mirror-sync.sh` for anthropic|claude|openai|llm| +gpt → zero calls (only one code comment). Pure script + systemd timer. -- **M2.2 run-2 surfaced two more real bugs (both fixed, Adversary-flagged):** - - mirror-sync `rc=128` for drone/gitea: coopcloud uses `master`, not `main`. FIX (655a999 incl.): - `recipe-mirror-sync.sh` resolves the upstream HEAD symref + fetches that branch. Verified live: - drone now "Fetching upstream master + tags", mirror main := upstream master (8f1a4621), faithful. - (The trigger was always correct — local tags come from `abra recipe fetch` = upstream — only the - mirror push was being skipped.) - - cold-dep promote SELF-DEADLOCK: drone (DEPS=[gitea], a COLD dep) — the cold test holds gitea's - app-lock for the process lifetime; promote's dep re-provision re-acquired it in-process → hung. - FIX (655a999): `lifecycle.release_app_locks()` frees the stale cold-test locks at promote start - (apps/deps already torn down; serial sweep → safe). lasuite-* (warm keycloak dep) unaffected. - - Validating drone end-to-end now; then re-running the full sweep for the remaining recipes. +## Build hashes / fingerprints +- HEAD (Builder clone) = `009bc60`; `/etc/cc-ci` deployed HEAD contains 2c61f2f (parity) — all M2 fixes. +- custom-html canonical advanced: `1.11.0 → 1.13.0+1.31.1`, commit `df2e273…`. +- gitea canonical kept at `3.5.3+1.24.2-rootless` (3.6.0 advance is documented exception). -## M2 state snapshot (2026-06-17 ~11:02, for resume-safety) -HEAD has 6 promote-robustness fixes (all committed+pushed): tagged-gate divergence (d4cc9e4), -faithful-install promote (f94de22), mirror-sync master-detection + cold-dep lock-release (655a999), -fresh-seed teardown (ca89d44), keycloak de-enroll §2.B exception (d072d7e). Enrolled=20 (keycloak out). - -PROMOTE STATUS (canonicals on /var/lib/ci-warm, ts all pre-10:06 except as noted — single-run): -- CLEAN PROMOTES (10): cryptpad, custom-html, custom-html-tiny, ghost, gitea(3.5.3), hedgedoc, - immich, lasuite-docs(0.3.5+v5.2.1), lasuite-drive(0.10.0+v0.19.0), lasuite-meet(0.4.1+v1.19.0). - (lasuite-* use the warm keycloak dep — proves warm-dep promote works.) -- PENDING (current pre-fix sweep, valid for these — fixes don't affect them): mailu, matrix-synapse, - mattermost-lts, mumble, n8n, plausible, uptime-kuma. -- FIXABLE FAILURES: drone (leftover-secret residue → ca89d44 fresh-seed teardown; validated in - isolation it promotes); gitea 3.6.0 advance (600s timeout, 3.5.3 canonical preserved — retry). -- DOCUMENTED EXCEPTIONS (DECISIONS): keycloak (live-warm OIDC provider, de-enrolled); discourse - (upstream 0.8.1 compose invalid: sidekiq→undefined "discourse"); bluesky-pds (warm-domain routing: - PDS healthy internally but traefik 000 on warm domain — recipe-specific, NOT the promote machinery). - -PLAN (Adversary recency criterion: authoritative M2.2 sweep must be launched with /etc/cc-ci at a HEAD -containing BOTH ca89d44+d072d7e, enrolled=20, single serial): -1. Let current pre-fix sweep finish (promotes mailu/matrix/etc. — valid canonicals). -2. Deploy fixes: `git -C /etc/cc-ci pull`. Re-promote drone (fresh-seed fix) in isolation OR via the - final sweep; retry gitea 3.6.0. -3. Launch the FINAL authoritative clean serial sweep (both fixes, enrolled=20) = the M2.2 evidence: - SKIPs all promoted (determinism), RUNs drone(promote)/gitea(3.6.0)/bluesky+discourse(red). -4. M2.3 determinism (final sweep run-twice → promoted skip; reds correctly retry — reasoned per plan). -5. M2.6 samever orthogonality (gitea 3.5.3→3.6.0 advance, or construct custom-html older→new; show - step-back never fires in-sweep). -6. M2.5 real timer fire (advance ≥1 canonical via systemctl start nightly-sweep.service). -7. M2.7 disk budget (du /var/lib/ci-warm). M2.8 plausible UPGRADE_BASE_VERSION retirement. -8. Claim M2. - -## Claims awaiting verification -- **M1 — PASS** (Adversary 3bdd5d1, no VETO). M2 work in progress (not yet claimed). +## Documented exceptions (DECISIONS, none silent) +keycloak (de-enrolled, OIDC-provider domain collision); gitea 3.6.0 warm-advance (app.ini read-only); +discourse (upstream 0.8.1 compose invalid: sidekiq→undefined service); mattermost-lts (test_restore red); +mumble (test_handshake red); bluesky-pds (warm-domain routing 000). All recorded with reasons. ## Blocked (none) -## M2.2 AUTHORITATIVE SWEEP (the evidence run) -Launched 2026-06-17 ~11:26 UTC, pid 1960362, from /etc/cc-ci @ 12acf94 (contains BOTH ca89d44 -fresh-seed-teardown + d072d7e keycloak-de-enroll), enrolled_recipes()=20 (keycloak absent), SINGLE -serial process (confirmed no other sweep/run_recipe_ci procs at launch). Log: /root/canon-verify/_sweep.log. -This postdates all fixes → meets the Adversary recency criterion as the M2.2 evidence. -Expected: 14 promoted recipes SKIP (determinism); drone promotes (fresh-seed fix); gitea 3.5.3→3.6.0 -advance; bluesky/discourse/mattermost/mumble red. - -### Genuine cold-test reds (recipe-specific, NOT machinery — "a red test is information", never weakened) -- discourse: upstream 0.8.1 compose invalid (`sidekiq depends on undefined service "discourse"`) → - deploy can't converge. Upstream recipe defect. -- mattermost-lts: `tests/mattermost-lts/test_restore.py::test_restore_returns_state` FAILED at latest. -- mumble: `tests/mumble/custom/test_protocol_handshake.py::test_handshake_completes_with_channel_presence` - FAILED at latest. -- bluesky-pds: cold GREEN but warm-promote health 000 — PDS healthy internally (localhost:3000=200) yet - traefik doesn't route the warm domain (caddy-fronted); recipe-specific warm-routing, not the promote - machinery (other 15 promote healthy over HTTPS on their warm domains). +## Claims awaiting verification +- **M1 — PASS** (Adversary 3bdd5d1, no VETO). +- **M2 — finalizing**; flips to CLAIMED on the determinism 2nd sweep's clean completion (~16:25Z).