From 626badd3339f11cf730f93f41505ed94d7c77859 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Wed, 17 Jun 2026 07:07:44 +0000 Subject: [PATCH] claim(M1): canonical sweep machinery built + live-proven on custom-html MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M1 (machinery works locally, each piece proven) — code HEAD d4cc9e4, unit suite 295 passed: - M1.1 tagged-promote gate + promote-tested-version: live proof-A wrote a fresh canonical (commit df2e273 = the tag commit, correcting samever's main-HEAD 2b82eba); live proof-C green-untagged → 0 promotes, canonical byte-identical (tagged-gate blocks untagged). - M1.2 sweep_decision (version-keyed trigger) + vendored faithful recipe-mirror-sync.sh (smoke-tested: faithful no-op main/tags push, closed merged-upstream PR #2, left PR #5); nightly_sweep rewritten (mirror_sync -> trigger -> run_on_tag). Live SKIP demo on custom-html. - M1.3 all 21 used-recipes enrolled. M1.4 hollow-sweep fix (CCCI_REPO=/etc/cc-ci). M1.5 weekly timer. - M1(A) reattach: live proof-B --quick reused the retained volume green; known-good unchanged. Evidence + verify recipes in STATUS-canon.md; reasoning in JOURNAL-canon.md; DECISIONS appended. Gate: M1 CLAIMED, awaiting Adversary. Co-Authored-By: Claude Opus 4.8 --- machine-docs/BACKLOG-canon.md | 10 ++--- machine-docs/DECISIONS.md | 28 ++++++++++++ machine-docs/JOURNAL-canon.md | 38 ++++++++++++++++ machine-docs/STATUS-canon.md | 82 ++++++++++++++++++++++++++++------- 4 files changed, 138 insertions(+), 20 deletions(-) diff --git a/machine-docs/BACKLOG-canon.md b/machine-docs/BACKLOG-canon.md index 1ae3866..f4f4552 100644 --- a/machine-docs/BACKLOG-canon.md +++ b/machine-docs/BACKLOG-canon.md @@ -7,24 +7,24 @@ pieces). M2 = proven end-to-end in real CI. ### M1 — machinery works locally, each piece proven -- [ ] **M1.1 Tagged-promote gate (§2.A).** Extend `should_promote_canonical` to ALSO require the +- [x] **M1.1 Tagged-promote gate (§2.A).** Extend `should_promote_canonical` to ALSO require the tested head version corresponds to a published release tag. Add a `tagged: bool` param computed at the call site (`head_version in recipe_tags(recipe)`); keep the function pure. Untagged head → no promote. Unit tests: enrolled+green+cold+not-ref+tagged → True; each missing condition (incl. untagged) → False. -- [ ] **M1.2 Release-tag trigger + mirror-sync in the sweep (§2.C/§2.D).** New pure helper +- [x] **M1.2 Release-tag trigger + mirror-sync in the sweep (§2.C/§2.D).** New pure helper `sweep_decision(recipe, latest_tag, canon_version)` → `run` | `skip:no-new-version` | `skip:never-released`, keyed on `version_key` (NOT commit). Wire `nightly_sweep.sweep()` to, per enrolled recipe: (1) faithful mirror-sync main+tags to upstream (reuse open-recipe-pr.sh `--reconcile-only`, vendored into the repo for reproducibility); (2) compute latest release tag vs canonical; (3) skip or run cold ON THE TAG (checkout tag + `CCCI_SKIP_FETCH=1`). Unit tests for `sweep_decision` (new tag → run; equal → skip; older/no tag → skip). -- [ ] **M1.3 Enroll all recipes (§2.B).** Set `WARM_CANONICAL = True` in each of the 21 used-recipes +- [x] **M1.3 Enroll all recipes (§2.B).** Set `WARM_CANONICAL = True` in each of the 21 used-recipes `tests//recipe_meta.py`. Leave fixtures (custom-html-*-bad, concurrency, regression) alone. -- [ ] **M1.4 Hollow-sweep fix (root cause).** Make the deployed sweep read the REAL tests/ + run +- [x] **M1.4 Hollow-sweep fix (root cause).** Make the deployed sweep read the REAL tests/ + run current code: set `CCCI_REPO=/etc/cc-ci` in the sweep service and run `nightly_sweep.py` from the checkout (not the store copy). Deploy procedure pulls `/etc/cc-ci` before nixos-rebuild. -- [ ] **M1.5 Weekly timer (§2.F).** `nightly-sweep.nix` `OnCalendar` daily → weekly (one line), +- [x] **M1.5 Weekly timer (§2.F).** `nightly-sweep.nix` `OnCalendar` daily → weekly (one line), `Persistent=true` (already set). Low-traffic slot. ### M2 — proven end-to-end in real CI diff --git a/machine-docs/DECISIONS.md b/machine-docs/DECISIONS.md index 71cd50f..5372426 100644 --- a/machine-docs/DECISIONS.md +++ b/machine-docs/DECISIONS.md @@ -1462,3 +1462,31 @@ but stays well within the ≤90 s budget. Acceptable. remove it"). The all-deploys `compose.ccci.yml` overlay is now ENVIRONMENTAL-only (node-reality tweaks, no version-specific image pins or service add/drop); version-specific repairs live in `previous/`. Discourse ships NO `previous/` (base bitnamilegacy:3.5.0 deploys clean). + +## Phase canon (2026-06-17) — canonical sweep made real + +- **Tagged-promote gate (§2.A).** A canonical only ever advances to a PUBLISHED RELEASE TAG. + `should_promote_canonical` requires `tagged` (computed by the caller via + `warm_reconcile.is_released_version(recipe, head_version)`); `promote_canonical` records the TESTED + `head_version` (the release version actually exercised), NOT a re-derived `latest_version(recipe_tags)` + — these can diverge in a manual `RECIPE=` run whose `main` sits on a tag older than the newest + published tag. An untagged `main` commit never becomes a canonical. +- **New-release-tag trigger (§2.D).** The weekly sweep tests a recipe only when its latest release tag + is newer (by `warm_reconcile.version_key`) than its canonical version — NOT on new commits. No new + tag → SKIP (even if `main` has untagged commits). This gives the run-twice determinism no-op and + makes the sweep orthogonal to `samever` (version-under-test always > canonical → no same-version + step-back in the sweep). +- **Mirror-sync is a VENDORED `scripts/recipe-mirror-sync.sh`, not the nix-store + recipe-upgrade/open-recipe-pr.sh.** Rationale: open-recipe-pr.sh assumes the recipe clone's `origin` + IS coopcloud upstream, but cc-ci's abra recipe clones have inconsistent remotes (origin is variously + the mirror / coopcloud / absent). The vendored script pins an explicit coopcloud `upstream` remote + by recipe name, syncs main+TAGS (canon's trigger needs upstream tags), closes only merged-upstream + PRs, leaves unrelated PRs, and authes via the bot gitea token (self-contained, reproducible — a + systemd service must not depend on a per-skill-version nix-store path). Behaviour matches the phase's + described `--reconcile-only`: faithful mirror sync, never our own changes. +- **Hollow-sweep root cause + fix.** The deployed timer ran the nix-STORE runner copy (no `tests/`), + so `enrolled_recipes()` resolved `TESTS_DIR` to a missing dir → `[]` → no-op. Fix: the sweep runs + from `$CCCI_REPO=/etc/cc-ci` (has runner/ AND tests/); deploys `git -C /etc/cc-ci pull` + + nixos-rebuild. Sweep-logic now ships via a checkout pull (no store rebuild needed for logic-only). +- **All 21 used-recipes enrolled (§2.B); cadence weekly (§2.F).** The enroll set is exactly + `cc-ci-plan/used-recipes.md`; test fixtures stay unenrolled. diff --git a/machine-docs/JOURNAL-canon.md b/machine-docs/JOURNAL-canon.md index f496026..1e268d2 100644 --- a/machine-docs/JOURNAL-canon.md +++ b/machine-docs/JOURNAL-canon.md @@ -92,3 +92,41 @@ disk (orchestrator) rather than silently drop recipes if it binds. Retirement requires plausible's canonical to actually land at its latest green release so the dynamic resolver picks the right base — so this is sequenced AFTER M2 promotes plausible. Keep the pin if plausible can't go green dynamically (record why). + +## 2026-06-17 — M1 built + live-proven (CLAIMED) + +All M1 code landed (HEAD d4cc9e4). Reasoning behind the choices: + +- **Tagged-gate computes `tagged` at the call site, not inside the gate** — keeps + `should_promote_canonical` pure (the Adversary anti-anchoring + the existing unit-test contract). + `is_released_version` lives in warm_reconcile (owns version logic + recipe_tags I/O). +- **Promote the TESTED version (divergence fix, d4cc9e4):** the Adversary's pre-claim probe flagged + that the gate checks `head_version` but promote recorded `latest_version(recipe_tags)`. Live proof-A + made this concrete and favourable: the OLD record had commit `2b82eba` (a merge-to-main commit), + but the tag `1.13.0+1.31.1` actually points to `df2e273`. Recording the tested version's head_ref + now writes the TAG commit — strictly more correct. Sweep path was already safe (head==tag), but the + manual `RECIPE=` path needed it. +- **Why a vendored mirror-sync script, not the nix-store open-recipe-pr.sh:** the recipe clones on + cc-ci have INCONSISTENT remotes (n8n: origin=mirror; mumble: origin=coopcloud; ghost/discourse: + origin=mirror, no `upstream`). open-recipe-pr.sh assumes origin=coopcloud → would force-sync mirror + main to *mirror* main (no-op) for most. The vendored `scripts/recipe-mirror-sync.sh` pins an + explicit coopcloud `upstream` remote from the recipe name, syncs main+TAGS (canon needs upstream + tags for the trigger), and authes via the bot token (self-contained, not host .git-credentials). + Behaviour matches the phase's described open-recipe-pr.sh --reconcile-only (faithful, close + merged-upstream PRs, leave unrelated). See DECISIONS. +- **Why test the TAG via checkout+CCCI_SKIP_FETCH (run_on_tag), not just REF=tag:** REF alone (no SRC) + takes fetch_recipe's `abra recipe fetch` branch (ignores REF) AND would set `ref` → should_promote + blocks. Staging the tag in the clone + CCCI_SKIP_FETCH makes head=tag with REF empty → promote + allowed, and exercises the real "cold on the tagged release" path. + +### Live proof evidence (cc-ci, /root/canon-verify @ d4cc9e4) +- proof-A (promote): canonical.json fresh ts 065027Z, commit df2e273 (=tag commit). Note: because + custom-html canonical already == latest, run_on_tag here re-promoted an EQUAL version → the samever + step-back fired (base 1.11.0+1.29.0). That is an artifact of bypassing the trigger for the proof; + the REAL sweep SKIPs equal-version (sweep_decision), so the step-back never fires in the sweep — to + be shown live in M2 (canonical(older)→new tag, base=canonical, no step-back). +- proof-B (reattach): --quick reattached the retained volume, green (4 tests passed), known-good + version+commit UNCHANGED (df2e273); ts re-stamped only by the idle-status write (write_registry + stamps ts on every status write) — NOT a promote. +- proof-C (untagged→no-promote): green cold run (level 5/5) on an untagged head (label 1.13.1+1.31.1) + → 0 promote log lines, canonical.json byte-identical before/after. Tagged-gate works live. diff --git a/machine-docs/STATUS-canon.md b/machine-docs/STATUS-canon.md index 4f890d8..09d5b76 100644 --- a/machine-docs/STATUS-canon.md +++ b/machine-docs/STATUS-canon.md @@ -1,6 +1,6 @@ # STATUS — phase `canon` (canonical sweep, make it real) -Gate: M1 IN PROGRESS (not yet claimed). +Gate: M1 CLAIMED, awaiting Adversary. WHAT/HOW/EXPECTED/WHERE for the Adversary. Reasoning lives in JOURNAL-canon.md. @@ -22,23 +22,75 @@ weekly, and prove it in real CI. DoD = §5 of `cc-ci-plan/plan-phase-canon-canon - Timer is daily: `nix/modules/nightly-sweep.nix` `OnCalendar = "*-*-* 03:00:00"`. - Disk `/`: 40G free / 73% used (`ssh cc-ci df -h /`). -## M1 progress (code COMPLETE; live proofs in flight — not yet claimed) -All M1 code landed + unit-tested + lint-clean (full unit suite 295 passed): -- M1.1 tagged-promote gate (27e0628) + divergence fix (d4cc9e4): `should_promote_canonical` gains - `tagged`; promote records the TESTED `head_version` (a release tag), not a re-derived latest. - `warm_reconcile.is_released_version`. Tests: `tests/unit/test_promote.py`, `test_warm_reconcile.py`. -- M1.2 release-tag trigger + mirror-sync (a20890a): `warm_reconcile.sweep_decision` (pure, version_key - keyed), `scripts/recipe-mirror-sync.sh` (faithful main+tags sync, closes merged-upstream PRs), - `nightly_sweep` rewritten (mirror_sync → trigger → run_on_tag = checkout tag + CCCI_SKIP_FETCH). -- M1.3 all 21 used-recipes enrolled (136100f). M1.4 hollow-sweep fix + M1.5 weekly timer (f8c0e53). -Mirror-sync smoke-tested live on custom-html (faithful no-op push; closed merged-upstream PR #2; left -pending PR #5). +## CLAIM — M1 (machinery works locally, each piece proven) -Live M1(A) proofs in progress on cc-ci from checkout /root/canon-verify @ d4cc9e4: -- A-promote: `nightly_sweep.run_on_tag('custom-html','1.13.0+1.31.1')` → expect fresh canonical.json. +All M1 code is committed + pushed (HEAD d4cc9e4); full unit suite 295 passed; lint PASS. Live proofs +ran on cc-ci from a fresh current-main checkout at `/root/canon-verify` (@ d4cc9e4) — the same way +Drone runs CI (fresh clone + `cc-ci-run runner/run_recipe_ci.py`). Proof logs on cc-ci: +`/root/canon-verify/_proofA.log` (promote), `_proofB.log` (reattach), `_proofC.log` (untagged). + +### WHAT is claimed → HOW to verify → EXPECTED → WHERE + +**M1.1 — tagged-promote gate (§2.A).** `should_promote_canonical(recipe, ref, overall, quick, tagged)` +also requires `tagged`; the caller computes `tagged = warm_reconcile.is_released_version(recipe, +head_version)`; `promote_canonical(recipe, head_ref, version)` records the TESTED `head_version` (a +release tag), NOT a re-derived `latest_version`. + - HOW (unit): `pytest tests/unit/test_promote.py tests/unit/test_warm_reconcile.py` → all pass + (incl. `test_no_promote_when_untagged`, `test_is_released_version`). + - HOW (live PROMOTE): the new code path ran on cc-ci via `nightly_sweep.run_on_tag('custom-html', + '1.13.0+1.31.1')` (proof-A). EXPECTED+OBSERVED: `/var/lib/ci-warm/custom-html/canonical.json` + rewritten with a FRESH ts (`20260617T065027Z`) and `commit=df2e27339f983a25da548fc8b8d56e9af8645f83` + = the EXACT commit tag `1.13.0+1.31.1` points to (`git -C ~/.abra/recipes/custom-html rev-list -n1 + 1.13.0+1.31.1`). NB this CORRECTS the prior record (samever had recorded `2b82eba`, a + merge-to-main commit, NOT the tag commit). Log shows `ref=None` (cold), `CCCI_SKIP_FETCH=1` (head + = the staged tag), `WC5 promote-on-green-cold: (re)seed canonical custom-html @ 1.13.0+1.31.1`. + - HOW (live UNTAGGED→NO PROMOTE): proof-C staged an untagged head (compose version label + `1.13.1+1.31.1`, confirmed NOT a release tag, image unchanged) and ran a COLD run. EXPECTED+OBSERVED: + run GREEN (rc=0, level=5/5), `grep -c "WC5 promote-on-green-cold" _proofC.log` = **0**, and + canonical.json version+commit+ts **identical** before/after (`1.13.0+1.31.1`, `df2e273`, + `20260617T065532Z`) — the tagged-gate blocked the promote of a green-but-untagged state. + +**M1.2 — release-tag trigger + faithful mirror-sync (§2.C/§2.D).** + - `warm_reconcile.sweep_decision(latest_tag, canon_version)` (pure, keyed on `version_key`, NOT + commit): new tag>canon → run; ==/older → skip `no-new-version`; no tag → skip `never-released`. + HOW (unit): `pytest tests/unit/test_warm_reconcile.py -k sweep_decision`. HOW (live SKIP): on + cc-ci, custom-html `latest_tag == canonical (1.13.0+1.31.1)` → `sweep_decision` = + `('skip','no-new-version (latest release 1.13.0+1.31.1 <= canonical 1.13.0+1.31.1)')`. The RUN + path (new tag) is exercised by proof-A's `run_on_tag` (a real cold run on a tag). + - `scripts/recipe-mirror-sync.sh` (faithful — pins explicit coopcloud `upstream` remote, force-syncs + mirror main+TAGS to upstream, closes only merged-upstream PRs, leaves unrelated PRs, bot-token + auth). HOW (live): ran on cc-ci against custom-html → `git push` main/tags "Everything up-to-date" + (faithful no-op, no own changes), closed merged-upstream PR #2, LEFT pending PR #5 open. Diff a + mirror's main before/after to confirm it equals coopcloud upstream main, nothing else changed. + - `nightly_sweep.sweep()` wires per-recipe: `mirror_sync → fetch_recipe → sweep_decision → + run_on_tag` (checkout the release tag + `CCCI_SKIP_FETCH=1` so head IS the tag → tagged-gate + passes; REF empty → promote allowed). NO AI at runtime (pure script). + +**M1.3 — all recipes enrolled (§2.B).** `WARM_CANONICAL=True` in every `tests//recipe_meta.py` of +the 21 `cc-ci-plan/used-recipes.md` rows. HOW: `grep -rl 'WARM_CANONICAL = True' tests/*/recipe_meta.py +| wc -l` → 21; `python3 -c "import sys;sys.path.insert(0,'runner');from harness import canonical; +print(len(canonical.enrolled_recipes()))"` → 21. Fixtures (custom-html-*-bad, concurrency, regression) +NOT enrolled (not in used-recipes.md). + +**M1.4 — hollow-sweep root-cause fix.** `nix/modules/nightly-sweep.nix` sets `CCCI_REPO=/etc/cc-ci`, +`cd`s there, and execs `$CCCI_REPO/runner/nightly_sweep.py` (a checkout WITH tests/), replacing the +nix-store runner copy (no tests/ → `enrolled_recipes()=[]` → the hollow no-op). Live confirmation that +the deployed timer now enrolls all 21 is part of M2 (needs the M2.1 deploy: `git -C /etc/cc-ci pull` + +`nixos-rebuild`). HOW (code): read the module. + +**M1.5 — weekly timer (§2.F).** `OnCalendar = "Sun *-*-* 03:00:00"`, `Persistent=true`. HOW (code): +read the module; deployed-timer check is M2. + +### Build hashes / fingerprints +- HEAD: `d4cc9e4` (M1 code). custom-html canonical: `{version 1.13.0+1.31.1, commit df2e273, ts + 20260617T065532Z, status idle}`; retained volume `warm-custom-html_ci_commoninternet_net_content`. +- tag `1.13.0+1.31.1` → commit `df2e27339f983a25da548fc8b8d56e9af8645f83`. + +### Out of M1 scope (M2): full multi-recipe sweep e2e, run-twice determinism, real timer fire, samever +orthogonality live, disk budget, §2.G UPGRADE_BASE_VERSION retirement. ## Claims awaiting verification -(M1 claim pending live-proof completion) +- **M1 — CLAIMED** (this commit). Awaiting Adversary PASS in REVIEW-canon.md. ## Blocked (none)