diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index 6915d0b..786d54d 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -74,7 +74,34 @@ hold). Concrete fix designs from M1 evidence: (Adversary-owned — do not edit.) -### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **OPEN, BLOCKING (VETO)** @2026-07-08T23:56Z +### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **CLOSED @2026-07-09T00:18Z (VETO CLEARED)** + +**CLOSED by Adversary re-test.** Fixed at `redfix-m2-harness`@`b5f2b10` (parent `07fc6d4`): `canonical_ns()` +is now the single namespace behind BOTH the canonical's domain and its warm-state slot, so a live-warm +provider gets slot `canon-/`, disjoint from the reconciler's `/`. Plus a +naming-independent `_assert_slot_not_foreign()` guard on snapshot AND restore. + +My cold re-test (full evidence in REVIEW-redfix.md @2026-07-09T00:18Z): the published clearing condition is +met verbatim — slots disjoint, `restore(canon)` and `restore(live)` each return their OWN stack's volumes, +reconciler `last_good` survives, foreign snapshot *and* foreign restore both refused. Beyond the Builder's +own checks I added four: (a) the canon mariadb volume is byte-identical across the destructive restore +round-trip (`4271926745 166164480`, 386 files); (b) **mutation testing** — reverting `canonical_ns()` reds 4 +of the new tests, removing the guard reds 2, so the 315→325 test delta is not vacuous; (c) every +`snapshot`/`restore`/`app_dir` caller now passes an explicit slot, no bare recipe survives; (d) all 21 +enrolled recipes still resolve to their existing on-disk dirs (`registry_path("bluesky-pds")` is +character-identical at parent and fix) — zero blast radius, no migration needed. + +Consequences 1–3 resolved. Consequence 4 (`prune_stale`) is now structural: `/` never gains a +`canonical.json`, verified in a scratch root. Enrollment retained (`WARM_CANONICAL = True`) — no silent +de-enrollment. The two false "can never touch each other" comments are gone. + +Residual **B-redfix-5** (reconciler rollback `restore()` outside the upgrade's `try/except`) is NOT part of +this finding's clearing condition and is **not** blocking: I confirmed it is verbatim present at parent +`07fc6d4`, so it predates the enrollment. F-redfix-4 made it *reachable*; that path is now closed. + +
Original report (as filed 2026-07-08T23:56Z) + +### [adversary] F-redfix-4 — original text — **OPEN, BLOCKING (VETO)** **Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges `redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone` @@ -173,6 +200,8 @@ de-enroll keycloak and re-open the DoD item, so it is not sufficient. survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and seeing each `restore()` return its OWN stack's volumes. +
+ --- ### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z** diff --git a/machine-docs/REVIEW-redfix.md b/machine-docs/REVIEW-redfix.md index 877d742..2bcb3ab 100644 --- a/machine-docs/REVIEW-redfix.md +++ b/machine-docs/REVIEW-redfix.md @@ -575,3 +575,77 @@ JOURNAL-redfix.md for context. It contains **zero** mentions of `snap_dir` / `ap `seed_canonical` / warm-state. The keycloak entry (@2026-06-18T01:05Z) reasons only about the *domain* namespace. So F-redfix-4 is a genuine blind spot in the fix's design, not a hazard the Builder identified and knowingly deferred. This does not change the verdict. + +### M2 (F-redfix-4 remedy) — **PASS @2026-07-09T00:18Z**. VETO CLEARED. F-redfix-4 CLOSED. + +Cold-verified the Builder's re-claim of `redfix-m2-harness` @`b5f2b10` (parent `07fc6d4` — exactly the sha +the earlier M2 PASS was given against, so the other five fixes are provably untouched by this commit: +`git diff --stat` shows only `warmsnap.py`, `canonical.py`, `warm_reconcile.py`, `run_recipe_ci.py`, +`tests/keycloak/recipe_meta.py` + two unit-test files). Fresh clone on cc-ci, my own runs throughout. + +**1. The clearing condition I published — MET, verbatim.** Scratch `CCCI_WARM_ROOT`, real idle +`warm-canon-keycloak` stack, throwaway `warm-advlive…` volume as the live stand-in: + +``` +live_slot = keycloak -> /tmp/advw/keycloak/snapshot +canonical_slot = canon-keycloak -> /tmp/advw/canon-keycloak/snapshot SLOTS DISJOINT = True +registry_path = /tmp/advw/canon-keycloak/canonical.json (NOT the reconciler's dir) +canonical_domain UNCHANGED = warm-canon-keycloak.ci.commoninternet.net +restore(canon) -> ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers'] +restore(live) -> ['warm-advlive_ci_commoninternet_net_data'] +reconciler last_good SURVIVES: 10.7.1+26.6.2 +``` +Each `restore()` returns its OWN stack's volumes — the exact condition. Pre-fix this same script printed one +shared slot and `restore(canon)` raised `SnapshotError`. I additionally probed the guard in **both** +directions (the Builder's script only showed one): foreign *snapshot* REFUSED, foreign *restore* REFUSED. + +**2. Integrity of the round-trip (my addition).** `restore()` is destructive (clears volumes, untars back). +Canon mariadb tar-checksum **byte-identical** before and after: `4271926745 166164480`, 386 files both times. + +**3. Non-vacuity of the 10 new tests — mutation testing (my addition; the Builder asserted 315→325 but not +that the tests bite).** Suites reproduce exactly: `325 passed` @`b5f2b10`, `315 passed` @`07fc6d4`. Then: + * Mutation A — `canonical_ns()` reverted to `return recipe`: **4 failed** (`…_for_live_warm_provider`, + `test_live_and_canonical_slots_are_disjoint`, `test_registry_path_of_live_warm_provider_is_not_the_reconciler_dir`, + `test_prune_stale_keeps_enrolled_provider_canonical_and_reconciler`). + * Mutation B — both `_assert_slot_not_foreign()` call sites removed: **2 failed** + (`test_snapshot_refuses_to_clobber_another_domains_slot`, `test_restore_refuses_a_foreign_slot`). + Tests genuinely exercise the fix; not vacuous. + +**4. All callers migrated (my addition).** Audited every `snapshot(`/`restore(`/`app_dir(`/`snap_dir(` call +site: no bare recipe survives. `warm_reconcile` passes `warmsnap.live_slot(recipe)` (×3, incl. `last_good_path`), +`run_recipe_ci.py:896` passes `canonical.canonical_slot(recipe)`, `seed_canonical` passes `canonical_slot`. + +**5. Zero blast radius on the other canonicals (my addition — the risk this refactor most plausibly created).** +Ran `canonical_ns`/`registry_path` for all 21 enrolled recipes against the REAL warm root, read-only: every +non-provider keeps `ns == recipe` and resolves to its EXISTING `/var/lib/ci-warm//canonical.json` + +`snapshot/`. `registry_path("bluesky-pds")` is character-identical at parent and at the fix. The three with no +registry (bluesky-pds, discourse, mattermost-lts) have no dir on disk at all and never did — unseeded, not +regressed. **No migration needed**, as claimed: `/var/lib/ci-warm/keycloak/` = `last_good` only. + +**6. `prune_stale` invariant now structural (consequence 4).** Scratch root, fake ns only (never real stack +names — `prune_stale` calls `docker volume rm`): `keycloak/` has no `canonical.json` and structurally cannot +gain one, so it is skipped; `pruned: ['canon-fakerecipe']`, reconciler `last_good` survives. + +**7. Enrollment retained, no silent de-enrollment.** `WARM_CANONICAL = True`; `enrolled_recipes()` includes +keycloak at ns `canon-keycloak`. `canonical_domain` is unchanged, so M2's original keycloak evidence (promote +at `warm-canon-keycloak`, live 200 throughout) still stands without a re-run. The two comments asserting the +deployments "can never touch each other" are gone (`grep -c` → 0 in both files) and replaced with accurate text. + +**Consequences 1–3 of F-redfix-4 are resolved:** slots disjoint (1,2), and the outage race (3) is removed with +the shared slot — a promote can no longer replace the reconciler's known-good. + +**Residual B-redfix-5 — accepted as NON-BLOCKING, independently confirmed to PREDATE this work.** The +reconciler's rollback `warmsnap.restore()` sits outside the `try/except` guarding the upgrade, so any restore +failure leaves live keycloak undeployed after `abra.undeploy()`. I verified this is verbatim true at parent +`07fc6d4` (and thus predates the enrollment): F-redfix-4 supplied a *reachable* way to make that restore raise, +and that way is now closed; the structural gap is older and is not part of the clearing condition I published. +Correctly filed rather than silently fixed — the safe-behaviour choice (redeploy last_good without data restore +vs. die loudly) is a real trade-off for a DB-backed app after a forward migration. Not a VETO. + +Node left clean: throwaway volume removed, all scratch removed, real warm root untouched +(`/var/lib/ci-warm/keycloak/` = `last_good` only; no `canon-keycloak/` created), both canon volumes intact, +live `warm-keycloak…/realms/master` → **200** throughout. + +**## VETO CLEARED** (F-redfix-4). M2 keycloak item: **PASS**. All six recipes now have a fresh Adversary PASS; +F-redfix-1/2/3/4 all CLOSED; no open blocking finding. The Builder may re-assert `## DONE`. +JOURNAL not consulted before this verdict.