review(redfix): VETO — keycloak enrollment not collision-free in warm state (F-redfix-4)

Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously
only sha-reachability checked, never content, never second-order effects).

mumble 07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged,
so a dead server still FAILs. Budget widened without weakening the test.

keycloak 61211db: correct at the domain/stack layer, but its own invariant is FALSE.
canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by
recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler
(warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical
(warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard).

Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack:
snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and
restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but:
 1. every stateful reconciler upgrade deterministically destroys the canonical snapshot;
 2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore
    window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes
    the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of
    the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's
    de-enrollment exception existed to prevent.
 3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded.

Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json,
no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran,
registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised.

VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/
mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint
warm-state paths and each restore() returns its own stack's volumes.

Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout.

machine-docs/BACKLOG-redfix.md

@@ -55,6 +55,107 @@ hold). Concrete fix designs from M1 evidence:
 
 (Adversary-owned — do not edit.)
 
+### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **OPEN, BLOCKING (VETO)** @2026-07-08T23:56Z
+
+**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
+`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
+depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
+resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.
+
+**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
+any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
+`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:
+
+```
+canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
+WARM_DOMAINS[keycloak]     = warm-keycloak.ci.commoninternet.net
+stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
+stack_volumes(LIVE)  = ['warm-keycloak_..._mariadb',       'warm-keycloak_..._providers']
+```
+
+**The defect.** Warm *state* is keyed by RECIPE, not by domain:
+`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
+`canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
+So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
+holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:
+
+| | producer | consumer |
+|---|---|---|
+| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
+| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |
+
+`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
+and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
+so restore raises `SnapshotError` instead of cross-writing data.
+
+Consequences, in descending certainty:
+
+1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
+   overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
+   then raises `SnapshotError` and cannot roll back.
+2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
+3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
+   and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
+   `promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
+   then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
+   OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
+   never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
+4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
+   dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
+   *has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
+   `shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.
+
+**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
+`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
+canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
+*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
+The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.
+
+**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
+touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
+impossible." Both are false for warm state. That claim is what I falsified.
+
+**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
+Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
+(the live one cannot be undeployed to snapshot it):
+
+```sh
+ssh cc-ci
+git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
+CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
+import sys; sys.path.insert(0, ".")
+from harness import warmsnap as ws, canonical as c
+CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
+print(ws.snap_dir("keycloak"))                       # one slot, domain-blind
+ws.snapshot("keycloak", CANON,   version="canon-known-good")
+ws.snapshot("keycloak", STANDIN, version="live-last-good")   # clobbers it
+print(ws.read_meta("keycloak")["domain"])            # -> warm-custom-html…  (canon known-good GONE)
+ws.restore("keycloak", CANON)                        # -> SnapshotError
+PY
+```
+
+EXPECTED (observed @2026-07-08T23:55Z):
+```
+/tmp/p8w/keycloak/snapshot                       <- SAME slot for BOTH domains
+read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net   <- canon known-good DESTROYED
+SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
+                 absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
+```
+Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
+volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.
+
+**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
+rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
+seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
+touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
+de-enroll keycloak and re-open the DoD item, so it is not sufficient.
+
+**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
+survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
+seeing each `restore()` return its OWN stack's volumes.
+
+---
+
 ### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**
 
 **CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the