fix(keycloak): key warm state by stack namespace, not bare recipe (F-redfix-4)

The M2 keycloak enrollment made the canonical collision-free at the DOMAIN layer
(warm-canon-keycloak vs warm-keycloak) but warm STATE stayed keyed by bare recipe:
warmsnap.app_dir("keycloak") resolved both the live-warm reconciler's last_good and
the data-warm canonical's canonical.json + snapshot/ into /var/lib/ci-warm/keycloak/.
snapshot() atomically REPLACES that slot, so the two deployments destroyed each
other's known-good; restore() then raised SnapshotError (fails closed, no cross-stack
data write). Worst case: a sweep promote landing inside the reconciler's
snapshot->wait_healthy window makes its rollback restore() raise after
abra.undeploy(live), leaving the shared OIDC provider undeployed.

Fix: canonical.canonical_ns() is now the single namespace from which BOTH the
canonical's domain and its warm-state slot derive, so they cannot drift apart. A
live-warm provider gets ns "canon-<recipe>": domain warm-canon-keycloak (unchanged)
and slot /var/lib/ci-warm/canon-keycloak/. Every other recipe keeps ns "<recipe>" —
zero on-disk change for the 15 existing canonicals, and no migration on cc-ci
(keycloak's canonical was never seeded: its dir holds only last_good).

- warmsnap: functions take a SLOT, not a recipe; add live_slot(); meta records "slot".
- warmsnap: _assert_slot_not_foreign() refuses to snapshot/restore a slot recorded
  against a different domain -- defence in depth, naming-scheme-independent, fails
  before the destructive swap rather than at the next restore.
- canonical: registry_path/seed_canonical/prune_stale go through canonical_slot().
- prune_stale: the "reconciler dirs are never pruned" invariant is now STRUCTURAL --
  <recipe>/ never gains a canonical.json, so de-enrolling keycloak can no longer
  rmtree the reconciler's last_good (consequence 4).
- warm_reconcile: last_good + snapshot/restore go through warmsnap.live_slot().
- run_recipe_ci: canonical rollback restores from canonical_slot(recipe).
- Correct the two comments that claimed the deployments "can never touch each other".

Tests: 10 new (slot disjointness for every WARM_DOMAINS recipe, slot<->stack 1:1,
registry not in the reconciler dir, prune spares last_good, foreign-slot refusal in
both snapshot and restore). Unit suite 315 -> 325, no regressions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX

runner/harness/canonical.py

@@ -39,19 +39,35 @@ def is_enrolled(recipe: str) -> bool:
     return bool(meta_mod.load(recipe).WARM_CANONICAL)
 
 
-def canonical_domain(recipe: str) -> str:
-    """Stable data-warm domain for the recipe's canonical.
+def canonical_ns(recipe: str) -> str:
+    """The data-warm canonical's NAMESPACE for a recipe — the single source from which BOTH its
+    domain and its warm-state slot are derived, so the two can never drift apart (F-redfix-4).
 
     For a recipe that is ALSO a live-warm provider (in `warm.WARM_DOMAINS` — e.g. keycloak, whose
-    always-on shared OIDC instance lives at `warm-keycloak…`), the data-warm canonical MUST use a
-    DISTINCT domain: otherwise the sweep's promote deploy/teardown at `warm-<recipe>` collides with —
-    and could disrupt — the live shared service that other recipes (lasuite-*/drone) depend on. Give
-    those recipes a collision-free `warm-canon-<recipe>` namespace (a separate stack/domain that can
-    never touch the live provider); every other recipe keeps the plain `warm-<recipe>` scheme
-    (zero blast radius on the 15 existing canonicals)."""
+    always-on shared OIDC instance lives at `warm-keycloak…`), the canonical MUST be namespaced apart
+    from the live provider: otherwise the sweep's promote deploy/teardown at `warm-<recipe>` collides
+    with — and could disrupt — the live shared service that other recipes (lasuite-*/drone) depend on,
+    and (F-redfix-4) the two deployments share one `/var/lib/ci-warm/<recipe>/` snapshot slot, so each
+    destroys the other's known-good. Every other recipe keeps the plain `<recipe>` namespace (zero
+    blast radius on the 15 existing canonicals, whose on-disk layout is unchanged)."""
     if recipe in warm.WARM_DOMAINS:
-        return f"warm-canon-{recipe}.ci.commoninternet.net"
-    return warm.stable_domain(recipe)
+        return f"canon-{recipe}"
+    return recipe
+
+
+def canonical_domain(recipe: str) -> str:
+    """Stable data-warm domain for the recipe's canonical (`warm-<ns>`)."""
+    return warm.stable_domain(canonical_ns(recipe))
+
+
+def canonical_slot(recipe: str) -> str:
+    """The canonical's warm-state slot: `$CCCI_WARM_ROOT/<ns>/` — holding `canonical.json` and the
+    known-good `snapshot/`.
+
+    DISJOINT from `warmsnap.live_slot(recipe)` (the live-warm reconciler's `<recipe>/`, holding its
+    `last_good`) exactly when the recipe is a live-warm provider. Asserted in
+    `tests/unit/test_canonical.py::test_live_and_canonical_slots_are_disjoint`."""
+    return canonical_ns(recipe)
 
 
 def enrolled_recipes() -> list[str]:
@@ -71,7 +87,7 @@ def enrolled_recipes() -> list[str]:
 
 
 def registry_path(recipe: str) -> str:
-    return os.path.join(warmsnap.app_dir(recipe), "canonical.json")
+    return os.path.join(warmsnap.app_dir(canonical_slot(recipe)), "canonical.json")
 
 
 def read_registry(recipe: str) -> dict | None:
@@ -84,7 +100,7 @@ def read_registry(recipe: str) -> dict | None:
 
 def write_registry(recipe: str, *, version: str, commit: str | None, status: str) -> dict:
     """Atomically write the canonical registry record for a recipe."""
-    os.makedirs(warmsnap.app_dir(recipe), exist_ok=True)
+    os.makedirs(warmsnap.app_dir(canonical_slot(recipe)), exist_ok=True)
     rec = {
         "recipe": recipe,
         "domain": canonical_domain(recipe),
@@ -151,16 +167,22 @@ def undeploy_keep_volume(recipe: str) -> None:
 
 
 def prune_stale() -> list[str]:
-    """WC8 disk hygiene: remove warm data for DE-ENROLLED canonicals — a `/var/lib/ci-warm/<recipe>/`
-    that carries a `canonical.json` but whose recipe is no longer enrolled (WARM_CANONICAL dropped).
-    Drops the dir (snapshot + registry) AND the retained `warm-<recipe>` data volumes. Leaves the
-    live-warm reconciler dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`),
-    `alerts/`, and currently-enrolled canonicals untouched. Returns the recipes pruned."""
+    """WC8 disk hygiene: remove warm data for DE-ENROLLED canonicals — a `/var/lib/ci-warm/<ns>/`
+    that carries a `canonical.json` but whose namespace is no longer an enrolled canonical's slot
+    (WARM_CANONICAL dropped). Drops the dir (snapshot + registry) AND the retained `warm-<ns>` data
+    volumes. Leaves the live-warm reconciler dirs (keycloak/traefik — they have a `last_good`, no
+    `canonical.json`), `alerts/`, and currently-enrolled canonicals untouched. Returns the namespaces
+    pruned.
+
+    The reconciler-dir invariant is STRUCTURAL, not incidental (F-redfix-4): a live-warm provider's
+    canonical registers under `canon-<recipe>/`, so `<recipe>/` never gains a `canonical.json` and is
+    never a prune candidate — de-enrolling keycloak can no longer `rmtree` the reconciler's
+    `last_good`. Dir name == namespace, so the stale stack's domain is `warm.stable_domain(name)`."""
     import shutil
     import subprocess
 
     root = warmsnap.warm_root()
-    keep = set(enrolled_recipes())
+    keep = {canonical_slot(r) for r in enrolled_recipes()}
     pruned: list[str] = []
     try:
         entries = sorted(os.listdir(root))
@@ -172,8 +194,8 @@ def prune_stale() -> list[str]:
             continue
         if not os.path.isfile(os.path.join(d, "canonical.json")):
             continue  # not a data-warm canonical (e.g. keycloak/traefik reconciler dir, alerts/)
-        # drop the retained warm-<recipe> volumes, then the snapshot/registry dir
-        for vol in warmsnap.stack_volumes(canonical_domain(name)):
+        # drop the retained warm-<ns> volumes, then the snapshot/registry dir
+        for vol in warmsnap.stack_volumes(warm.stable_domain(name)):
             subprocess.run(["docker", "volume", "rm", vol], capture_output=True, text=True)
         shutil.rmtree(d, ignore_errors=True)
         pruned.append(name)
@@ -186,5 +208,7 @@ def seed_canonical(recipe: str, version: str, commit: str | None = None) -> dict
     healthy first, then undeploys before calling this (WC3: snapshot while undeployed). The retained
     volume IS the canonical. Returns the registry record."""
     rec = write_registry(recipe, version=version, commit=commit, status="idle")
-    warmsnap.snapshot(recipe, canonical_domain(recipe), commit=commit, version=version)
+    warmsnap.snapshot(
+        canonical_slot(recipe), canonical_domain(recipe), commit=commit, version=version
+    )
     return rec