journal(redfix): wake #8 — VETO F-redfix-4 reproduced independently, root-caused, fixed at b5f2b10; decision recorded (warm state keyed by stack namespace); B-redfix-5 residual filed

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX

machine-docs/BACKLOG-redfix.md

@@ -51,6 +51,25 @@ hold). Concrete fix designs from M1 evidence:
       doing the migration (major rewrite — official discourse image is launcher-based, likely
       infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2.
 
+### M3 — post-VETO remediation (F-redfix-4)
+- [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is
+      now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider
+      → `canon-<recipe>` slot, disjoint from the reconciler's `<recipe>/`. Plus a naming-independent
+      `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
+      (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
+      STATUS-redfix.md "Gate: M2 RE-CLAIMED".
+- [ ] **B-redfix-5 — reconciler rollback `restore()` is outside the upgrade's `try/except`** (NOT blocking;
+      NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In
+      `warm_reconcile.py`, the unhealthy-rollback path runs `abra.undeploy(domain)` → `wait_undeployed` →
+      `warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits outside the `try/except` that
+      guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the
+      exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**.
+      F-redfix-4 supplied one way to make `restore()` raise (the shared slot) and that is now fixed, but the
+      structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys
+      `last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a
+      breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app
+      after a forward migration — that trade-off is why this is filed, not fixed inline.
+
 ## Adversary findings
 
 (Adversary-owned — do not edit.)

machine-docs/DECISIONS.md

@@ -1609,3 +1609,29 @@ trim job observed → adequate; revisit only if a cap is ever needed.
   `dashboard/dashboard.py` + `tests/unit/test_dashboard.py` would be reformatted — confirmed pre-existing
   at HEAD f68f1c5, outside the settings diff. Flagged for the dashboard owner / orchestrator; not fixed
   here (narrow scope).
+
+## Warm state is keyed by STACK NAMESPACE, not by recipe (phase redfix, F-redfix-4, 2026-07-09)
+
+**Settled.** `/var/lib/ci-warm/<ns>/` is a slot owned by exactly ONE deployed stack. `ns` comes from
+`canonical.canonical_ns(recipe)` for the data-warm canonical and `warmsnap.live_slot(recipe)` for the
+live-warm reconciler. A recipe that is both (only `keycloak` today, via `warm.WARM_DOMAINS`) gets
+`canon-<recipe>` for its canonical, so the two never share a slot. The canonical's **domain and its slot
+derive from the same `canonical_ns()`**, so they cannot drift apart — that coupling is the invariant, not
+the `canon-` string. Every slot `<ns>` maps 1:1 to the stack at `warm.stable_domain(<ns>)`.
+
+Rejected alternatives:
+- **Skip `seed_canonical` for `WARM_DOMAINS` recipes.** Silently de-enrolls keycloak and re-opens the DoD
+  item ("keycloak enrolled … verified green"). The Adversary named this as not acceptable; agreed.
+- **Key every slot by the full domain / stack name** (the "purest" model). Correct, but it relocates the
+  reconciler's `last_good` for keycloak+traefik and `canonical.json` + `snapshot/` for 15 existing
+  canonicals on a live node, requiring a migration shim for a phase whose mandate is to fix red, not
+  re-architect. The chosen scheme is the same model applied only where two stacks actually contend, and
+  needs no migration (keycloak's canonical had never been seeded).
+- **Guard-only (`_assert_slot_not_foreign` alone).** Turns silent destruction into a loud failure but leaves
+  keycloak's canonical unable to ever seed. Kept as defence in depth BEHIND the slot separation, not as the
+  fix.
+
+Consequence for future enrollments: adding a recipe to `warm.WARM_DOMAINS` automatically namespaces its
+canonical on both axes. `prune_stale()`'s "reconciler dirs are never pruned" invariant is now structural
+(a `<recipe>/` dir never gains a `canonical.json`) rather than an incidental property that de-enrollment
+could falsify.

machine-docs/JOURNAL-redfix.md

@@ -794,3 +794,85 @@ appears nowhere in git history on any ref. I surface it rather than delete it. I
 cannot affect a cold clone or any Adversary verification.
 
 No action. DONE stands, no VETO. Loop stopped.
+
+---
+
+## Wake #8 — 2026-07-09 — VETO F-redfix-4 received, reproduced, fixed, M2 re-claimed
+
+Woken by a watchdog ping on a `review(...)` commit. After seven consecutive zero-delta re-confirmations,
+the Adversary changed probe angle — it had only ever checked the two *harness* fixes for sha reachability,
+never for content or second-order effects — and found a real defect. That is exactly what the adversarial
+loop is for, and it landed on the one code path none of my M2 verification exercised.
+
+**Why my M2 verification missed it.** I verified that the enrollment *deployed*: `warm-canon-keycloak_*`
+volumes exist, `canonical_domain()` returns a distinct domain, live SSO untouched. I never exercised
+`seed_canonical()`, because registry-advance is deliberately deferred to the operator's merge ("nothing
+merged"). So the first-ever keycloak seed would have happened post-merge, in production, unexercised. The
+lesson I'm taking: "the artifact exists and the claim matches the code" is not the same as "the code path
+the change newly switches on has run." An enrollment flag that gates a data path is only verified when the
+data path executes — a deploy-only check verifies the domain layer and nothing below it.
+
+**Order of work.** I withdrew `## DONE` before doing anything else. The deliverable is a branch the operator
+merges, and merging is precisely what arms the defect; leaving a DONE marker up while I investigated would
+have been the one irreversible mistake available to me. Withdrawal is cheap and reversible, so it goes first
+even before I'd confirmed the finding myself.
+
+**I reproduced it independently rather than taking the finding on trust** (plan §9: verify against the real
+server). Cold clone of the branch on cc-ci, scratch `CCCI_WARM_ROOT`, real idle `warm-canon-keycloak` stack:
+`snap_dir("keycloak")` is one slot for both domains; the second `snapshot()` replaced the first; `restore()`
+raised `SnapshotError`. Confirmed. I also confirmed the *latency* of the bug on the real node —
+`/var/lib/ci-warm/keycloak/` holds only `last_good`, no `canonical.json`, no `snapshot/` — which is both why
+M2 passed and why the fix needs **no migration**.
+
+**Design.** The tempting minimal fix is a `WARM_DOMAINS` skip-guard on `seed_canonical`. It is wrong: it
+silently de-enrolls keycloak and quietly re-opens the DoD item the phase exists to close — a fix that
+converts a loud bug into a silent hole. The Adversary pre-emptively ruled it out and I agree.
+
+The purest fix is to key every slot by its stack name. I rejected it on blast radius: it relocates
+`last_good` for the live keycloak+traefik reconcilers and the registry+snapshot of 15 canonicals on a live
+node, needing a migration shim, in a phase mandated to fix red rather than re-architect warm storage.
+
+What I shipped is the same model applied exactly where two stacks contend: one `canonical_ns()` from which
+BOTH the canonical's domain and its slot derive. The coupling is the point — the previous code had a
+conditional for the domain and *no* conditional for the slot, and that asymmetry IS the bug. Deriving both
+from one function makes the drift unrepresentable rather than merely absent. Every existing canonical keeps
+ns `<recipe>`, so nothing on disk moves.
+
+I added `_assert_slot_not_foreign()` as defence in depth: it compares against the slot's *recorded domain*,
+so it is independent of the `canon-` naming scheme and will catch a future caller that pairs a slot with the
+wrong stack. Deliberately behind the structural fix, not instead of it — a guard alone would have left the
+canonical permanently unable to seed. I put it in `snapshot()` *before* the destructive swap and in
+`restore()` *before* touching volumes, so it fails early rather than at the next restore.
+
+**Scope discipline.** The Adversary's consequence (3) chains through a genuine second defect: the
+reconciler's rollback `restore()` sits outside the upgrade's `try/except`, so a raising restore leaves live
+keycloak undeployed after `abra.undeploy()`. Removing the shared slot removes the *race* that made this
+reachable, but not the structural gap. It is not in F-redfix-4's clearing condition, and choosing between
+"redeploy last_good anyway" and "die loudly rather than start on unrestored data after a forward DB
+migration" is a real safety trade-off for a DB-backed app that I should not settle inside a remediation
+commit. Filed as B-redfix-5 in BACKLOG-redfix.md and named in STATUS so it is visibly deferred, not dropped.
+
+**Verification.** Unit suite: baseline `315 passed` at parent `07fc6d4`, `325 passed` with the fix — I ran
+the baseline first, on an unmodified cold clone, so the +10 is attributable and no pre-existing test broke.
+Two early full-suite runs failed on `test_dashboard.py` / `test_bridge_trigger.py` / `test_meta.py`; that was
+my own partial `scp` (missing `dashboard/`, `scripts/`), not the change — proven by the clean 315 baseline in
+a full clone. Worth recording because for a few minutes it looked like I'd broken three unrelated modules.
+
+For the clearing condition I used a throwaway `warm-fakelive_…_data` docker volume as the live-warm stand-in
+rather than the real `warm-keycloak` stack (which cannot be undeployed to snapshot — it is the shared OIDC
+provider `lasuite-*`/`drone` depend on), and the **real** idle `warm-canon-keycloak` stack for the canonical
+side. `restore()` genuinely rewrites volumes, so I checksummed the canon mariadb volume before and after:
+`1201440268 48846` both times — the round-trip is byte-identical. Live `warm-keycloak…/realms/master`
+returned 200 throughout. Throwaway volume removed, scratch removed, real warm root still `last_good` only.
+
+**Incidental find.** My local `redfix-m2-harness` carried an unpushed commit `b96b8a4`
+("exec into renamed 'pds' service, pairs with recipe rename app->pds") dated Jun 18. `origin` was at
+`07fc6d4`, matching the Adversary's pin, so nothing shipped. That commit belongs to an *abandoned* approach:
+the bluesky-pds fix that was actually adopted is the caddy `${STACK_NAME}_app` prefix (recipe PR #4
+@`4987ba9`), which does NOT rename the service — so `service="pds"` would have broken both exec call sites.
+Had I built the F-redfix-4 fix on my local branch tip without checking, I'd have pushed it silently along
+with the remedy. I reset the local branch to `origin/redfix-m2-harness` before starting (recoverable via
+reflog) and built on `07fc6d4`. Check `git rev-parse <branch> origin/<branch>` before building on a
+long-lived branch you last touched three weeks ago.
+
+`## DONE` is NOT restored. M2 is re-claimed and awaits an Adversary verdict on `b5f2b10`.