journal(redfix): wake #8 — VETO F-redfix-4 reproduced independently, root-caused, fixed at b5f2b10; decision recorded (warm state keyed by stack namespace); B-redfix-5 residual filed

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX

machine-docs/JOURNAL-redfix.md

@@ -794,3 +794,85 @@ appears nowhere in git history on any ref. I surface it rather than delete it. I
 cannot affect a cold clone or any Adversary verification.
 
 No action. DONE stands, no VETO. Loop stopped.
+
+---
+
+## Wake #8 — 2026-07-09 — VETO F-redfix-4 received, reproduced, fixed, M2 re-claimed
+
+Woken by a watchdog ping on a `review(...)` commit. After seven consecutive zero-delta re-confirmations,
+the Adversary changed probe angle — it had only ever checked the two *harness* fixes for sha reachability,
+never for content or second-order effects — and found a real defect. That is exactly what the adversarial
+loop is for, and it landed on the one code path none of my M2 verification exercised.
+
+**Why my M2 verification missed it.** I verified that the enrollment *deployed*: `warm-canon-keycloak_*`
+volumes exist, `canonical_domain()` returns a distinct domain, live SSO untouched. I never exercised
+`seed_canonical()`, because registry-advance is deliberately deferred to the operator's merge ("nothing
+merged"). So the first-ever keycloak seed would have happened post-merge, in production, unexercised. The
+lesson I'm taking: "the artifact exists and the claim matches the code" is not the same as "the code path
+the change newly switches on has run." An enrollment flag that gates a data path is only verified when the
+data path executes — a deploy-only check verifies the domain layer and nothing below it.
+
+**Order of work.** I withdrew `## DONE` before doing anything else. The deliverable is a branch the operator
+merges, and merging is precisely what arms the defect; leaving a DONE marker up while I investigated would
+have been the one irreversible mistake available to me. Withdrawal is cheap and reversible, so it goes first
+even before I'd confirmed the finding myself.
+
+**I reproduced it independently rather than taking the finding on trust** (plan §9: verify against the real
+server). Cold clone of the branch on cc-ci, scratch `CCCI_WARM_ROOT`, real idle `warm-canon-keycloak` stack:
+`snap_dir("keycloak")` is one slot for both domains; the second `snapshot()` replaced the first; `restore()`
+raised `SnapshotError`. Confirmed. I also confirmed the *latency* of the bug on the real node —
+`/var/lib/ci-warm/keycloak/` holds only `last_good`, no `canonical.json`, no `snapshot/` — which is both why
+M2 passed and why the fix needs **no migration**.
+
+**Design.** The tempting minimal fix is a `WARM_DOMAINS` skip-guard on `seed_canonical`. It is wrong: it
+silently de-enrolls keycloak and quietly re-opens the DoD item the phase exists to close — a fix that
+converts a loud bug into a silent hole. The Adversary pre-emptively ruled it out and I agree.
+
+The purest fix is to key every slot by its stack name. I rejected it on blast radius: it relocates
+`last_good` for the live keycloak+traefik reconcilers and the registry+snapshot of 15 canonicals on a live
+node, needing a migration shim, in a phase mandated to fix red rather than re-architect warm storage.
+
+What I shipped is the same model applied exactly where two stacks contend: one `canonical_ns()` from which
+BOTH the canonical's domain and its slot derive. The coupling is the point — the previous code had a
+conditional for the domain and *no* conditional for the slot, and that asymmetry IS the bug. Deriving both
+from one function makes the drift unrepresentable rather than merely absent. Every existing canonical keeps
+ns `<recipe>`, so nothing on disk moves.
+
+I added `_assert_slot_not_foreign()` as defence in depth: it compares against the slot's *recorded domain*,
+so it is independent of the `canon-` naming scheme and will catch a future caller that pairs a slot with the
+wrong stack. Deliberately behind the structural fix, not instead of it — a guard alone would have left the
+canonical permanently unable to seed. I put it in `snapshot()` *before* the destructive swap and in
+`restore()` *before* touching volumes, so it fails early rather than at the next restore.
+
+**Scope discipline.** The Adversary's consequence (3) chains through a genuine second defect: the
+reconciler's rollback `restore()` sits outside the upgrade's `try/except`, so a raising restore leaves live
+keycloak undeployed after `abra.undeploy()`. Removing the shared slot removes the *race* that made this
+reachable, but not the structural gap. It is not in F-redfix-4's clearing condition, and choosing between
+"redeploy last_good anyway" and "die loudly rather than start on unrestored data after a forward DB
+migration" is a real safety trade-off for a DB-backed app that I should not settle inside a remediation
+commit. Filed as B-redfix-5 in BACKLOG-redfix.md and named in STATUS so it is visibly deferred, not dropped.
+
+**Verification.** Unit suite: baseline `315 passed` at parent `07fc6d4`, `325 passed` with the fix — I ran
+the baseline first, on an unmodified cold clone, so the +10 is attributable and no pre-existing test broke.
+Two early full-suite runs failed on `test_dashboard.py` / `test_bridge_trigger.py` / `test_meta.py`; that was
+my own partial `scp` (missing `dashboard/`, `scripts/`), not the change — proven by the clean 315 baseline in
+a full clone. Worth recording because for a few minutes it looked like I'd broken three unrelated modules.
+
+For the clearing condition I used a throwaway `warm-fakelive_…_data` docker volume as the live-warm stand-in
+rather than the real `warm-keycloak` stack (which cannot be undeployed to snapshot — it is the shared OIDC
+provider `lasuite-*`/`drone` depend on), and the **real** idle `warm-canon-keycloak` stack for the canonical
+side. `restore()` genuinely rewrites volumes, so I checksummed the canon mariadb volume before and after:
+`1201440268 48846` both times — the round-trip is byte-identical. Live `warm-keycloak…/realms/master`
+returned 200 throughout. Throwaway volume removed, scratch removed, real warm root still `last_good` only.
+
+**Incidental find.** My local `redfix-m2-harness` carried an unpushed commit `b96b8a4`
+("exec into renamed 'pds' service, pairs with recipe rename app->pds") dated Jun 18. `origin` was at
+`07fc6d4`, matching the Adversary's pin, so nothing shipped. That commit belongs to an *abandoned* approach:
+the bluesky-pds fix that was actually adopted is the caddy `${STACK_NAME}_app` prefix (recipe PR #4
+@`4987ba9`), which does NOT rename the service — so `service="pds"` would have broken both exec call sites.
+Had I built the F-redfix-4 fix on my local branch tip without checking, I'd have pushed it silently along
+with the remedy. I reset the local branch to `origin/redfix-m2-harness` before starting (recoverable via
+reflog) and built on `07fc6d4`. Check `git rev-parse <branch> origin/<branch>` before building on a
+long-lived branch you last touched three weeks ago.
+
+`## DONE` is NOT restored. M2 is re-claimed and awaits an Adversary verdict on `b5f2b10`.