From 397c3031888f6b9212b044d26a7343896c9c6f47 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Thu, 9 Jul 2026 00:13:47 +0000 Subject: [PATCH] =?UTF-8?q?journal(redfix):=20wake=20#8=20=E2=80=94=20VETO?= =?UTF-8?q?=20F-redfix-4=20reproduced=20independently,=20root-caused,=20fi?= =?UTF-8?q?xed=20at=20b5f2b10;=20decision=20recorded=20(warm=20state=20key?= =?UTF-8?q?ed=20by=20stack=20namespace);=20B-redfix-5=20residual=20filed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01FS8p1esg57UAC69riNvuBX --- machine-docs/BACKLOG-redfix.md | 19 ++++++++ machine-docs/DECISIONS.md | 26 +++++++++++ machine-docs/JOURNAL-redfix.md | 82 ++++++++++++++++++++++++++++++++++ 3 files changed, 127 insertions(+) diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index 1269e99..6915d0b 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -51,6 +51,25 @@ hold). Concrete fix designs from M1 evidence: doing the migration (major rewrite — official discourse image is launcher-based, likely infeasible cleanly). Lean (a)+tracked-upstream; may need operator input (DEFERRED?) — assess in M2. +### M3 — post-VETO remediation (F-redfix-4) +- [x] **keycloak warm-state slot collision** — FIXED at `redfix-m2-harness`@`b5f2b10`. `canonical_ns()` is + now the one namespace behind both the canonical's domain and its warm-state slot; live-warm provider + → `canon-` slot, disjoint from the reconciler's `/`. Plus a naming-independent + `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci + (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per + STATUS-redfix.md "Gate: M2 RE-CLAIMED". +- [ ] **B-redfix-5 — reconciler rollback `restore()` is outside the upgrade's `try/except`** (NOT blocking; + NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In + `warm_reconcile.py`, the unhealthy-rollback path runs `abra.undeploy(domain)` → `wait_undeployed` → + `warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits outside the `try/except` that + guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the + exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**. + F-redfix-4 supplied one way to make `restore()` raise (the shared slot) and that is now fixed, but the + structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys + `last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a + breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app + after a forward migration — that trade-off is why this is filed, not fixed inline. + ## Adversary findings (Adversary-owned — do not edit.) diff --git a/machine-docs/DECISIONS.md b/machine-docs/DECISIONS.md index fd0f7cf..909b22b 100644 --- a/machine-docs/DECISIONS.md +++ b/machine-docs/DECISIONS.md @@ -1609,3 +1609,29 @@ trim job observed → adequate; revisit only if a cap is ever needed. `dashboard/dashboard.py` + `tests/unit/test_dashboard.py` would be reformatted — confirmed pre-existing at HEAD f68f1c5, outside the settings diff. Flagged for the dashboard owner / orchestrator; not fixed here (narrow scope). + +## Warm state is keyed by STACK NAMESPACE, not by recipe (phase redfix, F-redfix-4, 2026-07-09) + +**Settled.** `/var/lib/ci-warm//` is a slot owned by exactly ONE deployed stack. `ns` comes from +`canonical.canonical_ns(recipe)` for the data-warm canonical and `warmsnap.live_slot(recipe)` for the +live-warm reconciler. A recipe that is both (only `keycloak` today, via `warm.WARM_DOMAINS`) gets +`canon-` for its canonical, so the two never share a slot. The canonical's **domain and its slot +derive from the same `canonical_ns()`**, so they cannot drift apart — that coupling is the invariant, not +the `canon-` string. Every slot `` maps 1:1 to the stack at `warm.stable_domain()`. + +Rejected alternatives: +- **Skip `seed_canonical` for `WARM_DOMAINS` recipes.** Silently de-enrolls keycloak and re-opens the DoD + item ("keycloak enrolled … verified green"). The Adversary named this as not acceptable; agreed. +- **Key every slot by the full domain / stack name** (the "purest" model). Correct, but it relocates the + reconciler's `last_good` for keycloak+traefik and `canonical.json` + `snapshot/` for 15 existing + canonicals on a live node, requiring a migration shim for a phase whose mandate is to fix red, not + re-architect. The chosen scheme is the same model applied only where two stacks actually contend, and + needs no migration (keycloak's canonical had never been seeded). +- **Guard-only (`_assert_slot_not_foreign` alone).** Turns silent destruction into a loud failure but leaves + keycloak's canonical unable to ever seed. Kept as defence in depth BEHIND the slot separation, not as the + fix. + +Consequence for future enrollments: adding a recipe to `warm.WARM_DOMAINS` automatically namespaces its +canonical on both axes. `prune_stale()`'s "reconciler dirs are never pruned" invariant is now structural +(a `/` dir never gains a `canonical.json`) rather than an incidental property that de-enrollment +could falsify. diff --git a/machine-docs/JOURNAL-redfix.md b/machine-docs/JOURNAL-redfix.md index 128a540..72ef73c 100644 --- a/machine-docs/JOURNAL-redfix.md +++ b/machine-docs/JOURNAL-redfix.md @@ -794,3 +794,85 @@ appears nowhere in git history on any ref. I surface it rather than delete it. I cannot affect a cold clone or any Adversary verification. No action. DONE stands, no VETO. Loop stopped. + +--- + +## Wake #8 — 2026-07-09 — VETO F-redfix-4 received, reproduced, fixed, M2 re-claimed + +Woken by a watchdog ping on a `review(...)` commit. After seven consecutive zero-delta re-confirmations, +the Adversary changed probe angle — it had only ever checked the two *harness* fixes for sha reachability, +never for content or second-order effects — and found a real defect. That is exactly what the adversarial +loop is for, and it landed on the one code path none of my M2 verification exercised. + +**Why my M2 verification missed it.** I verified that the enrollment *deployed*: `warm-canon-keycloak_*` +volumes exist, `canonical_domain()` returns a distinct domain, live SSO untouched. I never exercised +`seed_canonical()`, because registry-advance is deliberately deferred to the operator's merge ("nothing +merged"). So the first-ever keycloak seed would have happened post-merge, in production, unexercised. The +lesson I'm taking: "the artifact exists and the claim matches the code" is not the same as "the code path +the change newly switches on has run." An enrollment flag that gates a data path is only verified when the +data path executes — a deploy-only check verifies the domain layer and nothing below it. + +**Order of work.** I withdrew `## DONE` before doing anything else. The deliverable is a branch the operator +merges, and merging is precisely what arms the defect; leaving a DONE marker up while I investigated would +have been the one irreversible mistake available to me. Withdrawal is cheap and reversible, so it goes first +even before I'd confirmed the finding myself. + +**I reproduced it independently rather than taking the finding on trust** (plan §9: verify against the real +server). Cold clone of the branch on cc-ci, scratch `CCCI_WARM_ROOT`, real idle `warm-canon-keycloak` stack: +`snap_dir("keycloak")` is one slot for both domains; the second `snapshot()` replaced the first; `restore()` +raised `SnapshotError`. Confirmed. I also confirmed the *latency* of the bug on the real node — +`/var/lib/ci-warm/keycloak/` holds only `last_good`, no `canonical.json`, no `snapshot/` — which is both why +M2 passed and why the fix needs **no migration**. + +**Design.** The tempting minimal fix is a `WARM_DOMAINS` skip-guard on `seed_canonical`. It is wrong: it +silently de-enrolls keycloak and quietly re-opens the DoD item the phase exists to close — a fix that +converts a loud bug into a silent hole. The Adversary pre-emptively ruled it out and I agree. + +The purest fix is to key every slot by its stack name. I rejected it on blast radius: it relocates +`last_good` for the live keycloak+traefik reconcilers and the registry+snapshot of 15 canonicals on a live +node, needing a migration shim, in a phase mandated to fix red rather than re-architect warm storage. + +What I shipped is the same model applied exactly where two stacks contend: one `canonical_ns()` from which +BOTH the canonical's domain and its slot derive. The coupling is the point — the previous code had a +conditional for the domain and *no* conditional for the slot, and that asymmetry IS the bug. Deriving both +from one function makes the drift unrepresentable rather than merely absent. Every existing canonical keeps +ns ``, so nothing on disk moves. + +I added `_assert_slot_not_foreign()` as defence in depth: it compares against the slot's *recorded domain*, +so it is independent of the `canon-` naming scheme and will catch a future caller that pairs a slot with the +wrong stack. Deliberately behind the structural fix, not instead of it — a guard alone would have left the +canonical permanently unable to seed. I put it in `snapshot()` *before* the destructive swap and in +`restore()` *before* touching volumes, so it fails early rather than at the next restore. + +**Scope discipline.** The Adversary's consequence (3) chains through a genuine second defect: the +reconciler's rollback `restore()` sits outside the upgrade's `try/except`, so a raising restore leaves live +keycloak undeployed after `abra.undeploy()`. Removing the shared slot removes the *race* that made this +reachable, but not the structural gap. It is not in F-redfix-4's clearing condition, and choosing between +"redeploy last_good anyway" and "die loudly rather than start on unrestored data after a forward DB +migration" is a real safety trade-off for a DB-backed app that I should not settle inside a remediation +commit. Filed as B-redfix-5 in BACKLOG-redfix.md and named in STATUS so it is visibly deferred, not dropped. + +**Verification.** Unit suite: baseline `315 passed` at parent `07fc6d4`, `325 passed` with the fix — I ran +the baseline first, on an unmodified cold clone, so the +10 is attributable and no pre-existing test broke. +Two early full-suite runs failed on `test_dashboard.py` / `test_bridge_trigger.py` / `test_meta.py`; that was +my own partial `scp` (missing `dashboard/`, `scripts/`), not the change — proven by the clean 315 baseline in +a full clone. Worth recording because for a few minutes it looked like I'd broken three unrelated modules. + +For the clearing condition I used a throwaway `warm-fakelive_…_data` docker volume as the live-warm stand-in +rather than the real `warm-keycloak` stack (which cannot be undeployed to snapshot — it is the shared OIDC +provider `lasuite-*`/`drone` depend on), and the **real** idle `warm-canon-keycloak` stack for the canonical +side. `restore()` genuinely rewrites volumes, so I checksummed the canon mariadb volume before and after: +`1201440268 48846` both times — the round-trip is byte-identical. Live `warm-keycloak…/realms/master` +returned 200 throughout. Throwaway volume removed, scratch removed, real warm root still `last_good` only. + +**Incidental find.** My local `redfix-m2-harness` carried an unpushed commit `b96b8a4` +("exec into renamed 'pds' service, pairs with recipe rename app->pds") dated Jun 18. `origin` was at +`07fc6d4`, matching the Adversary's pin, so nothing shipped. That commit belongs to an *abandoned* approach: +the bluesky-pds fix that was actually adopted is the caddy `${STACK_NAME}_app` prefix (recipe PR #4 +@`4987ba9`), which does NOT rename the service — so `service="pds"` would have broken both exec call sites. +Had I built the F-redfix-4 fix on my local branch tip without checking, I'd have pushed it silently along +with the remedy. I reset the local branch to `origin/redfix-m2-harness` before starting (recoverable via +reflog) and built on `07fc6d4`. Check `git rev-parse origin/` before building on a +long-lived branch you last touched three weeks ago. + +`## DONE` is NOT restored. M2 is re-claimed and awaits an Adversary verdict on `b5f2b10`.