review(redfix): VETO — keycloak enrollment not collision-free in warm state (F-redfix-4)
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously only sha-reachability checked, never content, never second-order effects). mumble07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged, so a dead server still FAILs. Budget widened without weakening the test. keycloak61211db: correct at the domain/stack layer, but its own invariant is FALSE. canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler (warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical (warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard). Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack: snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but: 1. every stateful reconciler upgrade deterministically destroys the canonical snapshot; 2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's de-enrollment exception existed to prevent. 3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded. Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json, no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran, registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised. VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/ mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint warm-state paths and each restore() returns its own stack's volumes. Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout.
This commit is contained in:
@ -503,3 +503,69 @@ fixes are now content-anchored, not merely sha-anchored.
|
||||
|
||||
No new finding. No gate reopened. All findings CLOSED (F-redfix-1, F-redfix-2, F-redfix-3).
|
||||
Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.
|
||||
|
||||
### Post-reboot re-confirmation #8 @2026-07-08T23:56Z — **VETO RAISED** (new finding F-redfix-4)
|
||||
|
||||
Pulled: zero commit delta since #7. `## DONE` still in STATUS-redfix.md; no `ADVERSARY-INBOX.md`.
|
||||
F-redfix-1/2/3 remain CLOSED. This wake I ran a **new** break-it probe and it found a real defect.
|
||||
|
||||
**Probe angle (new).** #7 content-verified the three *recipe* fixes against their parents. The two
|
||||
*harness* fixes on `redfix-m2-harness` had only ever been checked for sha reachability (#6), never for
|
||||
content, and never for second-order effects. So I cold-cloned the harness repo and diffed both against
|
||||
their parents:
|
||||
|
||||
* **mumble `07fc6d4`** — CLEAN. `tests/mumble/custom/test_protocol_handshake.py`: `retry_handshake(attempts=12)`
|
||||
→ `attempts=36` at `interval=5.0` = the claimed 180s budget. Diff is +8/-1: the single retry line plus a
|
||||
comment. **Every assertion below it is unchanged**, so a genuinely dead server still exhausts retries and
|
||||
FAILs — the budget was widened without weakening the test. Claim matches code. ✔
|
||||
* **keycloak `61211db`** — `canonical_domain()` routes `WARM_DOMAINS` recipes to `warm-canon-<recipe>`, and
|
||||
`recipe_meta.WARM_CANONICAL` flips False→True. At the **domain/stack layer the fix is real and correct**
|
||||
(verified live on cc-ci: four volumes, two disjoint stack prefixes). **But its own stated invariant is false.**
|
||||
|
||||
**F-redfix-4 (filed in BACKLOG-redfix.md, full repro there).** The fix separates the two keycloak deployments
|
||||
by domain, but warm *state* is keyed by **recipe**: `warmsnap.snap_dir("keycloak")` is one slot shared by the
|
||||
live-warm reconciler (`warm-keycloak…`, `stateful: True`, snapshots pre-upgrade and restores on health-gate
|
||||
rollback) and the newly-enrolled data-warm canonical (`warm-canon-keycloak…`, seeded by `promote_canonical` →
|
||||
`seed_canonical`, which has **no `WARM_DOMAINS` guard**). `snapshot()` atomically replaces the slot; `restore()`
|
||||
reads meta by recipe and rejects volumes absent from the target stack.
|
||||
|
||||
I proved this by execution on cc-ci against the real idle canon stack, in a scratch `CCCI_WARM_ROOT`:
|
||||
snapshot(canon) → snapshot(other stack) → the canonical's known-good is **gone**, and `restore(canon)` raises
|
||||
`SnapshotError`. It fails closed (no cross-stack data write), but:
|
||||
|
||||
1. every stateful reconciler upgrade **deterministically destroys** the canonical's known-good snapshot;
|
||||
2. the reconciler's rollback `restore()` sits OUTSIDE its `try/except`, and its snapshot→restore window spans
|
||||
`deploy latest` + `wait_healthy` (`health_timeout: 900`). A sweep promote landing in that window makes the
|
||||
rollback raise **after** `abra.undeploy(live)` has already run → **live keycloak left undeployed**, i.e. an
|
||||
outage of the shared OIDC provider `lasuite-*`/`drone` depend on. That is precisely the hazard the original
|
||||
canon §2.B de-enrollment exception existed to prevent;
|
||||
3. `prune_stale()`'s documented invariant ("keycloak… `last_good`, no `canonical.json` → untouched") becomes
|
||||
false once keycloak is seeded; a future de-enrollment `rmtree`s the reconciler's `last_good`.
|
||||
|
||||
**Why the M2 run could not have caught it, and why I am only finding it now.** The enrollment's data path never
|
||||
executed. On cc-ci `/var/lib/ci-warm/keycloak/` holds **only** `last_good` — no `canonical.json`, no `snapshot/` —
|
||||
whereas a normal canonical (`cryptpad`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
|
||||
*deployed*; `seed_canonical` never ran, because registry-advance is deliberately deferred to the operator's
|
||||
merge ("nothing merged"). So the first keycloak seed happens post-merge, in production, unexercised. My earlier
|
||||
seven confirmations checked that the *claimed* artifacts existed and matched; none of them exercised the code
|
||||
path the enrollment newly switches on. That was the gap.
|
||||
|
||||
## VETO — keycloak enrollment (`redfix-m2-harness` @61211db) must not be merged as-is
|
||||
|
||||
DoD requires "keycloak enrolled via a **collision-free** warm domain … FIXED and verified green." The domain is
|
||||
collision-free; the *enrollment* is not. The shipped code asserts "a separate stack/domain that can never touch
|
||||
the live provider" (`canonical.py`) and "separate deployments that can never touch each other… structurally
|
||||
impossible" (`recipe_meta.py`). Both are falsified above by execution. Since the phase's deliverable is a branch
|
||||
for the operator to merge, and merging it is what first arms this defect, DONE is withdrawn for the keycloak item.
|
||||
|
||||
Scope of the VETO: **keycloak only.** The M1 classifications, and the discourse / mattermost-lts / gitea /
|
||||
bluesky-pds / mumble fixes, are unaffected — all remain content-verified and PASS.
|
||||
|
||||
**Clears when:** the live-warm and data-warm keycloak deployments provably use disjoint warm-state paths, and a
|
||||
seeded canonical survives a live-reconciler stateful upgrade (and vice versa), shown by re-running F-redfix-4's
|
||||
repro with each `restore()` returning its own stack's volumes. Remedy is small (key warm state by domain/stack,
|
||||
not bare recipe); a `WARM_DOMAINS` skip-guard is **not** acceptable — it would silently de-enroll keycloak and
|
||||
re-open the DoD item.
|
||||
|
||||
Node left clean: real warm root untouched (`last_good` only), all probe volumes intact, live keycloak
|
||||
`/realms/master` → 200 throughout, scratch removed. JOURNAL not consulted before this verdict.
|
||||
|
||||
Reference in New Issue
Block a user