review(redfix): VETO — keycloak enrollment not collision-free in warm state (F-redfix-4)
Some checks failed
continuous-integration/drone/push Build is failing

Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously
only sha-reachability checked, never content, never second-order effects).

mumble 07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged,
so a dead server still FAILs. Budget widened without weakening the test.

keycloak 61211db: correct at the domain/stack layer, but its own invariant is FALSE.
canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by
recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler
(warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical
(warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard).

Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack:
snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and
restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but:
 1. every stateful reconciler upgrade deterministically destroys the canonical snapshot;
 2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore
    window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes
    the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of
    the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's
    de-enrollment exception existed to prevent.
 3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded.

Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json,
no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran,
registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised.

VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/
mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint
warm-state paths and each restore() returns its own stack's volumes.

Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout.
This commit is contained in:
autonomic-bot
2026-07-08 23:58:01 +00:00
parent 1e513e7fe1
commit 941d5e81c1
2 changed files with 167 additions and 0 deletions

View File

@ -503,3 +503,69 @@ fixes are now content-anchored, not merely sha-anchored.
No new finding. No gate reopened. All findings CLOSED (F-redfix-1, F-redfix-2, F-redfix-3).
Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.
### Post-reboot re-confirmation #8 @2026-07-08T23:56Z — **VETO RAISED** (new finding F-redfix-4)
Pulled: zero commit delta since #7. `## DONE` still in STATUS-redfix.md; no `ADVERSARY-INBOX.md`.
F-redfix-1/2/3 remain CLOSED. This wake I ran a **new** break-it probe and it found a real defect.
**Probe angle (new).** #7 content-verified the three *recipe* fixes against their parents. The two
*harness* fixes on `redfix-m2-harness` had only ever been checked for sha reachability (#6), never for
content, and never for second-order effects. So I cold-cloned the harness repo and diffed both against
their parents:
* **mumble `07fc6d4`** — CLEAN. `tests/mumble/custom/test_protocol_handshake.py`: `retry_handshake(attempts=12)`
→ `attempts=36` at `interval=5.0` = the claimed 180s budget. Diff is +8/-1: the single retry line plus a
comment. **Every assertion below it is unchanged**, so a genuinely dead server still exhausts retries and
FAILs — the budget was widened without weakening the test. Claim matches code. ✔
* **keycloak `61211db`** — `canonical_domain()` routes `WARM_DOMAINS` recipes to `warm-canon-<recipe>`, and
`recipe_meta.WARM_CANONICAL` flips False→True. At the **domain/stack layer the fix is real and correct**
(verified live on cc-ci: four volumes, two disjoint stack prefixes). **But its own stated invariant is false.**
**F-redfix-4 (filed in BACKLOG-redfix.md, full repro there).** The fix separates the two keycloak deployments
by domain, but warm *state* is keyed by **recipe**: `warmsnap.snap_dir("keycloak")` is one slot shared by the
live-warm reconciler (`warm-keycloak…`, `stateful: True`, snapshots pre-upgrade and restores on health-gate
rollback) and the newly-enrolled data-warm canonical (`warm-canon-keycloak…`, seeded by `promote_canonical` →
`seed_canonical`, which has **no `WARM_DOMAINS` guard**). `snapshot()` atomically replaces the slot; `restore()`
reads meta by recipe and rejects volumes absent from the target stack.
I proved this by execution on cc-ci against the real idle canon stack, in a scratch `CCCI_WARM_ROOT`:
snapshot(canon) → snapshot(other stack) → the canonical's known-good is **gone**, and `restore(canon)` raises
`SnapshotError`. It fails closed (no cross-stack data write), but:
1. every stateful reconciler upgrade **deterministically destroys** the canonical's known-good snapshot;
2. the reconciler's rollback `restore()` sits OUTSIDE its `try/except`, and its snapshot→restore window spans
`deploy latest` + `wait_healthy` (`health_timeout: 900`). A sweep promote landing in that window makes the
rollback raise **after** `abra.undeploy(live)` has already run → **live keycloak left undeployed**, i.e. an
outage of the shared OIDC provider `lasuite-*`/`drone` depend on. That is precisely the hazard the original
canon §2.B de-enrollment exception existed to prevent;
3. `prune_stale()`'s documented invariant ("keycloak… `last_good`, no `canonical.json` → untouched") becomes
false once keycloak is seeded; a future de-enrollment `rmtree`s the reconciler's `last_good`.
**Why the M2 run could not have caught it, and why I am only finding it now.** The enrollment's data path never
executed. On cc-ci `/var/lib/ci-warm/keycloak/` holds **only** `last_good` — no `canonical.json`, no `snapshot/` —
whereas a normal canonical (`cryptpad`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
*deployed*; `seed_canonical` never ran, because registry-advance is deliberately deferred to the operator's
merge ("nothing merged"). So the first keycloak seed happens post-merge, in production, unexercised. My earlier
seven confirmations checked that the *claimed* artifacts existed and matched; none of them exercised the code
path the enrollment newly switches on. That was the gap.
## VETO — keycloak enrollment (`redfix-m2-harness` @61211db) must not be merged as-is
DoD requires "keycloak enrolled via a **collision-free** warm domain … FIXED and verified green." The domain is
collision-free; the *enrollment* is not. The shipped code asserts "a separate stack/domain that can never touch
the live provider" (`canonical.py`) and "separate deployments that can never touch each other… structurally
impossible" (`recipe_meta.py`). Both are falsified above by execution. Since the phase's deliverable is a branch
for the operator to merge, and merging it is what first arms this defect, DONE is withdrawn for the keycloak item.
Scope of the VETO: **keycloak only.** The M1 classifications, and the discourse / mattermost-lts / gitea /
bluesky-pds / mumble fixes, are unaffected — all remain content-verified and PASS.
**Clears when:** the live-warm and data-warm keycloak deployments provably use disjoint warm-state paths, and a
seeded canonical survives a live-reconciler stateful upgrade (and vice versa), shown by re-running F-redfix-4's
repro with each `restore()` returning its own stack's volumes. Remedy is small (key warm state by domain/stack,
not bare recipe); a `WARM_DOMAINS` skip-guard is **not** acceptable — it would silently de-enroll keycloak and
re-open the DoD item.
Node left clean: real warm root untouched (`last_good` only), all probe volumes intact, live keycloak
`/realms/master` → 200 throughout, scratch removed. JOURNAL not consulted before this verdict.