review(redfix): VETO — keycloak enrollment not collision-free in warm state (F-redfix-4)
Some checks failed
continuous-integration/drone/push Build is failing
Some checks failed
continuous-integration/drone/push Build is failing
Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously only sha-reachability checked, never content, never second-order effects). mumble07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged, so a dead server still FAILs. Budget widened without weakening the test. keycloak61211db: correct at the domain/stack layer, but its own invariant is FALSE. canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler (warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical (warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard). Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack: snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but: 1. every stateful reconciler upgrade deterministically destroys the canonical snapshot; 2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's de-enrollment exception existed to prevent. 3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded. Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json, no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran, registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised. VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/ mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint warm-state paths and each restore() returns its own stack's volumes. Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout.
This commit is contained in:
autonomic-bot
@ -55,6 +55,107 @@ hold). Concrete fix designs from M1 evidence:
|
||||
|
||||
(Adversary-owned — do not edit.)
|
||||
|
||||
### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **OPEN, BLOCKING (VETO)** @2026-07-08T23:56Z
|
||||
|
||||
**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges
|
||||
`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone`
|
||||
depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent,
|
||||
resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**.
|
||||
|
||||
**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes
|
||||
any recipe in `warm.WARM_DOMAINS` to `warm-canon-<recipe>`, and `tests/keycloak/recipe_meta.py` flips
|
||||
`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci:
|
||||
|
||||
```
|
||||
canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net
|
||||
WARM_DOMAINS[keycloak] = warm-keycloak.ci.commoninternet.net
|
||||
stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
|
||||
stack_volumes(LIVE) = ['warm-keycloak_..._mariadb', 'warm-keycloak_..._providers']
|
||||
```
|
||||
|
||||
**The defect.** Warm *state* is keyed by RECIPE, not by domain:
|
||||
`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT/<recipe>/snapshot` and
|
||||
`canonical.registry_path(recipe) = $CCCI_WARM_ROOT/<recipe>/canonical.json`.
|
||||
So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already
|
||||
holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot:
|
||||
|
||||
| | producer | consumer |
|
||||
|---|---|---|
|
||||
| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) |
|
||||
| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) |
|
||||
|
||||
`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe
|
||||
and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match,
|
||||
so restore raises `SnapshotError` instead of cross-writing data.
|
||||
|
||||
Consequences, in descending certainty:
|
||||
|
||||
1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak
|
||||
overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`)
|
||||
then raises `SnapshotError` and cannot roll back.
|
||||
2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes.
|
||||
3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()`
|
||||
and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep
|
||||
`promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback
|
||||
then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits
|
||||
OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)`
|
||||
never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`.
|
||||
4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler
|
||||
dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it
|
||||
*has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and
|
||||
`shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`.
|
||||
|
||||
**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci,
|
||||
`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal
|
||||
canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
|
||||
*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge).
|
||||
The first-ever keycloak seed will therefore happen post-merge, in production, unexercised.
|
||||
|
||||
**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never
|
||||
touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally
|
||||
impossible." Both are false for warm state. That claim is what I falsified.
|
||||
|
||||
**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).**
|
||||
Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack
|
||||
(the live one cannot be undeployed to snapshot it):
|
||||
|
||||
```sh
|
||||
ssh cc-ci
|
||||
git clone -q --branch redfix-m2-harness <cc-ci remote> /tmp/p8 && cd /tmp/p8/runner
|
||||
CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY'
|
||||
import sys; sys.path.insert(0, ".")
|
||||
from harness import warmsnap as ws, canonical as c
|
||||
CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net"
|
||||
print(ws.snap_dir("keycloak")) # one slot, domain-blind
|
||||
ws.snapshot("keycloak", CANON, version="canon-known-good")
|
||||
ws.snapshot("keycloak", STANDIN, version="live-last-good") # clobbers it
|
||||
print(ws.read_meta("keycloak")["domain"]) # -> warm-custom-html… (canon known-good GONE)
|
||||
ws.restore("keycloak", CANON) # -> SnapshotError
|
||||
PY
|
||||
```
|
||||
|
||||
EXPECTED (observed @2026-07-08T23:55Z):
|
||||
```
|
||||
/tmp/p8w/keycloak/snapshot <- SAME slot for BOTH domains
|
||||
read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net <- canon known-good DESTROYED
|
||||
SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content
|
||||
absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers']
|
||||
```
|
||||
Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three
|
||||
volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed.
|
||||
|
||||
**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain*
|
||||
rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical
|
||||
seeds under `<recipe>-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never
|
||||
touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently
|
||||
de-enroll keycloak and re-open the DoD item, so it is not sufficient.
|
||||
|
||||
**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical
|
||||
survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and
|
||||
seeing each `restore()` return its OWN stack's volumes.
|
||||
|
||||
---
|
||||
|
||||
### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z**
|
||||
|
||||
**CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the
|
||||
|
||||
@ -503,3 +503,69 @@ fixes are now content-anchored, not merely sha-anchored.
|
||||
|
||||
No new finding. No gate reopened. All findings CLOSED (F-redfix-1, F-redfix-2, F-redfix-3).
|
||||
Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped.
|
||||
|
||||
### Post-reboot re-confirmation #8 @2026-07-08T23:56Z — **VETO RAISED** (new finding F-redfix-4)
|
||||
|
||||
Pulled: zero commit delta since #7. `## DONE` still in STATUS-redfix.md; no `ADVERSARY-INBOX.md`.
|
||||
F-redfix-1/2/3 remain CLOSED. This wake I ran a **new** break-it probe and it found a real defect.
|
||||
|
||||
**Probe angle (new).** #7 content-verified the three *recipe* fixes against their parents. The two
|
||||
*harness* fixes on `redfix-m2-harness` had only ever been checked for sha reachability (#6), never for
|
||||
content, and never for second-order effects. So I cold-cloned the harness repo and diffed both against
|
||||
their parents:
|
||||
|
||||
* **mumble `07fc6d4`** — CLEAN. `tests/mumble/custom/test_protocol_handshake.py`: `retry_handshake(attempts=12)`
|
||||
→ `attempts=36` at `interval=5.0` = the claimed 180s budget. Diff is +8/-1: the single retry line plus a
|
||||
comment. **Every assertion below it is unchanged**, so a genuinely dead server still exhausts retries and
|
||||
FAILs — the budget was widened without weakening the test. Claim matches code. ✔
|
||||
* **keycloak `61211db`** — `canonical_domain()` routes `WARM_DOMAINS` recipes to `warm-canon-<recipe>`, and
|
||||
`recipe_meta.WARM_CANONICAL` flips False→True. At the **domain/stack layer the fix is real and correct**
|
||||
(verified live on cc-ci: four volumes, two disjoint stack prefixes). **But its own stated invariant is false.**
|
||||
|
||||
**F-redfix-4 (filed in BACKLOG-redfix.md, full repro there).** The fix separates the two keycloak deployments
|
||||
by domain, but warm *state* is keyed by **recipe**: `warmsnap.snap_dir("keycloak")` is one slot shared by the
|
||||
live-warm reconciler (`warm-keycloak…`, `stateful: True`, snapshots pre-upgrade and restores on health-gate
|
||||
rollback) and the newly-enrolled data-warm canonical (`warm-canon-keycloak…`, seeded by `promote_canonical` →
|
||||
`seed_canonical`, which has **no `WARM_DOMAINS` guard**). `snapshot()` atomically replaces the slot; `restore()`
|
||||
reads meta by recipe and rejects volumes absent from the target stack.
|
||||
|
||||
I proved this by execution on cc-ci against the real idle canon stack, in a scratch `CCCI_WARM_ROOT`:
|
||||
snapshot(canon) → snapshot(other stack) → the canonical's known-good is **gone**, and `restore(canon)` raises
|
||||
`SnapshotError`. It fails closed (no cross-stack data write), but:
|
||||
|
||||
1. every stateful reconciler upgrade **deterministically destroys** the canonical's known-good snapshot;
|
||||
2. the reconciler's rollback `restore()` sits OUTSIDE its `try/except`, and its snapshot→restore window spans
|
||||
`deploy latest` + `wait_healthy` (`health_timeout: 900`). A sweep promote landing in that window makes the
|
||||
rollback raise **after** `abra.undeploy(live)` has already run → **live keycloak left undeployed**, i.e. an
|
||||
outage of the shared OIDC provider `lasuite-*`/`drone` depend on. That is precisely the hazard the original
|
||||
canon §2.B de-enrollment exception existed to prevent;
|
||||
3. `prune_stale()`'s documented invariant ("keycloak… `last_good`, no `canonical.json` → untouched") becomes
|
||||
false once keycloak is seeded; a future de-enrollment `rmtree`s the reconciler's `last_good`.
|
||||
|
||||
**Why the M2 run could not have caught it, and why I am only finding it now.** The enrollment's data path never
|
||||
executed. On cc-ci `/var/lib/ci-warm/keycloak/` holds **only** `last_good` — no `canonical.json`, no `snapshot/` —
|
||||
whereas a normal canonical (`cryptpad`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote
|
||||
*deployed*; `seed_canonical` never ran, because registry-advance is deliberately deferred to the operator's
|
||||
merge ("nothing merged"). So the first keycloak seed happens post-merge, in production, unexercised. My earlier
|
||||
seven confirmations checked that the *claimed* artifacts existed and matched; none of them exercised the code
|
||||
path the enrollment newly switches on. That was the gap.
|
||||
|
||||
## VETO — keycloak enrollment (`redfix-m2-harness` @61211db) must not be merged as-is
|
||||
|
||||
DoD requires "keycloak enrolled via a **collision-free** warm domain … FIXED and verified green." The domain is
|
||||
collision-free; the *enrollment* is not. The shipped code asserts "a separate stack/domain that can never touch
|
||||
the live provider" (`canonical.py`) and "separate deployments that can never touch each other… structurally
|
||||
impossible" (`recipe_meta.py`). Both are falsified above by execution. Since the phase's deliverable is a branch
|
||||
for the operator to merge, and merging it is what first arms this defect, DONE is withdrawn for the keycloak item.
|
||||
|
||||
Scope of the VETO: **keycloak only.** The M1 classifications, and the discourse / mattermost-lts / gitea /
|
||||
bluesky-pds / mumble fixes, are unaffected — all remain content-verified and PASS.
|
||||
|
||||
**Clears when:** the live-warm and data-warm keycloak deployments provably use disjoint warm-state paths, and a
|
||||
seeded canonical survives a live-reconciler stateful upgrade (and vice versa), shown by re-running F-redfix-4's
|
||||
repro with each `restore()` returning its own stack's volumes. Remedy is small (key warm state by domain/stack,
|
||||
not bare recipe); a `WARM_DOMAINS` skip-guard is **not** acceptable — it would silently de-enroll keycloak and
|
||||
re-open the DoD item.
|
||||
|
||||
Node left clean: real warm root untouched (`last_good` only), all probe volumes intact, live keycloak
|
||||
`/realms/master` → 200 throughout, scratch removed. JOURNAL not consulted before this verdict.
|
||||
|
||||
Reference in New Issue
Block a user