From 941d5e81c17845e3dd56099167c6101f3dba5a15 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Wed, 8 Jul 2026 23:58:01 +0000 Subject: [PATCH] =?UTF-8?q?review(redfix):=20VETO=20=E2=80=94=20keycloak?= =?UTF-8?q?=20enrollment=20not=20collision-free=20in=20warm=20state=20(F-r?= =?UTF-8?q?edfix-4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-reboot re-confirmation #8. New break-it probe on the two HARNESS fixes (previously only sha-reachability checked, never content, never second-order effects). mumble 07fc6d4: CLEAN — attempts 12->36 @5.0s = claimed 180s; all assertions unchanged, so a dead server still FAILs. Budget widened without weakening the test. keycloak 61211db: correct at the domain/stack layer, but its own invariant is FALSE. canonical_domain() separates the two deployments by domain, yet warm STATE is keyed by recipe: warmsnap.snap_dir("keycloak") is ONE slot shared by the live-warm reconciler (warm-keycloak, stateful=True) and the newly-enrolled data-warm canonical (warm-canon-keycloak, seeded via promote_canonical -> seed_canonical, no WARM_DOMAINS guard). Proved by execution on cc-ci in a scratch CCCI_WARM_ROOT against the real idle canon stack: snapshot(canon) -> snapshot(other stack) destroys the canonical known-good, and restore(canon) raises SnapshotError. Fails closed (no cross-stack data write), but: 1. every stateful reconciler upgrade deterministically destroys the canonical snapshot; 2. the reconciler's rollback restore() is OUTSIDE its try/except and its snapshot->restore window spans deploy+wait_healthy (health_timeout 900). A sweep promote landing there makes the rollback raise AFTER abra.undeploy(live) -> live keycloak left undeployed = outage of the shared OIDC provider lasuite-*/drone depend on. Exactly the hazard canon §2.B's de-enrollment exception existed to prevent. 3. prune_stale()'s documented "keycloak untouched" invariant becomes false once seeded. Undetectable by the M2 run: /var/lib/ci-warm/keycloak/ holds only last_good (no canonical.json, no snapshot/) vs cryptpad which has both — the promote deployed but seed_canonical never ran, registry-advance being deferred to the operator's merge. First seed happens post-merge, unexercised. VETO scoped to keycloak. M1 classifications and the discourse/mattermost-lts/gitea/bluesky-pds/ mumble fixes are unaffected and remain PASS. Clears when the two deployments use disjoint warm-state paths and each restore() returns its own stack's volumes. Node left clean: real warm root untouched, volumes intact, live keycloak 200 throughout. --- machine-docs/BACKLOG-redfix.md | 101 +++++++++++++++++++++++++++++++++ machine-docs/REVIEW-redfix.md | 66 +++++++++++++++++++++ 2 files changed, 167 insertions(+) diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index ac5b885..1269e99 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -55,6 +55,107 @@ hold). Concrete fix designs from M1 evidence: (Adversary-owned — do not edit.) +### [adversary] F-redfix-4 — keycloak enrollment is collision-free in DOMAIN but NOT in warm-state: the live-warm reconciler and the new data-warm canonical share one per-recipe snapshot slot — **OPEN, BLOCKING (VETO)** @2026-07-08T23:56Z + +**Severity:** BLOCKS the phase's keycloak DoD item and must be fixed before the operator merges +`redfix-m2-harness`. Worst case is an outage of the live shared OIDC provider that `lasuite-*`/`drone` +depend on — the exact hazard the original canon §2.B de-enrollment exception existed to prevent, +resurrected in a different namespace. Fails closed (raises), so **no silent data corruption**. + +**What the M2 fix does (and it does work, as far as it goes).** `canonical.canonical_domain()` now routes +any recipe in `warm.WARM_DOMAINS` to `warm-canon-`, and `tests/keycloak/recipe_meta.py` flips +`WARM_CANONICAL = True`. The two stacks are genuinely distinct at the docker layer — verified on cc-ci: + +``` +canonical_domain(keycloak) = warm-canon-keycloak.ci.commoninternet.net +WARM_DOMAINS[keycloak] = warm-keycloak.ci.commoninternet.net +stack_volumes(CANON) = ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers'] +stack_volumes(LIVE) = ['warm-keycloak_..._mariadb', 'warm-keycloak_..._providers'] +``` + +**The defect.** Warm *state* is keyed by RECIPE, not by domain: +`warmsnap.snap_dir(recipe) = $CCCI_WARM_ROOT//snapshot` and +`canonical.registry_path(recipe) = $CCCI_WARM_ROOT//canonical.json`. +So both stacks now share **one** snapshot slot, `/var/lib/ci-warm/keycloak/snapshot/`, which already +holds the live reconciler's sibling `last_good`. Two producers, two consumers, one slot: + +| | producer | consumer | +|---|---|---| +| live-warm | `warm_reconcile.py:512` `snapshot(recipe, warm-keycloak…)` (stateful=True, pre-upgrade) | `warm_reconcile.py:534` `restore(recipe, warm-keycloak…)` (health-gate rollback) | +| data-warm | `canonical.seed_canonical` → `warmsnap.snapshot(recipe, warm-canon-keycloak…)` (via `run_recipe_ci.py:1047` `promote_canonical`, **no `WARM_DOMAINS` guard**) | `run_recipe_ci.py:896` `restore(recipe, warm-canon-keycloak…)` (quick-FAIL canonical rollback) | + +`warmsnap.snapshot()` atomically **replaces** the slot; `warmsnap.restore()` reads `meta.json` by recipe +and then requires every recorded volume to exist in the *target* stack. Cross-stack names never match, +so restore raises `SnapshotError` instead of cross-writing data. + +Consequences, in descending certainty: + +1. **Deterministic — canonical known-good destroyed.** Every stateful reconciler upgrade of live keycloak + overwrites the canonical's snapshot. The canonical's WC4 quick-FAIL rollback (`run_recipe_ci.py:896`) + then raises `SnapshotError` and cannot roll back. +2. **Deterministic — reverse direction.** After a promote seeds the canonical, the slot holds canon volumes. +3. **Race, high impact — live SSO outage.** The reconciler's window between its pre-upgrade `snapshot()` + and its rollback `restore()` spans `deploy latest` + `wait_healthy` (`health_timeout: 900`). A nightly-sweep + `promote_canonical(keycloak)` landing in that window replaces the slot with canon volumes. The rollback + then does `abra.undeploy(live)` → `wait_undeployed` → `warmsnap.restore(...)` → **raises**. `restore` sits + OUTSIDE the `try/except` that guards the upgrade, so the exception propagates and `deploy_version(last_good)` + never runs — **live keycloak is left undeployed**, taking SSO down for `lasuite-*`/`drone`. +4. **Latent — `prune_stale()` invariant now false.** Its docstring promises it "Leaves the live-warm reconciler + dirs (keycloak/traefik — they have a `last_good`, no `canonical.json`) untouched." Once keycloak is seeded it + *has* a `canonical.json`; if `WARM_CANONICAL` is ever flipped back to False, `prune_stale` matches it and + `shutil.rmtree(app_dir("keycloak"))` deletes the live reconciler's `last_good`. + +**Why M2's verification could not have caught this.** The enrollment's data path never executed: on cc-ci, +`/var/lib/ci-warm/keycloak/` contains **only** `last_good` — no `canonical.json`, no `snapshot/` — while a normal +canonical (`/var/lib/ci-warm/cryptpad/`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote +*deployed*, but `seed_canonical` never ran (registry-advance is deliberately deferred to the operator's merge). +The first-ever keycloak seed will therefore happen post-merge, in production, unexercised. + +**The shipped code asserts the opposite.** `canonical.py` docstring: "a separate stack/domain that can never +touch the live provider"; `recipe_meta.py`: "separate deployments that can never touch each other… structurally +impossible." Both are false for warm state. That claim is what I falsified. + +**Repro (cold, non-destructive — writes only to a scratch `CCCI_WARM_ROOT`; never touches the live stack).** +Uses the real idle `warm-canon-keycloak` stack and idle `warm-custom-html` as a stand-in for the live stack +(the live one cannot be undeployed to snapshot it): + +```sh +ssh cc-ci +git clone -q --branch redfix-m2-harness /tmp/p8 && cd /tmp/p8/runner +CCCI_WARM_ROOT=/tmp/p8w /nix/store/jag2131a95gw6ng7grig9pj3dn2q8vrv-python3-3.12.8-env/bin/python3 - <<'PY' +import sys; sys.path.insert(0, ".") +from harness import warmsnap as ws, canonical as c +CANON, STANDIN = c.canonical_domain("keycloak"), "warm-custom-html.ci.commoninternet.net" +print(ws.snap_dir("keycloak")) # one slot, domain-blind +ws.snapshot("keycloak", CANON, version="canon-known-good") +ws.snapshot("keycloak", STANDIN, version="live-last-good") # clobbers it +print(ws.read_meta("keycloak")["domain"]) # -> warm-custom-html… (canon known-good GONE) +ws.restore("keycloak", CANON) # -> SnapshotError +PY +``` + +EXPECTED (observed @2026-07-08T23:55Z): +``` +/tmp/p8w/keycloak/snapshot <- SAME slot for BOTH domains +read_meta(keycloak).domain = warm-custom-html.ci.commoninternet.net <- canon known-good DESTROYED +SnapshotError -> snapshot volume warm-custom-html_ci_commoninternet_net_content + absent from current stack ['warm-canon-keycloak_..._mariadb', 'warm-canon-keycloak_..._providers'] +``` +Post-probe the node was verified untouched: `/var/lib/ci-warm/keycloak/` still `last_good` only, all three +volumes intact (159M / 8.0K / 40K, file counts unchanged), live `warm-keycloak…/realms/master` → 200. Scratch removed. + +**Proposed remedy (Builder's to choose — mine to file, not to make).** Key warm state by the *stack/domain* +rather than the bare recipe for recipes in `WARM_DOMAINS` — e.g. `app_dir()` takes the domain, or the canonical +seeds under `-canon/`. Then: fix `prune_stale`'s now-false invariant, and correct the two "can never +touch each other" comments. A guard alone (skip `seed_canonical` for `WARM_DOMAINS` recipes) would silently +de-enroll keycloak and re-open the DoD item, so it is not sufficient. + +**Clears the VETO when:** the two stacks provably use disjoint warm-state paths, and a seeded keycloak canonical +survives a live-reconciler stateful upgrade (and vice versa) — demonstrated by re-running the repro above and +seeing each `restore()` return its OWN stack's volumes. + +--- + ### [adversary] F-redfix-1 — discourse migration INCOMPLETE: dangling image-less `sidekiq` in compose.smtpauth.yml (R011 lint regression + breaks SMTP-auth deploys) — **CLOSED @2026-06-18T07:06Z** **CLOSED by Adversary re-test.** Builder fixed in PR #4 @9ff5e19 (force-pushed onto 53ba0910): removed the diff --git a/machine-docs/REVIEW-redfix.md b/machine-docs/REVIEW-redfix.md index 4e39336..27fd395 100644 --- a/machine-docs/REVIEW-redfix.md +++ b/machine-docs/REVIEW-redfix.md @@ -503,3 +503,69 @@ fixes are now content-anchored, not merely sha-anchored. No new finding. No gate reopened. All findings CLOSED (F-redfix-1, F-redfix-2, F-redfix-3). Terminal condition met (DONE + fresh PASS on every gate + no VETO). Loop stopped. + +### Post-reboot re-confirmation #8 @2026-07-08T23:56Z — **VETO RAISED** (new finding F-redfix-4) + +Pulled: zero commit delta since #7. `## DONE` still in STATUS-redfix.md; no `ADVERSARY-INBOX.md`. +F-redfix-1/2/3 remain CLOSED. This wake I ran a **new** break-it probe and it found a real defect. + +**Probe angle (new).** #7 content-verified the three *recipe* fixes against their parents. The two +*harness* fixes on `redfix-m2-harness` had only ever been checked for sha reachability (#6), never for +content, and never for second-order effects. So I cold-cloned the harness repo and diffed both against +their parents: + +* **mumble `07fc6d4`** — CLEAN. `tests/mumble/custom/test_protocol_handshake.py`: `retry_handshake(attempts=12)` + → `attempts=36` at `interval=5.0` = the claimed 180s budget. Diff is +8/-1: the single retry line plus a + comment. **Every assertion below it is unchanged**, so a genuinely dead server still exhausts retries and + FAILs — the budget was widened without weakening the test. Claim matches code. ✔ +* **keycloak `61211db`** — `canonical_domain()` routes `WARM_DOMAINS` recipes to `warm-canon-`, and + `recipe_meta.WARM_CANONICAL` flips False→True. At the **domain/stack layer the fix is real and correct** + (verified live on cc-ci: four volumes, two disjoint stack prefixes). **But its own stated invariant is false.** + +**F-redfix-4 (filed in BACKLOG-redfix.md, full repro there).** The fix separates the two keycloak deployments +by domain, but warm *state* is keyed by **recipe**: `warmsnap.snap_dir("keycloak")` is one slot shared by the +live-warm reconciler (`warm-keycloak…`, `stateful: True`, snapshots pre-upgrade and restores on health-gate +rollback) and the newly-enrolled data-warm canonical (`warm-canon-keycloak…`, seeded by `promote_canonical` → +`seed_canonical`, which has **no `WARM_DOMAINS` guard**). `snapshot()` atomically replaces the slot; `restore()` +reads meta by recipe and rejects volumes absent from the target stack. + +I proved this by execution on cc-ci against the real idle canon stack, in a scratch `CCCI_WARM_ROOT`: +snapshot(canon) → snapshot(other stack) → the canonical's known-good is **gone**, and `restore(canon)` raises +`SnapshotError`. It fails closed (no cross-stack data write), but: + +1. every stateful reconciler upgrade **deterministically destroys** the canonical's known-good snapshot; +2. the reconciler's rollback `restore()` sits OUTSIDE its `try/except`, and its snapshot→restore window spans + `deploy latest` + `wait_healthy` (`health_timeout: 900`). A sweep promote landing in that window makes the + rollback raise **after** `abra.undeploy(live)` has already run → **live keycloak left undeployed**, i.e. an + outage of the shared OIDC provider `lasuite-*`/`drone` depend on. That is precisely the hazard the original + canon §2.B de-enrollment exception existed to prevent; +3. `prune_stale()`'s documented invariant ("keycloak… `last_good`, no `canonical.json` → untouched") becomes + false once keycloak is seeded; a future de-enrollment `rmtree`s the reconciler's `last_good`. + +**Why the M2 run could not have caught it, and why I am only finding it now.** The enrollment's data path never +executed. On cc-ci `/var/lib/ci-warm/keycloak/` holds **only** `last_good` — no `canonical.json`, no `snapshot/` — +whereas a normal canonical (`cryptpad`) has both. The `warm-canon-keycloak_*` volumes exist, so the promote +*deployed*; `seed_canonical` never ran, because registry-advance is deliberately deferred to the operator's +merge ("nothing merged"). So the first keycloak seed happens post-merge, in production, unexercised. My earlier +seven confirmations checked that the *claimed* artifacts existed and matched; none of them exercised the code +path the enrollment newly switches on. That was the gap. + +## VETO — keycloak enrollment (`redfix-m2-harness` @61211db) must not be merged as-is + +DoD requires "keycloak enrolled via a **collision-free** warm domain … FIXED and verified green." The domain is +collision-free; the *enrollment* is not. The shipped code asserts "a separate stack/domain that can never touch +the live provider" (`canonical.py`) and "separate deployments that can never touch each other… structurally +impossible" (`recipe_meta.py`). Both are falsified above by execution. Since the phase's deliverable is a branch +for the operator to merge, and merging it is what first arms this defect, DONE is withdrawn for the keycloak item. + +Scope of the VETO: **keycloak only.** The M1 classifications, and the discourse / mattermost-lts / gitea / +bluesky-pds / mumble fixes, are unaffected — all remain content-verified and PASS. + +**Clears when:** the live-warm and data-warm keycloak deployments provably use disjoint warm-state paths, and a +seeded canonical survives a live-reconciler stateful upgrade (and vice versa), shown by re-running F-redfix-4's +repro with each `restore()` returning its own stack's volumes. Remedy is small (key warm state by domain/stack, +not bare recipe); a `WARM_DOMAINS` skip-guard is **not** acceptable — it would silently de-enroll keycloak and +re-open the DoD item. + +Node left clean: real warm root untouched (`last_good` only), all probe volumes intact, live keycloak +`/realms/master` → 200 throughout, scratch removed. JOURNAL not consulted before this verdict.