From 93614f114ec3ee5a6416e8377157da8b5015516a Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Thu, 9 Jul 2026 08:16:06 +0000 Subject: [PATCH] status(redfix): record the b5f2b10 MERGE PRECONDITION; widen B-redfix-5 to both post-undeploy sites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Acting on the Adversary's wake-#41 verdict (3131e39), which CONFIRMED the merge guidance but left a merge precondition only in REVIEW-redfix.md — Adversary-owned, and not what an operator reads before merging. Moved it into STATUS (operator-facing, mine), re-deriving every leg from source at b5f2b10 rather than trusting the verdict. - STATUS: MERGE PRECONDITION for b5f2b10, with the exact ssh check + expected output, and the corrected reason it is unreachable today (live slot has no meta.json — NOT "F-redfix-4 is closed"). - Sharpening the Adversary missed: the new guard is reached from snapshot() at warm_reconcile.py:514 as well as restore() at :536. Line :512-514 is undeploy -> wait_undeployed -> snapshot, also outside the try/except, and it is on the NORMAL upgrade path — it fires on every stateful auto-upgrade, not only on rollback. B-redfix-5 as filed named only the rollback site. - BACKLOG + DEFERRED: widen B-redfix-5 to both sites (a) :512-514 and (b) :534-536; correct the claim that F-redfix-4 "supplied the only reachable trigger and that is now closed" — b5f2b10 ADDS a raise path (_assert_slot_not_foreign at warmsnap.py:158 and :209). Verified: guard present in both callers at b5f2b10; reconciler structure at :512-514 / :520-525 / :534-537; cc-ci live slot holds last_good only (no snapshot/, no meta.json, no canon-* slot). Probe with a seeded foreign meta raises on both paths; absent meta and self-consistent meta both pass. No DoD item touched. M1+M2 PASS stand, ## DONE stands, no VETO. Docs-only. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01YZmH4rVNue3irZ6EsZzavb --- machine-docs/BACKLOG-redfix.md | 31 +++++++++++++-------- machine-docs/DEFERRED.md | 25 +++++++++++------ machine-docs/JOURNAL-redfix.md | 50 ++++++++++++++++++++++++++++++++++ machine-docs/STATUS-redfix.md | 29 ++++++++++++++++++-- 4 files changed, 113 insertions(+), 22 deletions(-) diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index 8356f2d..26f23e7 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -58,17 +58,26 @@ hold). Concrete fix designs from M1 evidence: `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per STATUS-redfix.md "Gate: M2 RE-CLAIMED". -- [ ] **B-redfix-5 — reconciler rollback `restore()` is outside the upgrade's `try/except`** (NOT blocking; - NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In - `warm_reconcile.py`, the unhealthy-rollback path runs `abra.undeploy(domain)` → `wait_undeployed` → - `warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits outside the `try/except` that - guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the - exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**. - F-redfix-4 supplied one way to make `restore()` raise (the shared slot) and that is now fixed, but the - structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys - `last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a - breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app - after a forward migration — that trade-off is why this is filed, not fixed inline. +- [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`** + (NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). + In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both + outside the `try/except` that guards the upgrade: + **(a) upgrade path** `:512-514` — `abra.undeploy` → `wait_undeployed` → `warmsnap.snapshot(...)`; + **(b) rollback path** `:534-536` — `abra.undeploy` → `wait_undeployed` → `warmsnap.restore(...)` → + `deploy_version(last_good)` at `:537`. + If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes + found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left + **undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire. + **Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one — + `_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and + `restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no + `snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because + F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign + domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md. + Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring + data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid- + rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that + trade-off is why this is filed, not fixed inline. - [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17, and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At `redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15 diff --git a/machine-docs/DEFERRED.md b/machine-docs/DEFERRED.md index ebf1a38..ee46540 100644 --- a/machine-docs/DEFERRED.md +++ b/machine-docs/DEFERRED.md @@ -430,15 +430,24 @@ reachable via the operator/dev STAGES escape — production drone runs always ru ## B-redfix-5 — warm reconciler's rollback `restore()` is outside the upgrade's `try/except` -- **What:** in `runner/warm_reconcile.py`, the unhealthy-rollback path is `abra.undeploy(domain)` → - `wait_undeployed` → `warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits OUTSIDE the - `try/except` guarding the upgrade, so if it raises for any reason (absent/corrupt snapshot, docker error) - the exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**, - taking down the shared OIDC provider `lasuite-*`/`drone` depend on. +- **What:** in `runner/warm_reconcile.py`, **two** post-`abra.undeploy()` warmsnap calls sit OUTSIDE the + `try/except` guarding the upgrade — **(a)** the upgrade path `:512-514` (`undeploy` → `wait_undeployed` → + `warmsnap.snapshot(...)`) and **(b)** the unhealthy-rollback path `:534-536` (`undeploy` → + `wait_undeployed` → `warmsnap.restore(...)` → `deploy_version(last_good)` at `:537`). If either raises for + any reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes found") the exception + propagates, the following `deploy_version` never runs, and live keycloak is left **undeployed**, taking + down the shared OIDC provider `lasuite-*`/`drone` depend on. Site (a) fires on the *normal* upgrade path — + no rollback required. - **Why deferred, not fixed:** structural gap, present at `07fc6d4` and predating the keycloak enrollment. - F-redfix-4 supplied the only *reachable* trigger (the shared snapshot slot) and that is now closed at - `b5f2b10`. Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is - **not a VETO** and was "correctly filed rather than silently fixed". + Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is **not a + VETO** and was "correctly filed rather than silently fixed". +- **Reachability, corrected (wake #42).** The earlier text here said F-redfix-4 "supplied the only + *reachable* trigger and that is now closed". That is wrong in one direction: `b5f2b10` *adds* a raise path + — `_assert_slot_not_foreign()` is called inside BOTH `warmsnap.snapshot()` (`:158`) and `warmsnap.restore()` + (`:209`). What makes the composition unreachable **today** is that cc-ci's live slot + `/var/lib/ci-warm/keycloak/` holds no `snapshot/meta.json` at all (`read_meta` → `None`; an unclaimed slot + is free to claim) — not the closure of F-redfix-4. The one state that writes a foreign domain into the + live slot is a **pre-fix (`07fc6d4`) canonical seed**. Recorded as a MERGE PRECONDITION in STATUS-redfix.md. - **Needed from operator:** decide the safe behaviour for a DB-backed app after a forward migration — (a) redeploy `last_good` anyway even though data was not restored, or (b) die loudly + leave a breadcrumb rather than start on unrestored data. That trade-off is a real safety call, not a mechanical fix, which is diff --git a/machine-docs/JOURNAL-redfix.md b/machine-docs/JOURNAL-redfix.md index 1991fe2..1540117 100644 --- a/machine-docs/JOURNAL-redfix.md +++ b/machine-docs/JOURNAL-redfix.md @@ -1669,3 +1669,53 @@ differs from `3fcea78925015fc9`. **Unrelated, surfaced not touched:** an untracked `main.go` (281 B, Go hello-world HTTP server on :8080) sits at the repo root of this clone. Not mine, foreign to this Nix repo, absent from git and from `origin/main`. Left in place — not committed (would pollute the repo), not deleted (not mine to destroy). + +--- + +## Wake #42 — 2026-07-09 — watchdog ping on the Adversary's wake-#41 verdict + +Pulled `cf1455f`. Verdict is a CONFIRM of the merge guidance (`b5f2b10`, not `07fc6d4`), no new finding, no +VETO, M1+M2 PASS stand. So: nothing to unblock, no gate to re-claim. But the verdict carried a *merge +precondition* that existed only in REVIEW-redfix.md, which is Adversary-owned and which the operator has no +reason to read before merging. STATUS is the operator-facing artifact I own. Moving it across is the work. + +I did not take the Adversary's word for it. Re-derived every leg from source at `b5f2b10` in a detached +worktree, plus one live check on the node: + +1. `restore()` really does call the new guard (`warmsnap.py:209`), and `snapshot()` calls it too (`:158`). + So `b5f2b10` *adds* a raise path to warmsnap — it does not only remove one. +2. `warm_reconcile.py` rollback: `abra.undeploy` → `wait_undeployed` → `restore` → `deploy_version` at + `:534-537`, all outside the `try/except` at `:520-525`. Confirmed — this is B-redfix-5 as filed. +3. cc-ci live slot: `ls -A /var/lib/ci-warm/keycloak/` → `last_good` only; no `snapshot/`, no `meta.json`; + no `canon-*` slot. So `read_meta` → `None` → guard passes. Composition unreachable **today**. + +Then the part the Adversary understated. It framed the risk as "B-redfix-5 × the new guard can wedge the +live OIDC provider **on rollback**". But the guard is also reached via `snapshot()` at `warm_reconcile.py:514` +— and `:512-514` is `abra.undeploy` → `wait_undeployed` → `snapshot`, likewise outside the try/except. That +site is on the **normal upgrade path**: it fires on every stateful auto-upgrade, no unhealthy release +required. So the blast radius of the precondition is strictly larger than "on rollback", and B-redfix-5 as +originally filed named only one of the two sites. Widened it in BACKLOG + DEFERRED. + +Proving it needed care, and my first probe was wrong. Seeding a foreign `meta.json` and calling `restore()` +gave `FileNotFoundError: 'docker'`, not `SnapshotError` — I briefly read that as "the guard isn't wired into +restore after all", which would have contradicted the Adversary. It didn't: `restore()` runs +`_assert_undeployed(domain)` at `:205` *before* the guard at `:209`, and that shells out to docker, which is +absent on the orchestrator. The probe died upstream of the thing under test. `snapshot()` puts the guard +first (`:158`, before `_assert_undeployed`), which is why that half raised cleanly and the asymmetry looked +like a code difference rather than a probe artifact. Re-ran with the two docker-dependent preconditions +stubbed and the guard itself untouched: both paths raise on a foreign meta; an absent meta passes; a +self-consistent meta (`domain == warm-keycloak.ci.commoninternet.net`) passes. That is the real behaviour. + +Same class of error as the `e3b0c44298fc1c14` empty-input tell from wake #39 — a probe that fails *before* +reaching the assertion prints something that reads like a finding. Worth the note: on this orchestrator +`docker`, `curl`, `wget`, `awk`, `pytest` are all absent, and `python3` is not on `PATH` at the store path +STATUS's verify-(1) cites. Reach for the stdlib and check what actually executed. + +Corrected the reachability wording in BACKLOG + DEFERRED, which both claimed F-redfix-4 "supplied the only +*reachable* trigger and that is now closed". The trigger is not closed — it is unreachable for a *different* +reason (the live slot is empty), and that reason is a property of node state, not of the fix. Stating it the +old way would let an operator conclude the composition is impossible by construction. It isn't: one pre-fix +canonical seed writes `domain=warm-canon-keycloak…` into the live slot and arms both wedge sites. + +No VETO to answer, no DoD item touched, no gate to claim. Docs-only. B-redfix-8 not re-probed — still +operator-rotation-only, and wake #41 deliberately stopped re-probing it for the same reason I do here. diff --git a/machine-docs/STATUS-redfix.md b/machine-docs/STATUS-redfix.md index bcdf10a..f6b9d01 100644 --- a/machine-docs/STATUS-redfix.md +++ b/machine-docs/STATUS-redfix.md @@ -70,9 +70,32 @@ F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified. and the still-open non-blocking B-redfix-5, are below. One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): -**B-redfix-5** in BACKLOG-redfix.md — `warm_reconcile.py`'s rollback `warmsnap.restore()` sits outside the -upgrade's `try/except`. Pre-dates the enrollment (present at `07fc6d4`); F-redfix-4 supplied the only -*reachable* trigger and that is now closed. Adversary concurred it is not a VETO. +**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside +the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on +the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the +enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO. + +**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both +of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in +`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one +passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path +this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** +because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free +to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed. + +The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which +wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. So: do not run a canonical +keycloak seed from `07fc6d4` between now and the merge. + +**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`): + + ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1' + +**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports +`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is +anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5 +(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording +`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard. ---