status(redfix): record the b5f2b10 MERGE PRECONDITION; widen B-redfix-5 to both post-undeploy sites

Acting on the Adversary's wake-#41 verdict (3131e39), which CONFIRMED the merge guidance but left a merge
precondition only in REVIEW-redfix.md — Adversary-owned, and not what an operator reads before merging.
Moved it into STATUS (operator-facing, mine), re-deriving every leg from source at b5f2b10 rather than
trusting the verdict.

- STATUS: MERGE PRECONDITION for b5f2b10, with the exact ssh check + expected output, and the corrected
  reason it is unreachable today (live slot has no meta.json — NOT "F-redfix-4 is closed").
- Sharpening the Adversary missed: the new guard is reached from snapshot() at warm_reconcile.py:514 as
  well as restore() at :536. Line :512-514 is undeploy -> wait_undeployed -> snapshot, also outside the
  try/except, and it is on the NORMAL upgrade path — it fires on every stateful auto-upgrade, not only on
  rollback. B-redfix-5 as filed named only the rollback site.
- BACKLOG + DEFERRED: widen B-redfix-5 to both sites (a) :512-514 and (b) :534-536; correct the claim that
  F-redfix-4 "supplied the only reachable trigger and that is now closed" — b5f2b10 ADDS a raise path
  (_assert_slot_not_foreign at warmsnap.py:158 and :209).

Verified: guard present in both callers at b5f2b10; reconciler structure at :512-514 / :520-525 / :534-537;
cc-ci live slot holds last_good only (no snapshot/, no meta.json, no canon-* slot). Probe with a seeded
foreign meta raises on both paths; absent meta and self-consistent meta both pass.

No DoD item touched. M1+M2 PASS stand, ## DONE stands, no VETO. Docs-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZmH4rVNue3irZ6EsZzavb
This commit is contained in:
2026-07-09 08:16:06 +00:00
co-authored by Claude Opus 4.8
parent 3131e39f64
commit 93614f114e
4 changed files with 113 additions and 22 deletions
+20 -11
View File
@@ -58,17 +58,26 @@ hold). Concrete fix designs from M1 evidence:
`_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci `_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
(each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per (each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
STATUS-redfix.md "Gate: M2 RE-CLAIMED". STATUS-redfix.md "Gate: M2 RE-CLAIMED".
- [ ] **B-redfix-5 — reconciler rollback `restore()` is outside the upgrade's `try/except`** (NOT blocking; - [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`**
NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In (NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped).
`warm_reconcile.py`, the unhealthy-rollback path runs `abra.undeploy(domain)` `wait_undeployed` In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both
`warmsnap.restore(...)``deploy_version(last_good)`. `restore()` sits outside the `try/except` that outside the `try/except` that guards the upgrade:
guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the **(a) upgrade path** `:512-514``abra.undeploy``wait_undeployed``warmsnap.snapshot(...)`;
exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**. **(b) rollback path** `:534-536``abra.undeploy``wait_undeployed``warmsnap.restore(...)`
F-redfix-4 supplied one way to make `restore()` raise (the shared slot) and that is now fixed, but the `deploy_version(last_good)` at `:537`.
structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes
`last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left
breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app **undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire.
after a forward migration — that trade-off is why this is filed, not fixed inline. **Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one —
`_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and
`restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no
`snapshot/meta.json` (`read_meta``None`, and an unclaimed slot is free to claim) — *not* because
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that
trade-off is why this is filed, not fixed inline.
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17, - [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15 `redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
+17 -8
View File
@@ -430,15 +430,24 @@ reachable via the operator/dev STAGES escape — production drone runs always ru
## B-redfix-5 — warm reconciler's rollback `restore()` is outside the upgrade's `try/except` ## B-redfix-5 — warm reconciler's rollback `restore()` is outside the upgrade's `try/except`
- **What:** in `runner/warm_reconcile.py`, the unhealthy-rollback path is `abra.undeploy(domain)` - **What:** in `runner/warm_reconcile.py`, **two** post-`abra.undeploy()` warmsnap calls sit OUTSIDE the
`wait_undeployed``warmsnap.restore(...)``deploy_version(last_good)`. `restore()` sits OUTSIDE the `try/except` guarding the upgrade — **(a)** the upgrade path `:512-514` (`undeploy``wait_undeployed`
`try/except` guarding the upgrade, so if it raises for any reason (absent/corrupt snapshot, docker error) `warmsnap.snapshot(...)`) and **(b)** the unhealthy-rollback path `:534-536` (`undeploy`
the exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**, `wait_undeployed``warmsnap.restore(...)``deploy_version(last_good)` at `:537`). If either raises for
taking down the shared OIDC provider `lasuite-*`/`drone` depend on. any reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes found") the exception
propagates, the following `deploy_version` never runs, and live keycloak is left **undeployed**, taking
down the shared OIDC provider `lasuite-*`/`drone` depend on. Site (a) fires on the *normal* upgrade path —
no rollback required.
- **Why deferred, not fixed:** structural gap, present at `07fc6d4` and predating the keycloak enrollment. - **Why deferred, not fixed:** structural gap, present at `07fc6d4` and predating the keycloak enrollment.
F-redfix-4 supplied the only *reachable* trigger (the shared snapshot slot) and that is now closed at Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is **not a
`b5f2b10`. Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is VETO** and was "correctly filed rather than silently fixed".
**not a VETO** and was "correctly filed rather than silently fixed". - **Reachability, corrected (wake #42).** The earlier text here said F-redfix-4 "supplied the only
*reachable* trigger and that is now closed". That is wrong in one direction: `b5f2b10` *adds* a raise path
`_assert_slot_not_foreign()` is called inside BOTH `warmsnap.snapshot()` (`:158`) and `warmsnap.restore()`
(`:209`). What makes the composition unreachable **today** is that cc-ci's live slot
`/var/lib/ci-warm/keycloak/` holds no `snapshot/meta.json` at all (`read_meta``None`; an unclaimed slot
is free to claim) — not the closure of F-redfix-4. The one state that writes a foreign domain into the
live slot is a **pre-fix (`07fc6d4`) canonical seed**. Recorded as a MERGE PRECONDITION in STATUS-redfix.md.
- **Needed from operator:** decide the safe behaviour for a DB-backed app after a forward migration — - **Needed from operator:** decide the safe behaviour for a DB-backed app after a forward migration —
(a) redeploy `last_good` anyway even though data was not restored, or (b) die loudly + leave a breadcrumb (a) redeploy `last_good` anyway even though data was not restored, or (b) die loudly + leave a breadcrumb
rather than start on unrestored data. That trade-off is a real safety call, not a mechanical fix, which is rather than start on unrestored data. That trade-off is a real safety call, not a mechanical fix, which is
+50
View File
@@ -1669,3 +1669,53 @@ differs from `3fcea78925015fc9`.
**Unrelated, surfaced not touched:** an untracked `main.go` (281 B, Go hello-world HTTP server on :8080) **Unrelated, surfaced not touched:** an untracked `main.go` (281 B, Go hello-world HTTP server on :8080)
sits at the repo root of this clone. Not mine, foreign to this Nix repo, absent from git and from sits at the repo root of this clone. Not mine, foreign to this Nix repo, absent from git and from
`origin/main`. Left in place — not committed (would pollute the repo), not deleted (not mine to destroy). `origin/main`. Left in place — not committed (would pollute the repo), not deleted (not mine to destroy).
---
## Wake #42 — 2026-07-09 — watchdog ping on the Adversary's wake-#41 verdict
Pulled `cf1455f`. Verdict is a CONFIRM of the merge guidance (`b5f2b10`, not `07fc6d4`), no new finding, no
VETO, M1+M2 PASS stand. So: nothing to unblock, no gate to re-claim. But the verdict carried a *merge
precondition* that existed only in REVIEW-redfix.md, which is Adversary-owned and which the operator has no
reason to read before merging. STATUS is the operator-facing artifact I own. Moving it across is the work.
I did not take the Adversary's word for it. Re-derived every leg from source at `b5f2b10` in a detached
worktree, plus one live check on the node:
1. `restore()` really does call the new guard (`warmsnap.py:209`), and `snapshot()` calls it too (`:158`).
So `b5f2b10` *adds* a raise path to warmsnap — it does not only remove one.
2. `warm_reconcile.py` rollback: `abra.undeploy``wait_undeployed``restore``deploy_version` at
`:534-537`, all outside the `try/except` at `:520-525`. Confirmed — this is B-redfix-5 as filed.
3. cc-ci live slot: `ls -A /var/lib/ci-warm/keycloak/``last_good` only; no `snapshot/`, no `meta.json`;
no `canon-*` slot. So `read_meta``None` → guard passes. Composition unreachable **today**.
Then the part the Adversary understated. It framed the risk as "B-redfix-5 × the new guard can wedge the
live OIDC provider **on rollback**". But the guard is also reached via `snapshot()` at `warm_reconcile.py:514`
— and `:512-514` is `abra.undeploy``wait_undeployed``snapshot`, likewise outside the try/except. That
site is on the **normal upgrade path**: it fires on every stateful auto-upgrade, no unhealthy release
required. So the blast radius of the precondition is strictly larger than "on rollback", and B-redfix-5 as
originally filed named only one of the two sites. Widened it in BACKLOG + DEFERRED.
Proving it needed care, and my first probe was wrong. Seeding a foreign `meta.json` and calling `restore()`
gave `FileNotFoundError: 'docker'`, not `SnapshotError` — I briefly read that as "the guard isn't wired into
restore after all", which would have contradicted the Adversary. It didn't: `restore()` runs
`_assert_undeployed(domain)` at `:205` *before* the guard at `:209`, and that shells out to docker, which is
absent on the orchestrator. The probe died upstream of the thing under test. `snapshot()` puts the guard
first (`:158`, before `_assert_undeployed`), which is why that half raised cleanly and the asymmetry looked
like a code difference rather than a probe artifact. Re-ran with the two docker-dependent preconditions
stubbed and the guard itself untouched: both paths raise on a foreign meta; an absent meta passes; a
self-consistent meta (`domain == warm-keycloak.ci.commoninternet.net`) passes. That is the real behaviour.
Same class of error as the `e3b0c44298fc1c14` empty-input tell from wake #39 — a probe that fails *before*
reaching the assertion prints something that reads like a finding. Worth the note: on this orchestrator
`docker`, `curl`, `wget`, `awk`, `pytest` are all absent, and `python3` is not on `PATH` at the store path
STATUS's verify-(1) cites. Reach for the stdlib and check what actually executed.
Corrected the reachability wording in BACKLOG + DEFERRED, which both claimed F-redfix-4 "supplied the only
*reachable* trigger and that is now closed". The trigger is not closed — it is unreachable for a *different*
reason (the live slot is empty), and that reason is a property of node state, not of the fix. Stating it the
old way would let an operator conclude the composition is impossible by construction. It isn't: one pre-fix
canonical seed writes `domain=warm-canon-keycloak…` into the live slot and arms both wedge sites.
No VETO to answer, no DoD item touched, no gate to claim. Docs-only. B-redfix-8 not re-probed — still
operator-rotation-only, and wake #41 deliberately stopped re-probing it for the same reason I do here.
+26 -3
View File
@@ -70,9 +70,32 @@ F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified.
and the still-open non-blocking B-redfix-5, are below. and the still-open non-blocking B-redfix-5, are below.
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
**B-redfix-5** in BACKLOG-redfix.md — `warm_reconcile.py`'s rollback `warmsnap.restore()` sits outside the **B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
upgrade's `try/except`. Pre-dates the enrollment (present at `07fc6d4`); F-redfix-4 supplied the only the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
*reachable* trigger and that is now closed. Adversary concurred it is not a VETO. the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the
enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO.
**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both
of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in
`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one
passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path
this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today**
because the live slot holds no snapshot at all (`read_meta``None` → "a slot with no snapshot yet is free
to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed.
The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which
wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. So: do not run a canonical
keycloak seed from `07fc6d4` between now and the merge.
**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports
`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is
anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5
(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording
`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard.
--- ---