recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

status(redfix): record the b5f2b10 MERGE PRECONDITION; widen B-redfix-5 to both post-undeploy sites

Browse Source 
Acting on the Adversary's wake-#41 verdict (3131e39), which CONFIRMED the merge guidance but left a merge
precondition only in REVIEW-redfix.md — Adversary-owned, and not what an operator reads before merging.
Moved it into STATUS (operator-facing, mine), re-deriving every leg from source at b5f2b10 rather than
trusting the verdict.

- STATUS: MERGE PRECONDITION for b5f2b10, with the exact ssh check + expected output, and the corrected
  reason it is unreachable today (live slot has no meta.json — NOT "F-redfix-4 is closed").
- Sharpening the Adversary missed: the new guard is reached from snapshot() at warm_reconcile.py:514 as
  well as restore() at :536. Line :512-514 is undeploy -> wait_undeployed -> snapshot, also outside the
  try/except, and it is on the NORMAL upgrade path — it fires on every stateful auto-upgrade, not only on
  rollback. B-redfix-5 as filed named only the rollback site.
- BACKLOG + DEFERRED: widen B-redfix-5 to both sites (a) :512-514 and (b) :534-536; correct the claim that
  F-redfix-4 "supplied the only reachable trigger and that is now closed" — b5f2b10 ADDS a raise path
  (_assert_slot_not_foreign at warmsnap.py:158 and :209).

Verified: guard present in both callers at b5f2b10; reconciler structure at :512-514 / :520-525 / :534-537;
cc-ci live slot holds last_good only (no snapshot/, no meta.json, no canon-* slot). Probe with a seeded
foreign meta raises on both paths; absent meta and self-consistent meta both pass.

No DoD item touched. M1+M2 PASS stand, ## DONE stands, no VETO. Docs-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZmH4rVNue3irZ6EsZzavb
This commit is contained in:
autonomic-bot
2026-07-09 08:16:06 +00:00
parent 3131e39f64
commit 93614f114e
4 changed files with 113 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
machine-docs/BACKLOG-redfix.md
View File

@ -58,17 +58,26 @@ hold). Concrete fix designs from M1 evidence:
`_assert_slot_not_foreign()` guard. Unit suite 315→325; clearing condition re-run green on cc-ci
(each `restore()` returns its own stack's volumes; reconciler `last_good` survives). Verify per
STATUS-redfix.md "Gate: M2 RE-CLAIMED".
- [ ] **B-redfix-5 — reconciler rollback `restore()` is outside the upgrade's `try/except`** (NOT blocking;
NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped). In
`warm_reconcile.py`, the unhealthy-rollback path runs `abra.undeploy(domain)` → `wait_undeployed` →
`warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits outside the `try/except` that
guards the upgrade, so if it raises for ANY reason (absent/corrupt snapshot, docker error) the
exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**.
F-redfix-4 supplied one way to make `restore()` raise (the shared slot) and that is now fixed, but the
structural gap predates it. Remedy sketch: wrap the rollback so a restore failure still redeploys
`last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a
breadcrumb rather than dying mid-rollback). Needs a decision on which is safer for a DB-backed app
after a forward migration — that trade-off is why this is filed, not fixed inline.
- [ ] **B-redfix-5 — reconciler's post-`undeploy` warmsnap calls are outside the upgrade's `try/except`**
(NOT blocking; NOT part of F-redfix-4's clearing condition; recorded so it is not silently dropped).
In `warm_reconcile.py` there are **two** such sites, both after `abra.undeploy(domain)` and both
outside the `try/except` that guards the upgrade:
**(a) upgrade path** `:512-514` — `abra.undeploy` → `wait_undeployed` → `warmsnap.snapshot(...)`;
**(b) rollback path** `:534-536` — `abra.undeploy` → `wait_undeployed` → `warmsnap.restore(...)` →
`deploy_version(last_good)` at `:537`.
If either raises for ANY reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes
found") the exception propagates, the following `deploy_version` never runs, and live keycloak is left
**undeployed**. Site (a) is on the *normal* upgrade path — it does not need a rollback to fire.
**Reachability, corrected (wake #42):** F-redfix-4's fix does not remove the trigger, it *adds* one —
`_assert_slot_not_foreign()` is a new raise inside BOTH `snapshot()` (`warmsnap.py:158`) and
`restore()` (`:209`). It is unreachable on cc-ci **today** because the live slot holds no
`snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that
trade-off is why this is filed, not fixed inline.
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15

25
machine-docs/DEFERRED.md
View File

@ -430,15 +430,24 @@ reachable via the operator/dev STAGES escape — production drone runs always ru
## B-redfix-5 — warm reconciler's rollback `restore()` is outside the upgrade's `try/except`
- **What:** in `runner/warm_reconcile.py`, the unhealthy-rollback path is `abra.undeploy(domain)` →
`wait_undeployed` → `warmsnap.restore(...)` → `deploy_version(last_good)`. `restore()` sits OUTSIDE the
`try/except` guarding the upgrade, so if it raises for any reason (absent/corrupt snapshot, docker error)
the exception propagates and `deploy_version(last_good)` never runs — live keycloak is left **undeployed**,
taking down the shared OIDC provider `lasuite-*`/`drone` depend on.
- **What:** in `runner/warm_reconcile.py`, **two** post-`abra.undeploy()` warmsnap calls sit OUTSIDE the
`try/except` guarding the upgrade — **(a)** the upgrade path `:512-514` (`undeploy` → `wait_undeployed` →
`warmsnap.snapshot(...)`) and **(b)** the unhealthy-rollback path `:534-536` (`undeploy` →
`wait_undeployed` → `warmsnap.restore(...)` → `deploy_version(last_good)` at `:537`). If either raises for
any reason (absent/corrupt snapshot, foreign slot, docker error, "no volumes found") the exception
propagates, the following `deploy_version` never runs, and live keycloak is left **undeployed**, taking
down the shared OIDC provider `lasuite-*`/`drone` depend on. Site (a) fires on the *normal* upgrade path —
no rollback required.
- **Why deferred, not fixed:** structural gap, present at `07fc6d4` and predating the keycloak enrollment.
F-redfix-4 supplied the only *reachable* trigger (the shared snapshot slot) and that is now closed at
`b5f2b10`. Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is
**not a VETO** and was "correctly filed rather than silently fixed".
Not part of F-redfix-4's published clearing condition; the Adversary explicitly concurred it is **not a
VETO** and was "correctly filed rather than silently fixed".
- **Reachability, corrected (wake #42).** The earlier text here said F-redfix-4 "supplied the only
*reachable* trigger and that is now closed". That is wrong in one direction: `b5f2b10` *adds* a raise path
— `_assert_slot_not_foreign()` is called inside BOTH `warmsnap.snapshot()` (`:158`) and `warmsnap.restore()`
(`:209`). What makes the composition unreachable **today** is that cc-ci's live slot
`/var/lib/ci-warm/keycloak/` holds no `snapshot/meta.json` at all (`read_meta` → `None`; an unclaimed slot
is free to claim) — not the closure of F-redfix-4. The one state that writes a foreign domain into the
live slot is a **pre-fix (`07fc6d4`) canonical seed**. Recorded as a MERGE PRECONDITION in STATUS-redfix.md.
- **Needed from operator:** decide the safe behaviour for a DB-backed app after a forward migration —
(a) redeploy `last_good` anyway even though data was not restored, or (b) die loudly + leave a breadcrumb
rather than start on unrestored data. That trade-off is a real safety call, not a mechanical fix, which is

50
machine-docs/JOURNAL-redfix.md
View File

@ -1669,3 +1669,53 @@ differs from `3fcea78925015fc9`.
**Unrelated, surfaced not touched:** an untracked `main.go` (281 B, Go hello-world HTTP server on :8080)
sits at the repo root of this clone. Not mine, foreign to this Nix repo, absent from git and from
`origin/main`. Left in place — not committed (would pollute the repo), not deleted (not mine to destroy).
---
## Wake #42 — 2026-07-09 — watchdog ping on the Adversary's wake-#41 verdict
Pulled `cf1455f`. Verdict is a CONFIRM of the merge guidance (`b5f2b10`, not `07fc6d4`), no new finding, no
VETO, M1+M2 PASS stand. So: nothing to unblock, no gate to re-claim. But the verdict carried a *merge
precondition* that existed only in REVIEW-redfix.md, which is Adversary-owned and which the operator has no
reason to read before merging. STATUS is the operator-facing artifact I own. Moving it across is the work.
I did not take the Adversary's word for it. Re-derived every leg from source at `b5f2b10` in a detached
worktree, plus one live check on the node:
1. `restore()` really does call the new guard (`warmsnap.py:209`), and `snapshot()` calls it too (`:158`).
So `b5f2b10` *adds* a raise path to warmsnap — it does not only remove one.
2. `warm_reconcile.py` rollback: `abra.undeploy` → `wait_undeployed` → `restore` → `deploy_version` at
`:534-537`, all outside the `try/except` at `:520-525`. Confirmed — this is B-redfix-5 as filed.
3. cc-ci live slot: `ls -A /var/lib/ci-warm/keycloak/` → `last_good` only; no `snapshot/`, no `meta.json`;
no `canon-*` slot. So `read_meta` → `None` → guard passes. Composition unreachable **today**.
Then the part the Adversary understated. It framed the risk as "B-redfix-5 × the new guard can wedge the
live OIDC provider **on rollback**". But the guard is also reached via `snapshot()` at `warm_reconcile.py:514`
— and `:512-514` is `abra.undeploy` → `wait_undeployed` → `snapshot`, likewise outside the try/except. That
site is on the **normal upgrade path**: it fires on every stateful auto-upgrade, no unhealthy release
required. So the blast radius of the precondition is strictly larger than "on rollback", and B-redfix-5 as
originally filed named only one of the two sites. Widened it in BACKLOG + DEFERRED.
Proving it needed care, and my first probe was wrong. Seeding a foreign `meta.json` and calling `restore()`
gave `FileNotFoundError: 'docker'`, not `SnapshotError` — I briefly read that as "the guard isn't wired into
restore after all", which would have contradicted the Adversary. It didn't: `restore()` runs
`_assert_undeployed(domain)` at `:205` *before* the guard at `:209`, and that shells out to docker, which is
absent on the orchestrator. The probe died upstream of the thing under test. `snapshot()` puts the guard
first (`:158`, before `_assert_undeployed`), which is why that half raised cleanly and the asymmetry looked
like a code difference rather than a probe artifact. Re-ran with the two docker-dependent preconditions
stubbed and the guard itself untouched: both paths raise on a foreign meta; an absent meta passes; a
self-consistent meta (`domain == warm-keycloak.ci.commoninternet.net`) passes. That is the real behaviour.
Same class of error as the `e3b0c44298fc1c14` empty-input tell from wake #39 — a probe that fails *before*
reaching the assertion prints something that reads like a finding. Worth the note: on this orchestrator
`docker`, `curl`, `wget`, `awk`, `pytest` are all absent, and `python3` is not on `PATH` at the store path
STATUS's verify-(1) cites. Reach for the stdlib and check what actually executed.
Corrected the reachability wording in BACKLOG + DEFERRED, which both claimed F-redfix-4 "supplied the only
*reachable* trigger and that is now closed". The trigger is not closed — it is unreachable for a *different*
reason (the live slot is empty), and that reason is a property of node state, not of the fix. Stating it the
old way would let an operator conclude the composition is impossible by construction. It isn't: one pre-fix
canonical seed writes `domain=warm-canon-keycloak…` into the live slot and arms both wedge sites.
No VETO to answer, no DoD item touched, no gate to claim. Docs-only. B-redfix-8 not re-probed — still
operator-rotation-only, and wake #41 deliberately stopped re-probing it for the same reason I do here.

29
machine-docs/STATUS-redfix.md
View File

@ -70,9 +70,32 @@ F-redfix-4 and is re-asserted here only after the remedy was Adversary-verified.
and the still-open non-blocking B-redfix-5, are below.
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
**B-redfix-5** in BACKLOG-redfix.md — `warm_reconcile.py`'s rollback `warmsnap.restore()` sits outside the
upgrade's `try/except`. Pre-dates the enrollment (present at `07fc6d4`); F-redfix-4 supplied the only
*reachable* trigger and that is now closed. Adversary concurred it is not a VETO.
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the
enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO.
**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both
of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in
`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one
passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path
this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today**
because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free
to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed.
The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which
wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. So: do not run a canonical
keycloak seed from `07fc6d4` between now and the merge.
**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports
`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is
anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5
(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording
`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard.
---