status(redfix): accept A-redfix-4 — weekly nightly-sweep.timer re-triggers reconcile; wedge is silent + recurring
Independently re-verified every leg against the node before editing (I adopted A-redfix-2's probes wholesale at8276ecdand inherited its error; not repeating that). - Delete both 'no re-trigger' probes: they grep the app name, but the timer is named nightly-sweep, so they return empty while a timer drives it weekly. Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/ - Replace 'recovery is manual' / 'heals on reboot' with the weekly DOWN/UP oscillation, and record that it is SILENT (roll_warm_infra discards the rc; the raise is upstream of every write_alert). - Re-anchor the merge precondition: 'slot clean at merge time' -> 'never deploy 07fc6d4'. A git merge executes nothing;b5f2b10ships enrollment + fix together, so the precondition is self-maintaining. - Line refs pinned tob5f2b10coordinates (were main's, two lines off). - BACKLOG B-redfix-5: severity corrected, remedy sketch extended to rc propagation. Not armed and not self-arming: origin/main (the tree the sweep runs) has WARM_CANONICAL = False. Nothing reopens; DONE stands, no VETO,b5f2b10unchanged. Inbox consumed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW
This commit is contained in:
@ -74,10 +74,26 @@ hold). Concrete fix designs from M1 evidence:
|
|||||||
`snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
|
`snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
|
||||||
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
|
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
|
||||||
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
|
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
|
||||||
|
**Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking.**
|
||||||
|
The wedge is neither manual-recovery-only (my `68b51d5`) nor reboot-healed (A-redfix-2). A weekly
|
||||||
|
`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) re-invokes
|
||||||
|
`warm_reconcile.py keycloak` via `nightly_sweep.roll_warm_infra()` (`:60`), so once armed the wedge
|
||||||
|
**alternates DOWN/UP every 7 days, indefinitely**: sweep N wedges at site (a); sweep N+1 takes the
|
||||||
|
fresh-deploy branch (`:471-479`, no `warmsnap`) and comes back up on the old version; sweep N+2
|
||||||
|
wedges again. It is also **silent** — `roll_warm_infra()` discards the subprocess rc (`:59-62`) so
|
||||||
|
`nightly-sweep.service` reports `Result=success`, and site (a) is upstream of every `write_alert()`
|
||||||
|
(`:493,500,503,539`). A weekly outage of the shared SSO provider would surface nowhere. Verified
|
||||||
|
first-hand against the node, not adopted from the finding.
|
||||||
|
**Not armed, and not self-arming:** `origin/main` — the tree the sweep executes (`CCCI_REPO=/etc/cc-ci`)
|
||||||
|
— has `WARM_CANONICAL = False`; only `07fc6d4` and `b5f2b10` have `True`, and `b5f2b10` ships the
|
||||||
|
`canonical_ns()` fix so its seeds land in `canon-keycloak/`. The binding rule is **never deploy
|
||||||
|
`07fc6d4`**.
|
||||||
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
|
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
|
||||||
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
|
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
|
||||||
rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that
|
rollback), and make `roll_warm_infra()` propagate a non-zero rc so the failure is at least visible.
|
||||||
trade-off is why this is filed, not fixed inline.
|
Needs a decision on which is safer for a DB-backed app after a forward migration — that
|
||||||
|
trade-off is why this is filed, not fixed inline. **Fixing it now would move the merge target off
|
||||||
|
`b5f2b10`, against which the M2 PASS was given, so it stays deferred for the operator.**
|
||||||
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
|
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
|
||||||
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
|
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
|
||||||
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
|
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
|
||||||
|
|||||||
@ -1,57 +0,0 @@
|
|||||||
# BUILDER-INBOX
|
|
||||||
|
|
||||||
## From Adversary — wake #44 (2026-07-09T08:47Z) — I was wrong, and `f64d102` inherited it
|
|
||||||
|
|
||||||
Not a gate message. **Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10`
|
|
||||||
unchanged.** This is a correction to operator-facing STATUS text — including text you correctly took on my
|
|
||||||
authority. The error was mine, not yours.
|
|
||||||
|
|
||||||
**What I got wrong.** A-redfix-2 told you the wedge persists because there is **no re-trigger**
|
|
||||||
(`warm-keycloak.service`: oneshot, no timer, no `Restart=`) and heals on the next reboot / `nixos-rebuild
|
|
||||||
switch`. **False.** `warm_reconcile.py` has a second invoker:
|
|
||||||
|
|
||||||
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
|
||||||
└─ nightly_sweep.py main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
|
||||||
(WARM_APPS = ["keycloak","traefik"], nightly_sweep.py:39,57-61)
|
|
||||||
|
|
||||||
Proof that does not depend on reading code: `warm-keycloak.service` last ran **2026-06-17 17:29:31**, but
|
|
||||||
`/var/lib/ci-warm/keycloak/last_good` was written **2026-07-05 03:04:51.2** — 1.2 s after the sweep's last
|
|
||||||
trigger (03:04:50). `write_last_good()` is reachable only from `reconcile()`.
|
|
||||||
|
|
||||||
**The two probes now in STATUS cannot fail** — please delete them. The timer is named `nightly-sweep`, not
|
|
||||||
`*keycloak*`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak -- nix/` are both empty **while a
|
|
||||||
timer drives it weekly**. Sound probe greps the callee: `git grep warm_reconcile -- runner/ nix/`.
|
|
||||||
|
|
||||||
**Corrected behaviour (once armed).** Sweep N: undeploy → `snapshot()` guard raises → keycloak DOWN. Sweep N+1
|
|
||||||
(7 d): `is_deployed`=False → fresh-deploy branch → UP on the old version, `warmsnap` never called. Sweep N+2:
|
|
||||||
DOWN again. **Alternating weeks, indefinitely** — not "recovery is manual" (your `68b51d5`) and not "heals on
|
|
||||||
reboot" (my `b358b7e`). Worse: it is **silent**. `roll_warm_infra()` ignores the subprocess rc
|
|
||||||
(`nightly_sweep.py:59-63`) so `nightly-sweep.service` reports `Result=success`, and the raise at `:514` is
|
|
||||||
outside the `try/except` at `:520-525` (**that is B-redfix-5**) and upstream of every `write_alert()`.
|
|
||||||
|
|
||||||
**The reassuring half — and I checked before alarming you.** I suspected Sunday's sweep (2026-07-12 03:00)
|
|
||||||
could arm the trap by itself. It cannot. Arming needs the enrollment **without** the fix, and the deployed tree
|
|
||||||
the sweep runs (`CCCI_REPO=/etc/cc-ci`, = `origin/main`) has `WARM_CANONICAL = False`; only `07fc6d4` and
|
|
||||||
`b5f2b10` have `True`. **No autonomous node activity can arm it.** Slot verified clean (only `last_good`);
|
|
||||||
keycloak deployed and healthy.
|
|
||||||
|
|
||||||
So the precondition's landmark is wrong: a `git merge` executes nothing. The binding rule is **never deploy
|
|
||||||
`07fc6d4`** — enrollment-without-`canonical_ns()` is the only state that can arm it. `b5f2b10` ships both, so
|
|
||||||
seeds land in `canon-keycloak/`. The precondition is **self-maintaining**; the merge story is calmer than
|
|
||||||
STATUS currently tells it.
|
|
||||||
|
|
||||||
**Three asks — STATUS-redfix.md text only, no code:**
|
|
||||||
1. Delete both "no re-trigger" probe commands (they are unfalsifiable).
|
|
||||||
2. Replace "recovery is manual" / "heals on reboot / `nixos-rebuild switch`" with the weekly DOWN/UP
|
|
||||||
oscillation, and say it is **silent** (`Result=success`, no `write_alert`).
|
|
||||||
3. Re-anchor the precondition: "slot clean at merge time" → "**never deploy `07fc6d4`**".
|
|
||||||
|
|
||||||
Minor, same file: `nixos-rebuild switch` re-runs reconcile **iff `runner/**` changed** (`ExecStart` embeds
|
|
||||||
`/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` is in that same derivation). An unrelated switch
|
|
||||||
heals nothing; the merge's own activation switch *is* a reconcile trigger.
|
|
||||||
|
|
||||||
Full evidence + repro in **A-redfix-4** (BACKLOG-redfix.md). A-redfix-2 is partially withdrawn there: its
|
|
||||||
first half (a code path *does* redeploy it) stands. Remediation if ever armed is unchanged: **delete the
|
|
||||||
foreign `snapshot/`**, do not redeploy keycloak.
|
|
||||||
|
|
||||||
Delete this file when consumed.
|
|
||||||
@ -1815,3 +1815,63 @@ Neither is unblocked work, so there is no backlog item left to hold. Untracked `
|
|||||||
per the joint Builder/Adversary position: neither of us created it, it is inert (no `go.mod`, `go` not
|
per the joint Builder/Adversary position: neither of us created it, it is inert (no `go.mod`, `go` not
|
||||||
installed, nothing on `:8080`, in no commit on any ref), and the guardrails say I do not delete files I
|
installed, nothing on `:8080`, in no commit on any ref), and the guardrails say I do not delete files I
|
||||||
did not create.
|
did not create.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-07-09T08:55Z — wake #44: the Adversary refuted itself, and I had inherited the error
|
||||||
|
|
||||||
|
Re-opened the loop on a watchdog ping. The Adversary withdrew half of A-redfix-2 and filed A-redfix-4:
|
||||||
|
`warm_reconcile.py` has a *second* invoker — `nightly-sweep.timer` (`OnCalendar=Sun`, `Persistent=true`)
|
||||||
|
→ `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak`. So the "no re-trigger /
|
||||||
|
heals on reboot" mechanism I wrote into STATUS at `f64d102` is false.
|
||||||
|
|
||||||
|
**The part worth recording is how it got in.** At `f64d102` I accepted A-redfix-2 and copied its two probe
|
||||||
|
commands into STATUS as the HOW-to-verify. Both probes grep the *app* name — `git grep warm-keycloak -- nix/`
|
||||||
|
and `list-timers | grep -i keycloak`. The timer is named `nightly-sweep`. So both return empty *while a timer
|
||||||
|
drives the thing weekly*: they return the all-clear for a reason unrelated to the property under test. I ran
|
||||||
|
them, saw empty, and treated that as confirmation. It confirmed nothing. The Adversary named the class
|
||||||
|
correctly — it is the same shape as its own `e3b0c44298fc1c14` empty-input tell on B-redfix-8.
|
||||||
|
|
||||||
|
The lesson is not "the Adversary was wrong" — it self-corrected. It is that *I re-ran probes handed to me
|
||||||
|
rather than deriving probes from the property*. The property is "what invokes `reconcile()`", so the probe
|
||||||
|
must grep the **callee** (`git grep warm_reconcile -- runner/ nix/`), which finds both invokers immediately.
|
||||||
|
A probe supplied alongside the claim it is meant to test is not independent evidence.
|
||||||
|
|
||||||
|
So this wake I verified every leg of A-redfix-4 myself before editing:
|
||||||
|
- `git grep warm_reconcile -- runner/ nix/` → `nix/modules/warm-keycloak.nix:26` **and**
|
||||||
|
`runner/nightly_sweep.py:60`. Two invokers.
|
||||||
|
- `systemctl cat nightly-sweep.timer` → `OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`; next
|
||||||
|
`2026-07-12 03:05:34`, last `2026-07-05 03:04:50`.
|
||||||
|
- The timing inference, which is the load-bearing one: `warm-keycloak.service` `ExecMainStartTimestamp` =
|
||||||
|
`2026-06-17 17:29:31`, but `/var/lib/ci-warm/keycloak/last_good` mtime = `2026-07-05 03:04:51.209` — 18 days
|
||||||
|
after the unit last ran, 1.2 s after the timer fired. `write_last_good()` has four call sites
|
||||||
|
(`b5f2b10:478,484,491,527`), all inside `reconcile()`. Nothing else can write that file. ⇒ the sweep ran it.
|
||||||
|
- Silence: `roll_warm_infra()` binds `rc` and only prints it (`:59-62`); every `write_alert()` site
|
||||||
|
(`:493,500,503,539`) is downstream of the `snapshot()` raise at `:514`; `main():556` does not catch.
|
||||||
|
- Cannot self-arm: `git show origin/main:tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = False`. Only
|
||||||
|
`07fc6d4` and `b5f2b10` carry `True`. `/etc/cc-ci` is `origin/main`, so Sunday's sweep cannot arm the trap.
|
||||||
|
- `ExecStart` → `/nix/store/fcqh…-runner/warm_reconcile.py`; `harness/warmsnap.py` is in that same
|
||||||
|
derivation, so a switch re-runs reconcile iff `runner/**` changed.
|
||||||
|
|
||||||
|
I also caught a line-number slip while writing: the `write_alert` / `write_last_good` sites I first wrote were
|
||||||
|
`main`'s coordinates, but every other ref in that STATUS block is `b5f2b10`'s (they sit two lines apart).
|
||||||
|
Fixed to branch coordinates and said so in the text, since an operator will `git show` the branch.
|
||||||
|
|
||||||
|
Then I falsification-tested each *new* probe before committing it, which is the whole point: the sound probe
|
||||||
|
must print both invokers, the fresh-deploy grep must return `0` warmsnap hits, the timer must print its
|
||||||
|
schedule. All did.
|
||||||
|
|
||||||
|
**Severity re-recorded, scope unchanged.** B-redfix-5's consequence is worse than I had it — a silent weekly
|
||||||
|
alternating SSO outage rather than a one-shot manual-recovery wedge — but its *reachability* is unchanged and
|
||||||
|
still nil on this node. I did not fix it: the two-line `try/except` widening would move the merge target off
|
||||||
|
`b5f2b10`, which is the sha the M2 PASS was given against, and the Adversary's asks were explicitly
|
||||||
|
"STATUS text only, no code". Fixing it is the operator's call at merge time; the remedy sketch in
|
||||||
|
BACKLOG-redfix.md now also notes `roll_warm_infra()` should propagate the rc so the failure is visible at all.
|
||||||
|
|
||||||
|
Re-anchored the merge precondition per ask 3. "Slot clean at merge time" named the wrong landmark — a
|
||||||
|
`git merge` executes nothing, and the slot can only go foreign when a canonical seed *runs* from a tree with
|
||||||
|
the enrollment but without `canonical_ns()`. That tree is `07fc6d4`, and only `07fc6d4`. `b5f2b10` ships both
|
||||||
|
halves, so the precondition is self-maintaining. The operator-facing rule is now "never deploy `07fc6d4`",
|
||||||
|
and the slot check is kept as a cheap not-armed assertion rather than as the rule itself.
|
||||||
|
|
||||||
|
Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10` unchanged.
|
||||||
|
|||||||
@ -19,16 +19,27 @@ there is **no standing VETO**:
|
|||||||
|
|
||||||
Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
|
Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
|
||||||
|
|
||||||
The three `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
|
The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
|
||||||
- **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — out of redfix scope,
|
- **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — out of redfix scope,
|
||||||
pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
|
pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
|
||||||
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **ACCEPTED**; the mechanism and remediation
|
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
|
||||||
sentences in the B-redfix-5 precondition below are corrected. The precondition itself is unchanged.
|
wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
|
||||||
|
second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
|
||||||
- **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
|
- **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
|
||||||
`/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
|
`/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
|
||||||
|
- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
|
||||||
|
**recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
|
||||||
|
B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
|
||||||
|
described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
|
||||||
|
from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
|
||||||
|
armed by autonomous node activity** (`origin/main`, the tree the sweep runs, has `WARM_CANONICAL = False`).
|
||||||
|
|
||||||
A-redfix-2/3 corrected operator-facing *descriptions*, not DoD items; the Adversary confirmed no VETO at
|
All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
|
||||||
wake #43 (`b358b7e`, 2026-07-09T08:31Z).
|
(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
|
||||||
|
because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
|
||||||
|
probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
|
||||||
|
the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
|
||||||
|
re-checked first-hand.
|
||||||
|
|
||||||
## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
|
## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
|
||||||
|
|
||||||
@ -83,72 +94,108 @@ and the still-open non-blocking B-redfix-5, are below.
|
|||||||
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
|
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
|
||||||
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
|
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
|
||||||
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
|
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
|
||||||
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the
|
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
|
||||||
enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO.
|
upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
|
||||||
|
`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
|
||||||
|
merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
|
||||||
|
today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.
|
||||||
|
|
||||||
**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both
|
**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
|
||||||
of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in
|
`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
|
||||||
`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one
|
iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
|
||||||
passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path
|
B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
|
||||||
this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today**
|
stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
|
||||||
because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free
|
no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
|
||||||
to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed.
|
for that reason — **not** because F-redfix-4 is closed.
|
||||||
|
|
||||||
The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which
|
**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
|
||||||
wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. The seed is a real, not
|
earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
|
||||||
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
|
nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
|
||||||
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
|
a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
|
||||||
|
(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
|
||||||
|
**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
|
||||||
|
The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
|
||||||
|
the trap; only deploying the intermediate `07fc6d4` can.
|
||||||
|
|
||||||
**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
|
**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
|
||||||
(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
|
`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.
|
||||||
it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
|
|
||||||
|
|
||||||
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
|
**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
|
||||||
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
|
provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
|
||||||
with live keycloak already undeployed by the preceding `abra.undeploy()`.
|
half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
|
||||||
|
against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:
|
||||||
|
|
||||||
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
|
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
||||||
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
|
└─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
||||||
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
|
(WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)
|
||||||
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
|
|
||||||
block it.
|
|
||||||
|
|
||||||
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
|
Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
|
||||||
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
|
**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
|
||||||
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
|
(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
|
||||||
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
|
path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
|
||||||
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
|
removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
|
||||||
|
app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)
|
||||||
|
|
||||||
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
|
**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
|
||||||
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
|
(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
|
||||||
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
|
`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
|
||||||
upgrade re-wedges.
|
`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
|
||||||
|
site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
|
||||||
|
outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
|
||||||
|
block; on `main` the same sites sit two lines earlier.)
|
||||||
|
|
||||||
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
|
**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
|
||||||
|
**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
|
||||||
|
that was right, and it is the part that matters operationally.
|
||||||
|
|
||||||
|
**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
|
||||||
|
invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
|
||||||
|
nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
|
||||||
|
and are unfalsifiable by construction; they have been deleted, not fixed.
|
||||||
|
|
||||||
|
git grep -n warm_reconcile -- runner/ nix/ # SOUND probe: finds both invokers
|
||||||
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
||||||
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
||||||
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
|
ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
|
||||||
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
|
ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||||||
|
|
||||||
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
|
**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
|
||||||
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
|
`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
|
||||||
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
|
`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
|
||||||
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
|
`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
|
||||||
|
`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.
|
||||||
|
|
||||||
|
**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
|
||||||
|
observed 2026-07-09):
|
||||||
|
|
||||||
|
ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
|
||||||
|
stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
|
||||||
|
systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||||||
|
|
||||||
|
**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
|
||||||
|
last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
|
||||||
|
after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
|
||||||
|
`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.
|
||||||
|
|
||||||
|
**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
|
||||||
|
`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
|
||||||
|
`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
|
||||||
|
reconcile trigger (alongside the Sunday sweep).
|
||||||
|
|
||||||
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
||||||
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
||||||
warm unit reaches either call site.
|
warm unit reaches either call site.
|
||||||
|
|
||||||
**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
|
**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
|
||||||
|
|
||||||
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
|
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
|
||||||
|
|
||||||
**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports
|
**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
|
||||||
`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is
|
no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
|
||||||
anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5
|
`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
|
||||||
(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording
|
armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
|
||||||
`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard.
|
before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
|
||||||
|
the guard.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user