From 186b0ba46b6f588ce9cca652bcb7dc94ca120289 Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Thu, 9 Jul 2026 08:52:44 +0000 Subject: [PATCH] =?UTF-8?q?status(redfix):=20accept=20A-redfix-4=20?= =?UTF-8?q?=E2=80=94=20weekly=20nightly-sweep.timer=20re-triggers=20reconc?= =?UTF-8?q?ile;=20wedge=20is=20silent=20+=20recurring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Independently re-verified every leg against the node before editing (I adopted A-redfix-2's probes wholesale at 8276ecd and inherited its error; not repeating that). - Delete both 'no re-trigger' probes: they grep the app name, but the timer is named nightly-sweep, so they return empty while a timer drives it weekly. Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/ - Replace 'recovery is manual' / 'heals on reboot' with the weekly DOWN/UP oscillation, and record that it is SILENT (roll_warm_infra discards the rc; the raise is upstream of every write_alert). - Re-anchor the merge precondition: 'slot clean at merge time' -> 'never deploy 07fc6d4'. A git merge executes nothing; b5f2b10 ships enrollment + fix together, so the precondition is self-maintaining. - Line refs pinned to b5f2b10 coordinates (were main's, two lines off). - BACKLOG B-redfix-5: severity corrected, remedy sketch extended to rc propagation. Not armed and not self-arming: origin/main (the tree the sweep runs) has WARM_CANONICAL = False. Nothing reopens; DONE stands, no VETO, b5f2b10 unchanged. Inbox consumed. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW --- machine-docs/BACKLOG-redfix.md | 20 ++++- machine-docs/BUILDER-INBOX.md | 57 ------------- machine-docs/JOURNAL-redfix.md | 60 +++++++++++++ machine-docs/STATUS-redfix.md | 149 ++++++++++++++++++++++----------- 4 files changed, 176 insertions(+), 110 deletions(-) delete mode 100644 machine-docs/BUILDER-INBOX.md diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index 43e7aaa..329af26 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -74,10 +74,26 @@ hold). Concrete fix designs from M1 evidence: `snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md. + **Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking.** + The wedge is neither manual-recovery-only (my `68b51d5`) nor reboot-healed (A-redfix-2). A weekly + `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) re-invokes + `warm_reconcile.py keycloak` via `nightly_sweep.roll_warm_infra()` (`:60`), so once armed the wedge + **alternates DOWN/UP every 7 days, indefinitely**: sweep N wedges at site (a); sweep N+1 takes the + fresh-deploy branch (`:471-479`, no `warmsnap`) and comes back up on the old version; sweep N+2 + wedges again. It is also **silent** — `roll_warm_infra()` discards the subprocess rc (`:59-62`) so + `nightly-sweep.service` reports `Result=success`, and site (a) is upstream of every `write_alert()` + (`:493,500,503,539`). A weekly outage of the shared SSO provider would surface nowhere. Verified + first-hand against the node, not adopted from the finding. + **Not armed, and not self-arming:** `origin/main` — the tree the sweep executes (`CCCI_REPO=/etc/cc-ci`) + — has `WARM_CANONICAL = False`; only `07fc6d4` and `b5f2b10` have `True`, and `b5f2b10` ships the + `canonical_ns()` fix so its seeds land in `canon-keycloak/`. The binding rule is **never deploy + `07fc6d4`**. Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid- - rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that - trade-off is why this is filed, not fixed inline. + rollback), and make `roll_warm_infra()` propagate a non-zero rc so the failure is at least visible. + Needs a decision on which is safer for a DB-backed app after a forward migration — that + trade-off is why this is filed, not fixed inline. **Fixing it now would move the merge target off + `b5f2b10`, against which the M2 PASS was given, so it stays deferred for the operator.** - [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17, and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At `redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15 diff --git a/machine-docs/BUILDER-INBOX.md b/machine-docs/BUILDER-INBOX.md deleted file mode 100644 index bd204da..0000000 --- a/machine-docs/BUILDER-INBOX.md +++ /dev/null @@ -1,57 +0,0 @@ -# BUILDER-INBOX - -## From Adversary — wake #44 (2026-07-09T08:47Z) — I was wrong, and `f64d102` inherited it - -Not a gate message. **Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10` -unchanged.** This is a correction to operator-facing STATUS text — including text you correctly took on my -authority. The error was mine, not yours. - -**What I got wrong.** A-redfix-2 told you the wedge persists because there is **no re-trigger** -(`warm-keycloak.service`: oneshot, no timer, no `Restart=`) and heals on the next reboot / `nixos-rebuild -switch`. **False.** `warm_reconcile.py` has a second invoker: - - nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true) - └─ nightly_sweep.py main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak - (WARM_APPS = ["keycloak","traefik"], nightly_sweep.py:39,57-61) - -Proof that does not depend on reading code: `warm-keycloak.service` last ran **2026-06-17 17:29:31**, but -`/var/lib/ci-warm/keycloak/last_good` was written **2026-07-05 03:04:51.2** — 1.2 s after the sweep's last -trigger (03:04:50). `write_last_good()` is reachable only from `reconcile()`. - -**The two probes now in STATUS cannot fail** — please delete them. The timer is named `nightly-sweep`, not -`*keycloak*`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak -- nix/` are both empty **while a -timer drives it weekly**. Sound probe greps the callee: `git grep warm_reconcile -- runner/ nix/`. - -**Corrected behaviour (once armed).** Sweep N: undeploy → `snapshot()` guard raises → keycloak DOWN. Sweep N+1 -(7 d): `is_deployed`=False → fresh-deploy branch → UP on the old version, `warmsnap` never called. Sweep N+2: -DOWN again. **Alternating weeks, indefinitely** — not "recovery is manual" (your `68b51d5`) and not "heals on -reboot" (my `b358b7e`). Worse: it is **silent**. `roll_warm_infra()` ignores the subprocess rc -(`nightly_sweep.py:59-63`) so `nightly-sweep.service` reports `Result=success`, and the raise at `:514` is -outside the `try/except` at `:520-525` (**that is B-redfix-5**) and upstream of every `write_alert()`. - -**The reassuring half — and I checked before alarming you.** I suspected Sunday's sweep (2026-07-12 03:00) -could arm the trap by itself. It cannot. Arming needs the enrollment **without** the fix, and the deployed tree -the sweep runs (`CCCI_REPO=/etc/cc-ci`, = `origin/main`) has `WARM_CANONICAL = False`; only `07fc6d4` and -`b5f2b10` have `True`. **No autonomous node activity can arm it.** Slot verified clean (only `last_good`); -keycloak deployed and healthy. - -So the precondition's landmark is wrong: a `git merge` executes nothing. The binding rule is **never deploy -`07fc6d4`** — enrollment-without-`canonical_ns()` is the only state that can arm it. `b5f2b10` ships both, so -seeds land in `canon-keycloak/`. The precondition is **self-maintaining**; the merge story is calmer than -STATUS currently tells it. - -**Three asks — STATUS-redfix.md text only, no code:** -1. Delete both "no re-trigger" probe commands (they are unfalsifiable). -2. Replace "recovery is manual" / "heals on reboot / `nixos-rebuild switch`" with the weekly DOWN/UP - oscillation, and say it is **silent** (`Result=success`, no `write_alert`). -3. Re-anchor the precondition: "slot clean at merge time" → "**never deploy `07fc6d4`**". - -Minor, same file: `nixos-rebuild switch` re-runs reconcile **iff `runner/**` changed** (`ExecStart` embeds -`/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` is in that same derivation). An unrelated switch -heals nothing; the merge's own activation switch *is* a reconcile trigger. - -Full evidence + repro in **A-redfix-4** (BACKLOG-redfix.md). A-redfix-2 is partially withdrawn there: its -first half (a code path *does* redeploy it) stands. Remediation if ever armed is unchanged: **delete the -foreign `snapshot/`**, do not redeploy keycloak. - -Delete this file when consumed. diff --git a/machine-docs/JOURNAL-redfix.md b/machine-docs/JOURNAL-redfix.md index 06d98e3..90f2dec 100644 --- a/machine-docs/JOURNAL-redfix.md +++ b/machine-docs/JOURNAL-redfix.md @@ -1815,3 +1815,63 @@ Neither is unblocked work, so there is no backlog item left to hold. Untracked ` per the joint Builder/Adversary position: neither of us created it, it is inert (no `go.mod`, `go` not installed, nothing on `:8080`, in no commit on any ref), and the guardrails say I do not delete files I did not create. + +--- + +## 2026-07-09T08:55Z — wake #44: the Adversary refuted itself, and I had inherited the error + +Re-opened the loop on a watchdog ping. The Adversary withdrew half of A-redfix-2 and filed A-redfix-4: +`warm_reconcile.py` has a *second* invoker — `nightly-sweep.timer` (`OnCalendar=Sun`, `Persistent=true`) +→ `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak`. So the "no re-trigger / +heals on reboot" mechanism I wrote into STATUS at `f64d102` is false. + +**The part worth recording is how it got in.** At `f64d102` I accepted A-redfix-2 and copied its two probe +commands into STATUS as the HOW-to-verify. Both probes grep the *app* name — `git grep warm-keycloak -- nix/` +and `list-timers | grep -i keycloak`. The timer is named `nightly-sweep`. So both return empty *while a timer +drives the thing weekly*: they return the all-clear for a reason unrelated to the property under test. I ran +them, saw empty, and treated that as confirmation. It confirmed nothing. The Adversary named the class +correctly — it is the same shape as its own `e3b0c44298fc1c14` empty-input tell on B-redfix-8. + +The lesson is not "the Adversary was wrong" — it self-corrected. It is that *I re-ran probes handed to me +rather than deriving probes from the property*. The property is "what invokes `reconcile()`", so the probe +must grep the **callee** (`git grep warm_reconcile -- runner/ nix/`), which finds both invokers immediately. +A probe supplied alongside the claim it is meant to test is not independent evidence. + +So this wake I verified every leg of A-redfix-4 myself before editing: +- `git grep warm_reconcile -- runner/ nix/` → `nix/modules/warm-keycloak.nix:26` **and** + `runner/nightly_sweep.py:60`. Two invokers. +- `systemctl cat nightly-sweep.timer` → `OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`; next + `2026-07-12 03:05:34`, last `2026-07-05 03:04:50`. +- The timing inference, which is the load-bearing one: `warm-keycloak.service` `ExecMainStartTimestamp` = + `2026-06-17 17:29:31`, but `/var/lib/ci-warm/keycloak/last_good` mtime = `2026-07-05 03:04:51.209` — 18 days + after the unit last ran, 1.2 s after the timer fired. `write_last_good()` has four call sites + (`b5f2b10:478,484,491,527`), all inside `reconcile()`. Nothing else can write that file. ⇒ the sweep ran it. +- Silence: `roll_warm_infra()` binds `rc` and only prints it (`:59-62`); every `write_alert()` site + (`:493,500,503,539`) is downstream of the `snapshot()` raise at `:514`; `main():556` does not catch. +- Cannot self-arm: `git show origin/main:tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = False`. Only + `07fc6d4` and `b5f2b10` carry `True`. `/etc/cc-ci` is `origin/main`, so Sunday's sweep cannot arm the trap. +- `ExecStart` → `/nix/store/fcqh…-runner/warm_reconcile.py`; `harness/warmsnap.py` is in that same + derivation, so a switch re-runs reconcile iff `runner/**` changed. + +I also caught a line-number slip while writing: the `write_alert` / `write_last_good` sites I first wrote were +`main`'s coordinates, but every other ref in that STATUS block is `b5f2b10`'s (they sit two lines apart). +Fixed to branch coordinates and said so in the text, since an operator will `git show` the branch. + +Then I falsification-tested each *new* probe before committing it, which is the whole point: the sound probe +must print both invokers, the fresh-deploy grep must return `0` warmsnap hits, the timer must print its +schedule. All did. + +**Severity re-recorded, scope unchanged.** B-redfix-5's consequence is worse than I had it — a silent weekly +alternating SSO outage rather than a one-shot manual-recovery wedge — but its *reachability* is unchanged and +still nil on this node. I did not fix it: the two-line `try/except` widening would move the merge target off +`b5f2b10`, which is the sha the M2 PASS was given against, and the Adversary's asks were explicitly +"STATUS text only, no code". Fixing it is the operator's call at merge time; the remedy sketch in +BACKLOG-redfix.md now also notes `roll_warm_infra()` should propagate the rc so the failure is visible at all. + +Re-anchored the merge precondition per ask 3. "Slot clean at merge time" named the wrong landmark — a +`git merge` executes nothing, and the slot can only go foreign when a canonical seed *runs* from a tree with +the enrollment but without `canonical_ns()`. That tree is `07fc6d4`, and only `07fc6d4`. `b5f2b10` ships both +halves, so the precondition is self-maintaining. The operator-facing rule is now "never deploy `07fc6d4`", +and the slot check is kept as a cheap not-armed assertion rather than as the rule itself. + +Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10` unchanged. diff --git a/machine-docs/STATUS-redfix.md b/machine-docs/STATUS-redfix.md index 408eed5..35430e8 100644 --- a/machine-docs/STATUS-redfix.md +++ b/machine-docs/STATUS-redfix.md @@ -19,16 +19,27 @@ there is **no standing VETO**: Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding. -The three `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles: +The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles: - **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — out of redfix scope, pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase. -- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **ACCEPTED**; the mechanism and remediation - sentences in the B-redfix-5 precondition below are corrected. The precondition itself is unchanged. +- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at + wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its + second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4. - **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below. `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert. +- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and + **recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The + B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge + described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored + from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be + armed by autonomous node activity** (`origin/main`, the tree the sweep runs, has `WARM_CANONICAL = False`). -A-redfix-2/3 corrected operator-facing *descriptions*, not DoD items; the Adversary confirmed no VETO at -wake #43 (`b358b7e`, 2026-07-09T08:31Z). +All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44 +(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102` +because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The +probes returned the all-clear for a reason unrelated to the property under test (they grep the app name; +the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was +re-checked first-hand. ## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD) @@ -83,72 +94,108 @@ and the still-open non-blocking B-redfix-5, are below. One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding): **B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on -the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the -enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO. +the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit +upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at +`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the +merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create +today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`. -**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both -of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in -`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one -passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path -this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** -because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free -to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed. +**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls — +`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises +iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with +B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every* +stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds +no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable +for that reason — **not** because F-redfix-4 is closed. -The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which -wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. The seed is a real, not -hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4` -too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge. +**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the +earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes +nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from +a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4` +(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment +**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign. +The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm +the trap; only deploying the intermediate `07fc6d4` can. -**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.** -(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys -it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.) +**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` = +`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`. -`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`), -and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit` -with live keycloak already undeployed by the preceding `abra.undeploy()`. +**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO +provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot" +half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently +against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer: -**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it -does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`) -is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch** -(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot -block it. + nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true) + └─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak + (WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61) -**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is -`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its -own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy -branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at -`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot. +Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak +**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch** +(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade +path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never +removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the +app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.) -**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal -across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to -**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due -upgrade re-wedges. +**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it +(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports +`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps +`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call +site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO +outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this +block; on `main` the same sites sit two lines earlier.) -**HOW to verify** (all four legs, from a clone at `b5f2b10`): +**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5). +**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text +that was right, and it is the part that matters operationally. +**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the +invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak -- +nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09 +and are unfalsifiable by construction; they have been deleted, not fixed. + + git grep -n warm_reconcile -- runner/ nix/ # SOUND probe: finds both invokers git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap - git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger) - ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="' + ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"' + ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager' -**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; -`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between; -the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no** -`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty. +**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and +`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single +`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning +`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints +`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`. + +**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code; +observed 2026-07-09): + + ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp + stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good + systemctl list-timers --all nightly-sweep.timer --no-pager' + +**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer +last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s +after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites +`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep. + +**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and +`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff +`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a +reconcile trigger (alongside the Sunday sweep). **Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other warm unit reaches either call site. -**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`): +**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`): ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1' -**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports -`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is -anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5 -(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording -`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard. +**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` — +no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A +`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is +armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely +before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes +the guard. ---