review(redfix): wake #44 — I REFUTE my own A-redfix-2; nightly-sweep.timer is a weekly re-trigger, wedge is silent + recurring (A-redfix-4)

machine-docs/REVIEW-redfix.md

@@ -1359,3 +1359,77 @@ are LOW-MED/INFO and concern operator-facing *descriptions*, not DoD items. The
 itself is unchanged and still correct** — A-redfix-2 corrects only the mechanism + remediation sentences.
 B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design).
 Did not read JOURNAL-redfix.md before forming these verdicts.
+
+---
+
+## Wake #44 — 2026-07-09T08:47:06Z — **I REFUTE MY OWN A-redfix-2.** A weekly re-trigger exists; the wedge is silent and recurring. Filed A-redfix-4. Still no VETO.
+
+Autonomous tick. No inbox, no gate claimed. Phase DoD unaffected: **M1 + M2 PASS stand, `## DONE` stands, no
+VETO, merge target `b5f2b10` unchanged.** What follows corrects operator-facing *description*, not a DoD item.
+
+I set out to disbelieve my own last finding. A-redfix-2 (`b358b7e`) asserted "what blocks self-healing is the
+**absence of a re-trigger** — no timer, no `Restart=`", which I had established by grepping `warm-keycloak`
+inside `nix/`. The Builder accepted it into STATUS at `f64d102`, **including my two probe commands**. Both
+probes are unfalsifiable by construction, and the claim is **false**.
+
+**The node's own systemd disagrees.** `warm_reconcile.py` has **two** invokers:
+`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) → `nightly_sweep.main():147`
+→ `roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS = ["keycloak","traefik"]`,
+`nightly_sweep.py:39,57-61`). `docs/warm.md:20` and `docs/concurrency.md:114` both say "systemd timer" in
+plain English. I read past them for three wakes.
+
+**Why the probes lie.** The invoking timer's name never contains the app name, so
+`list-timers | grep -i keycloak` → empty and `git grep warm-keycloak -- nix/` → empty, **while a timer drives
+it every Sunday.** Same class as the `e3b0c44298fc1c14` empty-input tell and the `restore()`/docker ordering
+tell: a probe returning the all-clear value for a reason unrelated to the property under test. Sound probe:
+grep the **callee** (`git grep warm_reconcile -- runner/ nix/`), never the app name.
+
+**Observational proof (cold, cc-ci), independent of reading any code:**
+- `warm-keycloak.service` `ExecMainStartTimestamp` = **Wed 2026-06-17 17:29:31** — ran once, `RemainAfterExit`.
+- `/var/lib/ci-warm/keycloak/last_good` mtime = **2026-07-05 03:04:51.209** (content `10.8.0+26.6.3`).
+- `nightly-sweep.timer` last trigger = **2026-07-05 03:04:50**.
+`write_last_good()` is reachable **only** from `reconcile()`. The slot was written 18 days after the unit last
+ran and 1.2 s after the sweep fired ⇒ `reconcile("keycloak")` executed **from the sweep**. (`journalctl -u
+nightly-sweep.service` retains 1 line; my earlier empty grep there was retention, not absence — I checked
+before concluding.)
+
+**Corrected wedge dynamics** (only once armed: foreign meta in slot **and** `current != latest`): sweep N
+undeploys → `snapshot()` guard raises → keycloak **DOWN**; sweep N+1 finds `is_deployed`=False → fresh-deploy
+branch (`:471-479`, never calls `warmsnap`) → **UP** on the old version; sweep N+2 → **DOWN** again.
+**Alternating weeks, indefinitely.** Neither "recovery is manual" (Builder, `68b51d5`) nor "heals on reboot /
+`nixos-rebuild switch`" (me, `b358b7e`) is right.
+
+**And it is silent** — the part that actually worries me. `roll_warm_infra()` captures the subprocess rc and
+ignores it (`nightly_sweep.py:59-63`), so `nightly-sweep.service` still reports `Result=success`. The guard
+raises at `:514`, outside the `try/except` at `:520-525` (**that is exactly B-redfix-5**) and upstream of every
+`write_alert()` ⇒ **no alert**. A weekly outage of the shared SSO provider would surface nowhere.
+
+**Countervailing check — and it is reassuring.** I hypothesised that Sunday's sweep (2026-07-12 03:00, 3 days
+out) might arm the trap on its own, and **disproved it**: arming needs the canonical seed run from a tree with
+the enrollment but *without* the fix. `git show origin/main:tests/keycloak/recipe_meta.py` → `WARM_CANONICAL =
+False`, and `/etc/cc-ci` (`CCCI_REPO`, the tree the sweep executes) carries that same `False`. Only `07fc6d4`
+and `b5f2b10` have `True`. **No autonomous node activity can arm it.** Confirmed live: slot holds only
+`last_good` (no `snapshot/`, no `canon-*`); keycloak deployed and healthy (`…_app` + `…_db`).
+
+⇒ **The precondition's landmark in STATUS is wrong.** "Slot clean *at merge time*" — a `git merge` executes
+nothing. The binding rule is **never deploy `07fc6d4`** (enrollment without `canonical_ns()`): that
+intermediate state is the *only* thing that can arm the trap. `b5f2b10` ships enrollment **and** fix together,
+so canonical seeds land in `canon-keycloak/` and the slot never goes foreign. The precondition is
+**self-maintaining**, which is a materially calmer merge story than the one STATUS currently tells.
+
+**Also corrected:** `warm-keycloak.service`'s `ExecStart` embeds `/nix/store/…-runner/warm_reconcile.py`, and
+`warmsnap.py` sits in that **same** derivation. So a `nixos-rebuild switch` re-runs reconcile **iff `runner/**`
+changed — an unrelated switch does not heal anything. Deploying the merge does change `runner/**`, so **the
+merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
+
+**Re-derived from scratch this wake (both still hold):** blast radius = keycloak alone
+(`WARM_DOMAINS = {"keycloak"}` at `warm.py:27-29`; `SPECS` keycloak `stateful: True` `:110`, traefik `False`
+`:125`); merge precondition currently satisfied on the node.
+
+**Verdict: no VETO. `## DONE` stands. M1 + M2 PASS stand. Merge target `b5f2b10` unchanged.** A-redfix-4 filed
+(MED, operator-facing). A-redfix-2 **partially withdrawn by me** — its first half (a code path redeploys it)
+survives; its "no re-trigger / heals on reboot" half is superseded. A-redfix-3 stands. B-redfix-5 stays
+deferred/open. B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design).
+Three asks of the Builder in A-redfix-4 — all STATUS text, no code, nothing reopens.
+
+Did not read JOURNAL-redfix.md before forming this verdict.