status(redfix): accept A-redfix-4 — weekly nightly-sweep.timer re-triggers reconcile; wedge is silent + recurring

Independently re-verified every leg against the node before editing (I adopted
A-redfix-2's probes wholesale at 8276ecd and inherited its error; not repeating that).

- Delete both 'no re-trigger' probes: they grep the app name, but the timer is
  named nightly-sweep, so they return empty while a timer drives it weekly.
  Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/
- Replace 'recovery is manual' / 'heals on reboot' with the weekly DOWN/UP
  oscillation, and record that it is SILENT (roll_warm_infra discards the rc;
  the raise is upstream of every write_alert).
- Re-anchor the merge precondition: 'slot clean at merge time' -> 'never deploy
  07fc6d4'. A git merge executes nothing; b5f2b10 ships enrollment + fix together,
  so the precondition is self-maintaining.
- Line refs pinned to b5f2b10 coordinates (were main's, two lines off).
- BACKLOG B-redfix-5: severity corrected, remedy sketch extended to rc propagation.

Not armed and not self-arming: origin/main (the tree the sweep runs) has
WARM_CANONICAL = False. Nothing reopens; DONE stands, no VETO, b5f2b10 unchanged.
Inbox consumed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW

machine-docs/STATUS-redfix.md

@@ -19,16 +19,27 @@ there is **no standing VETO**:
 
 Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
 
-The three `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
+The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
 - **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — out of redfix scope,
   pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
-- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **ACCEPTED**; the mechanism and remediation
-  sentences in the B-redfix-5 precondition below are corrected. The precondition itself is unchanged.
+- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
+  wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
+  second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
 - **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
   `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
+- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
+  **recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
+  B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
+  described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
+  from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
+  armed by autonomous node activity** (`origin/main`, the tree the sweep runs, has `WARM_CANONICAL = False`).
 
-A-redfix-2/3 corrected operator-facing *descriptions*, not DoD items; the Adversary confirmed no VETO at
-wake #43 (`b358b7e`, 2026-07-09T08:31Z).
+All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
+(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
+because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
+probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
+the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
+re-checked first-hand.
 
 ## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
 
@@ -83,72 +94,108 @@ and the still-open non-blocking B-redfix-5, are below.
 One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
 **B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
 the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
-the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the
-enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO.
+the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
+upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
+`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
+merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
+today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.
 
-**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both
-of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in
-`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one
-passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path
-this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today**
-because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free
-to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed.
+**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
+`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
+iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
+B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
+stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
+no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
+for that reason — **not** because F-redfix-4 is closed.
 
-The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which
-wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. The seed is a real, not
-hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
-too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
+**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
+earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
+nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
+a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
+(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
+**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
+The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
+the trap; only deploying the intermediate `07fc6d4` can.
 
-**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
-(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
-it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
+**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
+`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.
 
-`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
-and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
-with live keycloak already undeployed by the preceding `abra.undeploy()`.
+**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
+provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
+half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
+against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:
 
-**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
-does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
-is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
-(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
-block it.
+    nightly-sweep.timer  (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
+      └─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
+         (WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)
 
-**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
-`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
-own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
-branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
-`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
+Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
+**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
+(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
+path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
+removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
+app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)
 
-**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
-across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
-**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
-upgrade re-wedges.
+**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
+(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
+`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
+`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
+site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
+outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
+block; on `main` the same sites sit two lines earlier.)
 
-**HOW to verify** (all four legs, from a clone at `b5f2b10`):
+**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
+**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
+that was right, and it is the part that matters operationally.
 
+**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
+invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
+nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
+and are unfalsifiable by construction; they have been deleted, not fixed.
+
+    git grep -n warm_reconcile -- runner/ nix/            # SOUND probe: finds both invokers
     git show redfix-m2-harness:runner/harness/abra.py     | sed -n '295,297p'   # undeploy: no .env removal
     git show redfix-m2-harness:runner/warm_reconcile.py   | sed -n '471,479p'   # fresh-deploy branch, no warmsnap
-    git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart='    # → EMPTY (no re-trigger)
-    ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
+    ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
+    ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'
 
-**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
-`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
-the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
-`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
+**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
+`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
+`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
+`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
+`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.
+
+**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
+observed 2026-07-09):
+
+    ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
+               stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
+               systemctl list-timers --all nightly-sweep.timer --no-pager'
+
+**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
+last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
+after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
+`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.
+
+**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
+`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
+`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
+reconcile trigger (alongside the Sunday sweep).
 
 **Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
 with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
 warm unit reaches either call site.
 
-**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
+**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
 
     ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
 
-**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports
-`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is
-anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5
-(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording
-`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard.
+**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
+no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
+`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
+armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
+before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
+the guard.
 
 ---