diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index 433292c..43e7aaa 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -520,6 +520,16 @@ or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db8 ### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent +> **⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4.** +> The *first* half (a code path redeploys it → the "recovery is manual" wording is wrong) **stands**. +> The *second* half — "what blocks self-healing is the **absence of a re-trigger**; `warm-keycloak.service` +> has no timer / no `Restart=`; it heals on the next **reboot / `nixos-rebuild switch`**" — is **WRONG**. +> There IS an autonomous re-trigger: `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) +> → `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS=["keycloak",…]`). +> My probe was scoped to the wrong name: `list-timers | grep -i keycloak` is empty because the timer is called +> `nightly-sweep`, and `git grep warm-keycloak -- nix/` never sees the sweep. **Do not use either command.** +> Corrected dynamics, evidence, and remediation are in **A-redfix-4**, which supersedes this half. + Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred* item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the wrong mechanism points the operator at the wrong remediation. @@ -598,3 +608,101 @@ clone paths**, which is consistent with a single stray write into one working tr **Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed on not deleting it: neither of us created it. + +--- + +### A-redfix-4 [adversary] — the B-redfix-5 wedge has a **weekly autonomous re-trigger** (`nightly-sweep.timer`) and fails **silently**: post-arming, keycloak oscillates down-a-week / up-a-week with `nightly-sweep.service` reporting `Result=success` + +Severity **MED** (operator-facing; describes a *deferred* item's blast profile, **not** a DoD item). +**No VETO. Does not reopen the phase.** M1 + M2 PASS stand. Merge target `b5f2b10` unchanged and still correct. +Supersedes the "absence of a re-trigger" half of **A-redfix-2** (my own error) and, with it, the text the +Builder accepted into STATUS-redfix.md at `f64d102`. + +**Both prior descriptions of the wedge are wrong, in opposite directions.** +- Builder (`68b51d5`): "no code path redeploys it. **Recovery is manual.**" → wrong; it recovers unattended. +- Me (`b358b7e`, A-redfix-2): "no re-trigger … heals on the next **reboot / `nixos-rebuild switch`**" → wrong; + it re-triggers **weekly**, and an unrelated `switch` does *not* heal it. + +**The re-trigger.** `warm_reconcile.py` has **two** invokers, not one. The second is the sweep: + + nightly-sweep.timer OnCalendar=Sun *-*-* 03:00:00, Persistent=true + └─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py (CCCI_REPO=/etc/cc-ci) + └─ main():147 roll_warm_infra() # unconditional after the _another_run_active() guard + └─ for app in WARM_APPS = ["keycloak", "traefik"]: # nightly_sweep.py:39,57 + subprocess.run([python, warm_reconcile.py, app]) # nightly_sweep.py:59-61 + +**Observational proof that the sweep — not `warm-keycloak.service` — reconciles keycloak** (cc-ci, cold): + + systemctl show warm-keycloak.service -p ExecMainStartTimestamp # → Wed 2026-06-17 17:29:31 UTC (once) + stat -c %y /var/lib/ci-warm/keycloak/last_good # → 2026-07-05 03:04:51.209 +0000 + systemctl list-timers --all | grep nightly-sweep # → last trigger Sun 2026-07-05 03:04:50 + +`write_last_good()` is called **only** from `reconcile()`. The slot file was written **18 days after** the unit +last ran, and **1.2 s after** the sweep fired. ⇒ `reconcile("keycloak")` ran from the sweep. (`journalctl -u +nightly-sweep.service` retains 1 line — the empty grep is *retention*, not absence. Do not read it as evidence.) + +**Why both of our probes missed it — the tell.** The invoking timer's name never contains the app name: + + systemctl list-timers --all | grep -i keycloak # → EMPTY, yet a timer DOES drive it + git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart=' # → EMPTY, yet a timer DOES drive it + +Both are **unfalsifiable by construction**. STATUS-redfix.md (`f64d102`) currently prescribes *both* as the +verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the **callee**: + + git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py" + +Same class as the `e3b0c44298fc1c14` empty-input tell (B-redfix-8) and the `restore()`/`docker` ordering tell: +a probe that returns the "all clear" value for a reason unrelated to the property under test. + +**Corrected dynamics** (only once armed — i.e. only if a foreign `snapshot/meta.json` sits in the slot **and** +an upgrade is pending, `current != latest`): + +| sweep | state on entry | branch taken | outcome | +|---|---|---|---| +| N | deployed, upgrade pending | stateful upgrade `:511-514` → `abra.undeploy` → `snapshot()` → guard raises | **keycloak DOWN** | +| N+1 | not deployed | fresh-deploy `:471-479` → redeploys `current`; **never calls `warmsnap`** | keycloak UP (old version) | +| N+2 | deployed, upgrade still pending | stateful upgrade again → guard raises | **keycloak DOWN** | + +⇒ **keycloak alternates DOWN one week / UP the next**, indefinitely, until the foreign `snapshot/` is deleted +or `current == latest`. Not a one-shot outage, and not manual-recovery-required. + +**And it is silent.** `roll_warm_infra()` captures the subprocess rc and **ignores it** (`nightly_sweep.py:59-63` +— `print(f"nightly: reconcile {app} rc={rc}")`, no raise), so a wedged reconcile does **not** fail +`nightly-sweep.service` (`Result=success`). The guard raises at `:514`, which is **outside** the try/except at +`:520-525` (that is B-redfix-5) and **upstream of every `write_alert()`** — so **no alert is written**. The only +trace is a journal line `nightly: reconcile keycloak rc=1`, in a unit whose output is not retained. +**A weekly production outage of the shared SSO provider would surface nowhere.** + +**Good news — the trap cannot arm itself.** Arming requires the canonical seed to run from a tree that has the +keycloak enrollment but **not** the fix. The enrollment is **branch-only**: + + git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → False (= deployed /etc/cc-ci) + git show 07fc6d4:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment, NO fix) + git show b5f2b10:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment + fix) + +The deployed checkout `/etc/cc-ci` carries `WARM_CANONICAL = False`, so **no autonomous node activity — including +Sunday's sweep — can write foreign meta into `/var/lib/ci-warm/keycloak/`.** Verified: the slot holds only +`last_good` (no `snapshot/`, no `canon-*`), and keycloak is currently deployed and healthy (`_app` + `_db`). + +**⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong.** It is not "the slot must be clean +*at merge time*" (a `git merge` executes nothing). The binding statement is: + +> **Never deploy `07fc6d4`** — or any tree carrying the enrollment without `canonical_ns()`. That intermediate +> state is the *only* thing that can arm the trap. Merging `b5f2b10` ships enrollment **and** fix together, so +> canonical seeds land in `canon-keycloak/` and the slot never goes foreign. + +**Secondary correction (`nixos-rebuild switch`).** `warm-keycloak.service`'s `ExecStart` embeds the runner by +store path — `/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` lives in that **same** derivation +(`…-runner/harness/warmsnap.py`). So a switch restarts the unit **iff `runner/**` changed**. A switch that +touches anything else (e.g. sshd config) leaves the store path identical and does **not** re-run reconcile — +so "heals on `nixos-rebuild switch`" is false in general. Deploying the merge *does* change `runner/**`, which +means **the merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep. + +**Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):** +1. Delete both `grep -i keycloak` / `git grep warm-keycloak -- nix/` "no re-trigger" probes; they cannot fail. +2. Replace "recovery is manual" **and** "heals on reboot / `nixos-rebuild switch`" with the weekly + down/up oscillation above, and state that it is **silent** (`Result=success`, no `write_alert`). +3. Re-anchor the merge precondition from "slot clean at merge time" to "**never deploy `07fc6d4`**". + +Remediation if ever armed is unchanged and still correct: **delete the foreign `snapshot/` from the slot**; +do not redeploy keycloak (the sweep does that for you, and it is what masks the fault). diff --git a/machine-docs/BUILDER-INBOX.md b/machine-docs/BUILDER-INBOX.md new file mode 100644 index 0000000..bd204da --- /dev/null +++ b/machine-docs/BUILDER-INBOX.md @@ -0,0 +1,57 @@ +# BUILDER-INBOX + +## From Adversary — wake #44 (2026-07-09T08:47Z) — I was wrong, and `f64d102` inherited it + +Not a gate message. **Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10` +unchanged.** This is a correction to operator-facing STATUS text — including text you correctly took on my +authority. The error was mine, not yours. + +**What I got wrong.** A-redfix-2 told you the wedge persists because there is **no re-trigger** +(`warm-keycloak.service`: oneshot, no timer, no `Restart=`) and heals on the next reboot / `nixos-rebuild +switch`. **False.** `warm_reconcile.py` has a second invoker: + + nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true) + └─ nightly_sweep.py main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak + (WARM_APPS = ["keycloak","traefik"], nightly_sweep.py:39,57-61) + +Proof that does not depend on reading code: `warm-keycloak.service` last ran **2026-06-17 17:29:31**, but +`/var/lib/ci-warm/keycloak/last_good` was written **2026-07-05 03:04:51.2** — 1.2 s after the sweep's last +trigger (03:04:50). `write_last_good()` is reachable only from `reconcile()`. + +**The two probes now in STATUS cannot fail** — please delete them. The timer is named `nightly-sweep`, not +`*keycloak*`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak -- nix/` are both empty **while a +timer drives it weekly**. Sound probe greps the callee: `git grep warm_reconcile -- runner/ nix/`. + +**Corrected behaviour (once armed).** Sweep N: undeploy → `snapshot()` guard raises → keycloak DOWN. Sweep N+1 +(7 d): `is_deployed`=False → fresh-deploy branch → UP on the old version, `warmsnap` never called. Sweep N+2: +DOWN again. **Alternating weeks, indefinitely** — not "recovery is manual" (your `68b51d5`) and not "heals on +reboot" (my `b358b7e`). Worse: it is **silent**. `roll_warm_infra()` ignores the subprocess rc +(`nightly_sweep.py:59-63`) so `nightly-sweep.service` reports `Result=success`, and the raise at `:514` is +outside the `try/except` at `:520-525` (**that is B-redfix-5**) and upstream of every `write_alert()`. + +**The reassuring half — and I checked before alarming you.** I suspected Sunday's sweep (2026-07-12 03:00) +could arm the trap by itself. It cannot. Arming needs the enrollment **without** the fix, and the deployed tree +the sweep runs (`CCCI_REPO=/etc/cc-ci`, = `origin/main`) has `WARM_CANONICAL = False`; only `07fc6d4` and +`b5f2b10` have `True`. **No autonomous node activity can arm it.** Slot verified clean (only `last_good`); +keycloak deployed and healthy. + +So the precondition's landmark is wrong: a `git merge` executes nothing. The binding rule is **never deploy +`07fc6d4`** — enrollment-without-`canonical_ns()` is the only state that can arm it. `b5f2b10` ships both, so +seeds land in `canon-keycloak/`. The precondition is **self-maintaining**; the merge story is calmer than +STATUS currently tells it. + +**Three asks — STATUS-redfix.md text only, no code:** +1. Delete both "no re-trigger" probe commands (they are unfalsifiable). +2. Replace "recovery is manual" / "heals on reboot / `nixos-rebuild switch`" with the weekly DOWN/UP + oscillation, and say it is **silent** (`Result=success`, no `write_alert`). +3. Re-anchor the precondition: "slot clean at merge time" → "**never deploy `07fc6d4`**". + +Minor, same file: `nixos-rebuild switch` re-runs reconcile **iff `runner/**` changed** (`ExecStart` embeds +`/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` is in that same derivation). An unrelated switch +heals nothing; the merge's own activation switch *is* a reconcile trigger. + +Full evidence + repro in **A-redfix-4** (BACKLOG-redfix.md). A-redfix-2 is partially withdrawn there: its +first half (a code path *does* redeploy it) stands. Remediation if ever armed is unchanged: **delete the +foreign `snapshot/`**, do not redeploy keycloak. + +Delete this file when consumed. diff --git a/machine-docs/REVIEW-redfix.md b/machine-docs/REVIEW-redfix.md index ea64a8c..02a1166 100644 --- a/machine-docs/REVIEW-redfix.md +++ b/machine-docs/REVIEW-redfix.md @@ -1359,3 +1359,77 @@ are LOW-MED/INFO and concern operator-facing *descriptions*, not DoD items. The itself is unchanged and still correct** — A-redfix-2 corrects only the mechanism + remediation sentences. B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design). Did not read JOURNAL-redfix.md before forming these verdicts. + +--- + +## Wake #44 — 2026-07-09T08:47:06Z — **I REFUTE MY OWN A-redfix-2.** A weekly re-trigger exists; the wedge is silent and recurring. Filed A-redfix-4. Still no VETO. + +Autonomous tick. No inbox, no gate claimed. Phase DoD unaffected: **M1 + M2 PASS stand, `## DONE` stands, no +VETO, merge target `b5f2b10` unchanged.** What follows corrects operator-facing *description*, not a DoD item. + +I set out to disbelieve my own last finding. A-redfix-2 (`b358b7e`) asserted "what blocks self-healing is the +**absence of a re-trigger** — no timer, no `Restart=`", which I had established by grepping `warm-keycloak` +inside `nix/`. The Builder accepted it into STATUS at `f64d102`, **including my two probe commands**. Both +probes are unfalsifiable by construction, and the claim is **false**. + +**The node's own systemd disagrees.** `warm_reconcile.py` has **two** invokers: +`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) → `nightly_sweep.main():147` +→ `roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS = ["keycloak","traefik"]`, +`nightly_sweep.py:39,57-61`). `docs/warm.md:20` and `docs/concurrency.md:114` both say "systemd timer" in +plain English. I read past them for three wakes. + +**Why the probes lie.** The invoking timer's name never contains the app name, so +`list-timers | grep -i keycloak` → empty and `git grep warm-keycloak -- nix/` → empty, **while a timer drives +it every Sunday.** Same class as the `e3b0c44298fc1c14` empty-input tell and the `restore()`/docker ordering +tell: a probe returning the all-clear value for a reason unrelated to the property under test. Sound probe: +grep the **callee** (`git grep warm_reconcile -- runner/ nix/`), never the app name. + +**Observational proof (cold, cc-ci), independent of reading any code:** +- `warm-keycloak.service` `ExecMainStartTimestamp` = **Wed 2026-06-17 17:29:31** — ran once, `RemainAfterExit`. +- `/var/lib/ci-warm/keycloak/last_good` mtime = **2026-07-05 03:04:51.209** (content `10.8.0+26.6.3`). +- `nightly-sweep.timer` last trigger = **2026-07-05 03:04:50**. +`write_last_good()` is reachable **only** from `reconcile()`. The slot was written 18 days after the unit last +ran and 1.2 s after the sweep fired ⇒ `reconcile("keycloak")` executed **from the sweep**. (`journalctl -u +nightly-sweep.service` retains 1 line; my earlier empty grep there was retention, not absence — I checked +before concluding.) + +**Corrected wedge dynamics** (only once armed: foreign meta in slot **and** `current != latest`): sweep N +undeploys → `snapshot()` guard raises → keycloak **DOWN**; sweep N+1 finds `is_deployed`=False → fresh-deploy +branch (`:471-479`, never calls `warmsnap`) → **UP** on the old version; sweep N+2 → **DOWN** again. +**Alternating weeks, indefinitely.** Neither "recovery is manual" (Builder, `68b51d5`) nor "heals on reboot / +`nixos-rebuild switch`" (me, `b358b7e`) is right. + +**And it is silent** — the part that actually worries me. `roll_warm_infra()` captures the subprocess rc and +ignores it (`nightly_sweep.py:59-63`), so `nightly-sweep.service` still reports `Result=success`. The guard +raises at `:514`, outside the `try/except` at `:520-525` (**that is exactly B-redfix-5**) and upstream of every +`write_alert()` ⇒ **no alert**. A weekly outage of the shared SSO provider would surface nowhere. + +**Countervailing check — and it is reassuring.** I hypothesised that Sunday's sweep (2026-07-12 03:00, 3 days +out) might arm the trap on its own, and **disproved it**: arming needs the canonical seed run from a tree with +the enrollment but *without* the fix. `git show origin/main:tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = +False`, and `/etc/cc-ci` (`CCCI_REPO`, the tree the sweep executes) carries that same `False`. Only `07fc6d4` +and `b5f2b10` have `True`. **No autonomous node activity can arm it.** Confirmed live: slot holds only +`last_good` (no `snapshot/`, no `canon-*`); keycloak deployed and healthy (`…_app` + `…_db`). + +⇒ **The precondition's landmark in STATUS is wrong.** "Slot clean *at merge time*" — a `git merge` executes +nothing. The binding rule is **never deploy `07fc6d4`** (enrollment without `canonical_ns()`): that +intermediate state is the *only* thing that can arm the trap. `b5f2b10` ships enrollment **and** fix together, +so canonical seeds land in `canon-keycloak/` and the slot never goes foreign. The precondition is +**self-maintaining**, which is a materially calmer merge story than the one STATUS currently tells. + +**Also corrected:** `warm-keycloak.service`'s `ExecStart` embeds `/nix/store/…-runner/warm_reconcile.py`, and +`warmsnap.py` sits in that **same** derivation. So a `nixos-rebuild switch` re-runs reconcile **iff `runner/**` +changed — an unrelated switch does not heal anything. Deploying the merge does change `runner/**`, so **the +merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep. + +**Re-derived from scratch this wake (both still hold):** blast radius = keycloak alone +(`WARM_DOMAINS = {"keycloak"}` at `warm.py:27-29`; `SPECS` keycloak `stateful: True` `:110`, traefik `False` +`:125`); merge precondition currently satisfied on the node. + +**Verdict: no VETO. `## DONE` stands. M1 + M2 PASS stand. Merge target `b5f2b10` unchanged.** A-redfix-4 filed +(MED, operator-facing). A-redfix-2 **partially withdrawn by me** — its first half (a code path redeploys it) +survives; its "no re-trigger / heals on reboot" half is superseded. A-redfix-3 stands. B-redfix-5 stays +deferred/open. B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design). +Three asks of the Builder in A-redfix-4 — all STATUS text, no code, nothing reopens. + +Did not read JOURNAL-redfix.md before forming this verdict.