review(redfix): wake #44 — I REFUTE my own A-redfix-2; nightly-sweep.timer is a weekly re-trigger, wedge is silent + recurring (A-redfix-4)
This commit is contained in:
autonomic-bot
@ -520,6 +520,16 @@ or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db8
|
||||
|
||||
### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
|
||||
|
||||
> **⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4.**
|
||||
> The *first* half (a code path redeploys it → the "recovery is manual" wording is wrong) **stands**.
|
||||
> The *second* half — "what blocks self-healing is the **absence of a re-trigger**; `warm-keycloak.service`
|
||||
> has no timer / no `Restart=`; it heals on the next **reboot / `nixos-rebuild switch`**" — is **WRONG**.
|
||||
> There IS an autonomous re-trigger: `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`)
|
||||
> → `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS=["keycloak",…]`).
|
||||
> My probe was scoped to the wrong name: `list-timers | grep -i keycloak` is empty because the timer is called
|
||||
> `nightly-sweep`, and `git grep warm-keycloak -- nix/` never sees the sweep. **Do not use either command.**
|
||||
> Corrected dynamics, evidence, and remediation are in **A-redfix-4**, which supersedes this half.
|
||||
|
||||
Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
|
||||
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
|
||||
wrong mechanism points the operator at the wrong remediation.
|
||||
@ -598,3 +608,101 @@ clone paths**, which is consistent with a single stray write into one working tr
|
||||
|
||||
**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
|
||||
on not deleting it: neither of us created it.
|
||||
|
||||
---
|
||||
|
||||
### A-redfix-4 [adversary] — the B-redfix-5 wedge has a **weekly autonomous re-trigger** (`nightly-sweep.timer`) and fails **silently**: post-arming, keycloak oscillates down-a-week / up-a-week with `nightly-sweep.service` reporting `Result=success`
|
||||
|
||||
Severity **MED** (operator-facing; describes a *deferred* item's blast profile, **not** a DoD item).
|
||||
**No VETO. Does not reopen the phase.** M1 + M2 PASS stand. Merge target `b5f2b10` unchanged and still correct.
|
||||
Supersedes the "absence of a re-trigger" half of **A-redfix-2** (my own error) and, with it, the text the
|
||||
Builder accepted into STATUS-redfix.md at `f64d102`.
|
||||
|
||||
**Both prior descriptions of the wedge are wrong, in opposite directions.**
|
||||
- Builder (`68b51d5`): "no code path redeploys it. **Recovery is manual.**" → wrong; it recovers unattended.
|
||||
- Me (`b358b7e`, A-redfix-2): "no re-trigger … heals on the next **reboot / `nixos-rebuild switch`**" → wrong;
|
||||
it re-triggers **weekly**, and an unrelated `switch` does *not* heal it.
|
||||
|
||||
**The re-trigger.** `warm_reconcile.py` has **two** invokers, not one. The second is the sweep:
|
||||
|
||||
nightly-sweep.timer OnCalendar=Sun *-*-* 03:00:00, Persistent=true
|
||||
└─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py (CCCI_REPO=/etc/cc-ci)
|
||||
└─ main():147 roll_warm_infra() # unconditional after the _another_run_active() guard
|
||||
└─ for app in WARM_APPS = ["keycloak", "traefik"]: # nightly_sweep.py:39,57
|
||||
subprocess.run([python, warm_reconcile.py, app]) # nightly_sweep.py:59-61
|
||||
|
||||
**Observational proof that the sweep — not `warm-keycloak.service` — reconciles keycloak** (cc-ci, cold):
|
||||
|
||||
systemctl show warm-keycloak.service -p ExecMainStartTimestamp # → Wed 2026-06-17 17:29:31 UTC (once)
|
||||
stat -c %y /var/lib/ci-warm/keycloak/last_good # → 2026-07-05 03:04:51.209 +0000
|
||||
systemctl list-timers --all | grep nightly-sweep # → last trigger Sun 2026-07-05 03:04:50
|
||||
|
||||
`write_last_good()` is called **only** from `reconcile()`. The slot file was written **18 days after** the unit
|
||||
last ran, and **1.2 s after** the sweep fired. ⇒ `reconcile("keycloak")` ran from the sweep. (`journalctl -u
|
||||
nightly-sweep.service` retains 1 line — the empty grep is *retention*, not absence. Do not read it as evidence.)
|
||||
|
||||
**Why both of our probes missed it — the tell.** The invoking timer's name never contains the app name:
|
||||
|
||||
systemctl list-timers --all | grep -i keycloak # → EMPTY, yet a timer DOES drive it
|
||||
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart=' # → EMPTY, yet a timer DOES drive it
|
||||
|
||||
Both are **unfalsifiable by construction**. STATUS-redfix.md (`f64d102`) currently prescribes *both* as the
|
||||
verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the **callee**:
|
||||
|
||||
git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py"
|
||||
|
||||
Same class as the `e3b0c44298fc1c14` empty-input tell (B-redfix-8) and the `restore()`/`docker` ordering tell:
|
||||
a probe that returns the "all clear" value for a reason unrelated to the property under test.
|
||||
|
||||
**Corrected dynamics** (only once armed — i.e. only if a foreign `snapshot/meta.json` sits in the slot **and**
|
||||
an upgrade is pending, `current != latest`):
|
||||
|
||||
| sweep | state on entry | branch taken | outcome |
|
||||
|---|---|---|---|
|
||||
| N | deployed, upgrade pending | stateful upgrade `:511-514` → `abra.undeploy` → `snapshot()` → guard raises | **keycloak DOWN** |
|
||||
| N+1 | not deployed | fresh-deploy `:471-479` → redeploys `current`; **never calls `warmsnap`** | keycloak UP (old version) |
|
||||
| N+2 | deployed, upgrade still pending | stateful upgrade again → guard raises | **keycloak DOWN** |
|
||||
|
||||
⇒ **keycloak alternates DOWN one week / UP the next**, indefinitely, until the foreign `snapshot/` is deleted
|
||||
or `current == latest`. Not a one-shot outage, and not manual-recovery-required.
|
||||
|
||||
**And it is silent.** `roll_warm_infra()` captures the subprocess rc and **ignores it** (`nightly_sweep.py:59-63`
|
||||
— `print(f"nightly: reconcile {app} rc={rc}")`, no raise), so a wedged reconcile does **not** fail
|
||||
`nightly-sweep.service` (`Result=success`). The guard raises at `:514`, which is **outside** the try/except at
|
||||
`:520-525` (that is B-redfix-5) and **upstream of every `write_alert()`** — so **no alert is written**. The only
|
||||
trace is a journal line `nightly: reconcile keycloak rc=1`, in a unit whose output is not retained.
|
||||
**A weekly production outage of the shared SSO provider would surface nowhere.**
|
||||
|
||||
**Good news — the trap cannot arm itself.** Arming requires the canonical seed to run from a tree that has the
|
||||
keycloak enrollment but **not** the fix. The enrollment is **branch-only**:
|
||||
|
||||
git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → False (= deployed /etc/cc-ci)
|
||||
git show 07fc6d4:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment, NO fix)
|
||||
git show b5f2b10:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment + fix)
|
||||
|
||||
The deployed checkout `/etc/cc-ci` carries `WARM_CANONICAL = False`, so **no autonomous node activity — including
|
||||
Sunday's sweep — can write foreign meta into `/var/lib/ci-warm/keycloak/`.** Verified: the slot holds only
|
||||
`last_good` (no `snapshot/`, no `canon-*`), and keycloak is currently deployed and healthy (`_app` + `_db`).
|
||||
|
||||
**⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong.** It is not "the slot must be clean
|
||||
*at merge time*" (a `git merge` executes nothing). The binding statement is:
|
||||
|
||||
> **Never deploy `07fc6d4`** — or any tree carrying the enrollment without `canonical_ns()`. That intermediate
|
||||
> state is the *only* thing that can arm the trap. Merging `b5f2b10` ships enrollment **and** fix together, so
|
||||
> canonical seeds land in `canon-keycloak/` and the slot never goes foreign.
|
||||
|
||||
**Secondary correction (`nixos-rebuild switch`).** `warm-keycloak.service`'s `ExecStart` embeds the runner by
|
||||
store path — `/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` lives in that **same** derivation
|
||||
(`…-runner/harness/warmsnap.py`). So a switch restarts the unit **iff `runner/**` changed**. A switch that
|
||||
touches anything else (e.g. sshd config) leaves the store path identical and does **not** re-run reconcile —
|
||||
so "heals on `nixos-rebuild switch`" is false in general. Deploying the merge *does* change `runner/**`, which
|
||||
means **the merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
|
||||
|
||||
**Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):**
|
||||
1. Delete both `grep -i keycloak` / `git grep warm-keycloak -- nix/` "no re-trigger" probes; they cannot fail.
|
||||
2. Replace "recovery is manual" **and** "heals on reboot / `nixos-rebuild switch`" with the weekly
|
||||
down/up oscillation above, and state that it is **silent** (`Result=success`, no `write_alert`).
|
||||
3. Re-anchor the merge precondition from "slot clean at merge time" to "**never deploy `07fc6d4`**".
|
||||
|
||||
Remediation if ever armed is unchanged and still correct: **delete the foreign `snapshot/` from the slot**;
|
||||
do not redeploy keycloak (the sweep does that for you, and it is what masks the fault).
|
||||
|
||||
57
machine-docs/BUILDER-INBOX.md
Normal file
57
machine-docs/BUILDER-INBOX.md
Normal file
@ -0,0 +1,57 @@
|
||||
# BUILDER-INBOX
|
||||
|
||||
## From Adversary — wake #44 (2026-07-09T08:47Z) — I was wrong, and `f64d102` inherited it
|
||||
|
||||
Not a gate message. **Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10`
|
||||
unchanged.** This is a correction to operator-facing STATUS text — including text you correctly took on my
|
||||
authority. The error was mine, not yours.
|
||||
|
||||
**What I got wrong.** A-redfix-2 told you the wedge persists because there is **no re-trigger**
|
||||
(`warm-keycloak.service`: oneshot, no timer, no `Restart=`) and heals on the next reboot / `nixos-rebuild
|
||||
switch`. **False.** `warm_reconcile.py` has a second invoker:
|
||||
|
||||
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
||||
└─ nightly_sweep.py main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
||||
(WARM_APPS = ["keycloak","traefik"], nightly_sweep.py:39,57-61)
|
||||
|
||||
Proof that does not depend on reading code: `warm-keycloak.service` last ran **2026-06-17 17:29:31**, but
|
||||
`/var/lib/ci-warm/keycloak/last_good` was written **2026-07-05 03:04:51.2** — 1.2 s after the sweep's last
|
||||
trigger (03:04:50). `write_last_good()` is reachable only from `reconcile()`.
|
||||
|
||||
**The two probes now in STATUS cannot fail** — please delete them. The timer is named `nightly-sweep`, not
|
||||
`*keycloak*`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak -- nix/` are both empty **while a
|
||||
timer drives it weekly**. Sound probe greps the callee: `git grep warm_reconcile -- runner/ nix/`.
|
||||
|
||||
**Corrected behaviour (once armed).** Sweep N: undeploy → `snapshot()` guard raises → keycloak DOWN. Sweep N+1
|
||||
(7 d): `is_deployed`=False → fresh-deploy branch → UP on the old version, `warmsnap` never called. Sweep N+2:
|
||||
DOWN again. **Alternating weeks, indefinitely** — not "recovery is manual" (your `68b51d5`) and not "heals on
|
||||
reboot" (my `b358b7e`). Worse: it is **silent**. `roll_warm_infra()` ignores the subprocess rc
|
||||
(`nightly_sweep.py:59-63`) so `nightly-sweep.service` reports `Result=success`, and the raise at `:514` is
|
||||
outside the `try/except` at `:520-525` (**that is B-redfix-5**) and upstream of every `write_alert()`.
|
||||
|
||||
**The reassuring half — and I checked before alarming you.** I suspected Sunday's sweep (2026-07-12 03:00)
|
||||
could arm the trap by itself. It cannot. Arming needs the enrollment **without** the fix, and the deployed tree
|
||||
the sweep runs (`CCCI_REPO=/etc/cc-ci`, = `origin/main`) has `WARM_CANONICAL = False`; only `07fc6d4` and
|
||||
`b5f2b10` have `True`. **No autonomous node activity can arm it.** Slot verified clean (only `last_good`);
|
||||
keycloak deployed and healthy.
|
||||
|
||||
So the precondition's landmark is wrong: a `git merge` executes nothing. The binding rule is **never deploy
|
||||
`07fc6d4`** — enrollment-without-`canonical_ns()` is the only state that can arm it. `b5f2b10` ships both, so
|
||||
seeds land in `canon-keycloak/`. The precondition is **self-maintaining**; the merge story is calmer than
|
||||
STATUS currently tells it.
|
||||
|
||||
**Three asks — STATUS-redfix.md text only, no code:**
|
||||
1. Delete both "no re-trigger" probe commands (they are unfalsifiable).
|
||||
2. Replace "recovery is manual" / "heals on reboot / `nixos-rebuild switch`" with the weekly DOWN/UP
|
||||
oscillation, and say it is **silent** (`Result=success`, no `write_alert`).
|
||||
3. Re-anchor the precondition: "slot clean at merge time" → "**never deploy `07fc6d4`**".
|
||||
|
||||
Minor, same file: `nixos-rebuild switch` re-runs reconcile **iff `runner/**` changed** (`ExecStart` embeds
|
||||
`/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` is in that same derivation). An unrelated switch
|
||||
heals nothing; the merge's own activation switch *is* a reconcile trigger.
|
||||
|
||||
Full evidence + repro in **A-redfix-4** (BACKLOG-redfix.md). A-redfix-2 is partially withdrawn there: its
|
||||
first half (a code path *does* redeploy it) stands. Remediation if ever armed is unchanged: **delete the
|
||||
foreign `snapshot/`**, do not redeploy keycloak.
|
||||
|
||||
Delete this file when consumed.
|
||||
@ -1359,3 +1359,77 @@ are LOW-MED/INFO and concern operator-facing *descriptions*, not DoD items. The
|
||||
itself is unchanged and still correct** — A-redfix-2 corrects only the mechanism + remediation sentences.
|
||||
B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design).
|
||||
Did not read JOURNAL-redfix.md before forming these verdicts.
|
||||
|
||||
---
|
||||
|
||||
## Wake #44 — 2026-07-09T08:47:06Z — **I REFUTE MY OWN A-redfix-2.** A weekly re-trigger exists; the wedge is silent and recurring. Filed A-redfix-4. Still no VETO.
|
||||
|
||||
Autonomous tick. No inbox, no gate claimed. Phase DoD unaffected: **M1 + M2 PASS stand, `## DONE` stands, no
|
||||
VETO, merge target `b5f2b10` unchanged.** What follows corrects operator-facing *description*, not a DoD item.
|
||||
|
||||
I set out to disbelieve my own last finding. A-redfix-2 (`b358b7e`) asserted "what blocks self-healing is the
|
||||
**absence of a re-trigger** — no timer, no `Restart=`", which I had established by grepping `warm-keycloak`
|
||||
inside `nix/`. The Builder accepted it into STATUS at `f64d102`, **including my two probe commands**. Both
|
||||
probes are unfalsifiable by construction, and the claim is **false**.
|
||||
|
||||
**The node's own systemd disagrees.** `warm_reconcile.py` has **two** invokers:
|
||||
`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) → `nightly_sweep.main():147`
|
||||
→ `roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS = ["keycloak","traefik"]`,
|
||||
`nightly_sweep.py:39,57-61`). `docs/warm.md:20` and `docs/concurrency.md:114` both say "systemd timer" in
|
||||
plain English. I read past them for three wakes.
|
||||
|
||||
**Why the probes lie.** The invoking timer's name never contains the app name, so
|
||||
`list-timers | grep -i keycloak` → empty and `git grep warm-keycloak -- nix/` → empty, **while a timer drives
|
||||
it every Sunday.** Same class as the `e3b0c44298fc1c14` empty-input tell and the `restore()`/docker ordering
|
||||
tell: a probe returning the all-clear value for a reason unrelated to the property under test. Sound probe:
|
||||
grep the **callee** (`git grep warm_reconcile -- runner/ nix/`), never the app name.
|
||||
|
||||
**Observational proof (cold, cc-ci), independent of reading any code:**
|
||||
- `warm-keycloak.service` `ExecMainStartTimestamp` = **Wed 2026-06-17 17:29:31** — ran once, `RemainAfterExit`.
|
||||
- `/var/lib/ci-warm/keycloak/last_good` mtime = **2026-07-05 03:04:51.209** (content `10.8.0+26.6.3`).
|
||||
- `nightly-sweep.timer` last trigger = **2026-07-05 03:04:50**.
|
||||
`write_last_good()` is reachable **only** from `reconcile()`. The slot was written 18 days after the unit last
|
||||
ran and 1.2 s after the sweep fired ⇒ `reconcile("keycloak")` executed **from the sweep**. (`journalctl -u
|
||||
nightly-sweep.service` retains 1 line; my earlier empty grep there was retention, not absence — I checked
|
||||
before concluding.)
|
||||
|
||||
**Corrected wedge dynamics** (only once armed: foreign meta in slot **and** `current != latest`): sweep N
|
||||
undeploys → `snapshot()` guard raises → keycloak **DOWN**; sweep N+1 finds `is_deployed`=False → fresh-deploy
|
||||
branch (`:471-479`, never calls `warmsnap`) → **UP** on the old version; sweep N+2 → **DOWN** again.
|
||||
**Alternating weeks, indefinitely.** Neither "recovery is manual" (Builder, `68b51d5`) nor "heals on reboot /
|
||||
`nixos-rebuild switch`" (me, `b358b7e`) is right.
|
||||
|
||||
**And it is silent** — the part that actually worries me. `roll_warm_infra()` captures the subprocess rc and
|
||||
ignores it (`nightly_sweep.py:59-63`), so `nightly-sweep.service` still reports `Result=success`. The guard
|
||||
raises at `:514`, outside the `try/except` at `:520-525` (**that is exactly B-redfix-5**) and upstream of every
|
||||
`write_alert()` ⇒ **no alert**. A weekly outage of the shared SSO provider would surface nowhere.
|
||||
|
||||
**Countervailing check — and it is reassuring.** I hypothesised that Sunday's sweep (2026-07-12 03:00, 3 days
|
||||
out) might arm the trap on its own, and **disproved it**: arming needs the canonical seed run from a tree with
|
||||
the enrollment but *without* the fix. `git show origin/main:tests/keycloak/recipe_meta.py` → `WARM_CANONICAL =
|
||||
False`, and `/etc/cc-ci` (`CCCI_REPO`, the tree the sweep executes) carries that same `False`. Only `07fc6d4`
|
||||
and `b5f2b10` have `True`. **No autonomous node activity can arm it.** Confirmed live: slot holds only
|
||||
`last_good` (no `snapshot/`, no `canon-*`); keycloak deployed and healthy (`…_app` + `…_db`).
|
||||
|
||||
⇒ **The precondition's landmark in STATUS is wrong.** "Slot clean *at merge time*" — a `git merge` executes
|
||||
nothing. The binding rule is **never deploy `07fc6d4`** (enrollment without `canonical_ns()`): that
|
||||
intermediate state is the *only* thing that can arm the trap. `b5f2b10` ships enrollment **and** fix together,
|
||||
so canonical seeds land in `canon-keycloak/` and the slot never goes foreign. The precondition is
|
||||
**self-maintaining**, which is a materially calmer merge story than the one STATUS currently tells.
|
||||
|
||||
**Also corrected:** `warm-keycloak.service`'s `ExecStart` embeds `/nix/store/…-runner/warm_reconcile.py`, and
|
||||
`warmsnap.py` sits in that **same** derivation. So a `nixos-rebuild switch` re-runs reconcile **iff `runner/**`
|
||||
changed — an unrelated switch does not heal anything. Deploying the merge does change `runner/**`, so **the
|
||||
merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
|
||||
|
||||
**Re-derived from scratch this wake (both still hold):** blast radius = keycloak alone
|
||||
(`WARM_DOMAINS = {"keycloak"}` at `warm.py:27-29`; `SPECS` keycloak `stateful: True` `:110`, traefik `False`
|
||||
`:125`); merge precondition currently satisfied on the node.
|
||||
|
||||
**Verdict: no VETO. `## DONE` stands. M1 + M2 PASS stand. Merge target `b5f2b10` unchanged.** A-redfix-4 filed
|
||||
(MED, operator-facing). A-redfix-2 **partially withdrawn by me** — its first half (a code path redeploys it)
|
||||
survives; its "no re-trigger / heals on reboot" half is superseded. A-redfix-3 stands. B-redfix-5 stays
|
||||
deferred/open. B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design).
|
||||
Three asks of the Builder in A-redfix-4 — all STATUS text, no code, nothing reopens.
|
||||
|
||||
Did not read JOURNAL-redfix.md before forming this verdict.
|
||||
|
||||
Reference in New Issue
Block a user