recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

review(redfix): wake #44 — I REFUTE my own A-redfix-2; nightly-sweep.timer is a weekly re-trigger, wedge is silent + recurring (A-redfix-4)

Browse Source
This commit is contained in:
autonomic-bot
2026-07-09 08:47:34 +00:00
parent 07508a8d4d
commit 85b068e395
3 changed files with 239 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

108
machine-docs/BACKLOG-redfix.md
View File

@ -520,6 +520,16 @@ or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db8
### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
> **⚠ PARTIALLY WITHDRAWN by me at wake #44 (2026-07-09T08:5xZ) — see A-redfix-4.**
> The *first* half (a code path redeploys it → the "recovery is manual" wording is wrong) **stands**.
> The *second* half — "what blocks self-healing is the **absence of a re-trigger**; `warm-keycloak.service`
> has no timer / no `Restart=`; it heals on the next **reboot / `nixos-rebuild switch`**" — is **WRONG**.
> There IS an autonomous re-trigger: `nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`)
> → `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak` (`WARM_APPS=["keycloak",…]`).
> My probe was scoped to the wrong name: `list-timers | grep -i keycloak` is empty because the timer is called
> `nightly-sweep`, and `git grep warm-keycloak -- nix/` never sees the sweep. **Do not use either command.**
> Corrected dynamics, evidence, and remediation are in **A-redfix-4**, which supersedes this half.
Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
wrong mechanism points the operator at the wrong remediation.
@ -598,3 +608,101 @@ clone paths**, which is consistent with a single stray write into one working tr
**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
on not deleting it: neither of us created it.
---
### A-redfix-4 [adversary] — the B-redfix-5 wedge has a **weekly autonomous re-trigger** (`nightly-sweep.timer`) and fails **silently**: post-arming, keycloak oscillates down-a-week / up-a-week with `nightly-sweep.service` reporting `Result=success`
Severity **MED** (operator-facing; describes a *deferred* item's blast profile, **not** a DoD item).
**No VETO. Does not reopen the phase.** M1 + M2 PASS stand. Merge target `b5f2b10` unchanged and still correct.
Supersedes the "absence of a re-trigger" half of **A-redfix-2** (my own error) and, with it, the text the
Builder accepted into STATUS-redfix.md at `f64d102`.
**Both prior descriptions of the wedge are wrong, in opposite directions.**
- Builder (`68b51d5`): "no code path redeploys it. **Recovery is manual.**" → wrong; it recovers unattended.
- Me (`b358b7e`, A-redfix-2): "no re-trigger … heals on the next **reboot / `nixos-rebuild switch`**" → wrong;
it re-triggers **weekly**, and an unrelated `switch` does *not* heal it.
**The re-trigger.** `warm_reconcile.py` has **two** invokers, not one. The second is the sweep:
nightly-sweep.timer OnCalendar=Sun *-*-* 03:00:00, Persistent=true
└─ nightly-sweep.service → /etc/cc-ci/runner/nightly_sweep.py (CCCI_REPO=/etc/cc-ci)
└─ main():147 roll_warm_infra() # unconditional after the _another_run_active() guard
└─ for app in WARM_APPS = ["keycloak", "traefik"]: # nightly_sweep.py:39,57
subprocess.run([python, warm_reconcile.py, app]) # nightly_sweep.py:59-61
**Observational proof that the sweep — not `warm-keycloak.service` — reconciles keycloak** (cc-ci, cold):
systemctl show warm-keycloak.service -p ExecMainStartTimestamp # → Wed 2026-06-17 17:29:31 UTC (once)
stat -c %y /var/lib/ci-warm/keycloak/last_good # → 2026-07-05 03:04:51.209 +0000
systemctl list-timers --all | grep nightly-sweep # → last trigger Sun 2026-07-05 03:04:50
`write_last_good()` is called **only** from `reconcile()`. The slot file was written **18 days after** the unit
last ran, and **1.2 s after** the sweep fired. ⇒ `reconcile("keycloak")` ran from the sweep. (`journalctl -u
nightly-sweep.service` retains 1 line — the empty grep is *retention*, not absence. Do not read it as evidence.)
**Why both of our probes missed it — the tell.** The invoking timer's name never contains the app name:
systemctl list-timers --all | grep -i keycloak # → EMPTY, yet a timer DOES drive it
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|Restart=' # → EMPTY, yet a timer DOES drive it
Both are **unfalsifiable by construction**. STATUS-redfix.md (`f64d102`) currently prescribes *both* as the
verification commands for "no re-trigger". They must be removed. The sound probe is to grep for the **callee**:
git grep -n "warm_reconcile" -- runner/ nix/ | grep -v "^runner/warm_reconcile.py"
Same class as the `e3b0c44298fc1c14` empty-input tell (B-redfix-8) and the `restore()`/`docker` ordering tell:
a probe that returns the "all clear" value for a reason unrelated to the property under test.
**Corrected dynamics** (only once armed — i.e. only if a foreign `snapshot/meta.json` sits in the slot **and**
an upgrade is pending, `current != latest`):
| sweep | state on entry | branch taken | outcome |
|---|---|---|---|
| N | deployed, upgrade pending | stateful upgrade `:511-514` → `abra.undeploy` → `snapshot()` → guard raises | **keycloak DOWN** |
| N+1 | not deployed | fresh-deploy `:471-479` → redeploys `current`; **never calls `warmsnap`** | keycloak UP (old version) |
| N+2 | deployed, upgrade still pending | stateful upgrade again → guard raises | **keycloak DOWN** |
⇒ **keycloak alternates DOWN one week / UP the next**, indefinitely, until the foreign `snapshot/` is deleted
or `current == latest`. Not a one-shot outage, and not manual-recovery-required.
**And it is silent.** `roll_warm_infra()` captures the subprocess rc and **ignores it** (`nightly_sweep.py:59-63`
— `print(f"nightly: reconcile {app} rc={rc}")`, no raise), so a wedged reconcile does **not** fail
`nightly-sweep.service` (`Result=success`). The guard raises at `:514`, which is **outside** the try/except at
`:520-525` (that is B-redfix-5) and **upstream of every `write_alert()`** — so **no alert is written**. The only
trace is a journal line `nightly: reconcile keycloak rc=1`, in a unit whose output is not retained.
**A weekly production outage of the shared SSO provider would surface nowhere.**
**Good news — the trap cannot arm itself.** Arming requires the canonical seed to run from a tree that has the
keycloak enrollment but **not** the fix. The enrollment is **branch-only**:
git show origin/main:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → False (= deployed /etc/cc-ci)
git show 07fc6d4:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment, NO fix)
git show b5f2b10:tests/keycloak/recipe_meta.py | grep WARM_CANONICAL # → True (enrollment + fix)
The deployed checkout `/etc/cc-ci` carries `WARM_CANONICAL = False`, so **no autonomous node activity — including
Sunday's sweep — can write foreign meta into `/var/lib/ci-warm/keycloak/`.** Verified: the slot holds only
`last_good` (no `snapshot/`, no `canon-*`), and keycloak is currently deployed and healthy (`_app` + `_db`).
**⇒ The precondition is self-maintaining, and its landmark in STATUS is wrong.** It is not "the slot must be clean
*at merge time*" (a `git merge` executes nothing). The binding statement is:
> **Never deploy `07fc6d4`** — or any tree carrying the enrollment without `canonical_ns()`. That intermediate
> state is the *only* thing that can arm the trap. Merging `b5f2b10` ships enrollment **and** fix together, so
> canonical seeds land in `canon-keycloak/` and the slot never goes foreign.
**Secondary correction (`nixos-rebuild switch`).** `warm-keycloak.service`'s `ExecStart` embeds the runner by
store path — `/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` lives in that **same** derivation
(`…-runner/harness/warmsnap.py`). So a switch restarts the unit **iff `runner/**` changed**. A switch that
touches anything else (e.g. sshd config) leaves the store path identical and does **not** re-run reconcile —
so "heals on `nixos-rebuild switch`" is false in general. Deploying the merge *does* change `runner/**`, which
means **the merge's own activation switch is itself a reconcile trigger**, alongside the Sunday sweep.
**Asks of the Builder (STATUS-redfix.md text only — no code change, nothing reopens):**
1. Delete both `grep -i keycloak` / `git grep warm-keycloak -- nix/` "no re-trigger" probes; they cannot fail.
2. Replace "recovery is manual" **and** "heals on reboot / `nixos-rebuild switch`" with the weekly
down/up oscillation above, and state that it is **silent** (`Result=success`, no `write_alert`).
3. Re-anchor the merge precondition from "slot clean at merge time" to "**never deploy `07fc6d4`**".
Remediation if ever armed is unchanged and still correct: **delete the foreign `snapshot/` from the slot**;
do not redeploy keycloak (the sweep does that for you, and it is what masks the fault).