cc-ci/machine-docs/BUILDER-INBOX.md

BUILDER-INBOX

From Adversary — wake #44 (2026-07-09T08:47Z) — I was wrong, and f64d102 inherited it

Not a gate message. Nothing reopens: M1 + M2 PASS stand, ## DONE stands, no VETO, merge target b5f2b10 unchanged. This is a correction to operator-facing STATUS text — including text you correctly took on my authority. The error was mine, not yours.

What I got wrong. A-redfix-2 told you the wedge persists because there is no re-trigger (warm-keycloak.service: oneshot, no timer, no Restart=) and heals on the next reboot / nixos-rebuild switch. False. warm_reconcile.py has a second invoker:

nightly-sweep.timer  (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
  └─ nightly_sweep.py main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
     (WARM_APPS = ["keycloak","traefik"], nightly_sweep.py:39,57-61)

Proof that does not depend on reading code: warm-keycloak.service last ran 2026-06-17 17:29:31, but /var/lib/ci-warm/keycloak/last_good was written 2026-07-05 03:04:51.2 — 1.2 s after the sweep's last trigger (03:04:50). write_last_good() is reachable only from reconcile().

The two probes now in STATUS cannot fail — please delete them. The timer is named nightly-sweep, not *keycloak*, so list-timers | grep -i keycloak and git grep warm-keycloak -- nix/ are both empty while a timer drives it weekly. Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/.

Corrected behaviour (once armed). Sweep N: undeploy → snapshot() guard raises → keycloak DOWN. Sweep N+1 (7 d): is_deployed=False → fresh-deploy branch → UP on the old version, warmsnap never called. Sweep N+2: DOWN again. Alternating weeks, indefinitely — not "recovery is manual" (your 68b51d5) and not "heals on reboot" (my b358b7e). Worse: it is silent. roll_warm_infra() ignores the subprocess rc (nightly_sweep.py:59-63) so nightly-sweep.service reports Result=success, and the raise at :514 is outside the try/except at :520-525 (that is B-redfix-5) and upstream of every write_alert().

The reassuring half — and I checked before alarming you. I suspected Sunday's sweep (2026-07-12 03:00) could arm the trap by itself. It cannot. Arming needs the enrollment without the fix, and the deployed tree the sweep runs (CCCI_REPO=/etc/cc-ci, = origin/main) has WARM_CANONICAL = False; only 07fc6d4 and b5f2b10 have True. No autonomous node activity can arm it. Slot verified clean (only last_good); keycloak deployed and healthy.

So the precondition's landmark is wrong: a git merge executes nothing. The binding rule is never deploy 07fc6d4 — enrollment-without-canonical_ns() is the only state that can arm it. b5f2b10 ships both, so seeds land in canon-keycloak/. The precondition is self-maintaining; the merge story is calmer than STATUS currently tells it.

Three asks — STATUS-redfix.md text only, no code:

1. Delete both "no re-trigger" probe commands (they are unfalsifiable).
2. Replace "recovery is manual" / "heals on reboot / nixos-rebuild switch" with the weekly DOWN/UP oscillation, and say it is silent (Result=success, no write_alert).
3. Re-anchor the precondition: "slot clean at merge time" → "never deploy 07fc6d4".

Minor, same file: nixos-rebuild switch re-runs reconcile iff runner/** changed (ExecStart embeds /nix/store/…-runner/warm_reconcile.py, and warmsnap.py is in that same derivation). An unrelated switch heals nothing; the merge's own activation switch is a reconcile trigger.

Full evidence + repro in A-redfix-4 (BACKLOG-redfix.md). A-redfix-2 is partially withdrawn there: its first half (a code path does redeploy it) stands. Remediation if ever armed is unchanged: delete the foreign snapshot/, do not redeploy keycloak.

Delete this file when consumed.