status(redfix): accept A-redfix-4 — weekly nightly-sweep.timer re-triggers reconcile; wedge is silent + recurring
Independently re-verified every leg against the node before editing (I adopted A-redfix-2's probes wholesale at8276ecdand inherited its error; not repeating that). - Delete both 'no re-trigger' probes: they grep the app name, but the timer is named nightly-sweep, so they return empty while a timer drives it weekly. Sound probe greps the callee: git grep warm_reconcile -- runner/ nix/ - Replace 'recovery is manual' / 'heals on reboot' with the weekly DOWN/UP oscillation, and record that it is SILENT (roll_warm_infra discards the rc; the raise is upstream of every write_alert). - Re-anchor the merge precondition: 'slot clean at merge time' -> 'never deploy 07fc6d4'. A git merge executes nothing;b5f2b10ships enrollment + fix together, so the precondition is self-maintaining. - Line refs pinned tob5f2b10coordinates (were main's, two lines off). - BACKLOG B-redfix-5: severity corrected, remedy sketch extended to rc propagation. Not armed and not self-arming: origin/main (the tree the sweep runs) has WARM_CANONICAL = False. Nothing reopens; DONE stands, no VETO,b5f2b10unchanged. Inbox consumed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TkHo7RixQswYvvNnEcNaqW
This commit is contained in:
@ -74,10 +74,26 @@ hold). Concrete fix designs from M1 evidence:
|
||||
`snapshot/meta.json` (`read_meta` → `None`, and an unclaimed slot is free to claim) — *not* because
|
||||
F-redfix-4 is closed. A pre-fix (`07fc6d4`) canonical seed is the one state that writes a foreign
|
||||
domain into the live slot; see the MERGE PRECONDITION in STATUS-redfix.md.
|
||||
**Consequence, corrected (wake #44, A-redfix-4) — worse than recorded above, still not blocking.**
|
||||
The wedge is neither manual-recovery-only (my `68b51d5`) nor reboot-healed (A-redfix-2). A weekly
|
||||
`nightly-sweep.timer` (`OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`) re-invokes
|
||||
`warm_reconcile.py keycloak` via `nightly_sweep.roll_warm_infra()` (`:60`), so once armed the wedge
|
||||
**alternates DOWN/UP every 7 days, indefinitely**: sweep N wedges at site (a); sweep N+1 takes the
|
||||
fresh-deploy branch (`:471-479`, no `warmsnap`) and comes back up on the old version; sweep N+2
|
||||
wedges again. It is also **silent** — `roll_warm_infra()` discards the subprocess rc (`:59-62`) so
|
||||
`nightly-sweep.service` reports `Result=success`, and site (a) is upstream of every `write_alert()`
|
||||
(`:493,500,503,539`). A weekly outage of the shared SSO provider would surface nowhere. Verified
|
||||
first-hand against the node, not adopted from the finding.
|
||||
**Not armed, and not self-arming:** `origin/main` — the tree the sweep executes (`CCCI_REPO=/etc/cc-ci`)
|
||||
— has `WARM_CANONICAL = False`; only `07fc6d4` and `b5f2b10` have `True`, and `b5f2b10` ships the
|
||||
`canonical_ns()` fix so its seeds land in `canon-keycloak/`. The binding rule is **never deploy
|
||||
`07fc6d4`**.
|
||||
Remedy sketch: wrap both sites so a warmsnap failure still redeploys `last_good` (or, if restoring
|
||||
data is judged mandatory before redeploy, alert loudly + leave a breadcrumb rather than dying mid-
|
||||
rollback). Needs a decision on which is safer for a DB-backed app after a forward migration — that
|
||||
trade-off is why this is filed, not fixed inline.
|
||||
rollback), and make `roll_warm_infra()` propagate a non-zero rc so the failure is at least visible.
|
||||
Needs a decision on which is safer for a DB-backed app after a forward migration — that
|
||||
trade-off is why this is filed, not fixed inline. **Fixing it now would move the merge target off
|
||||
`b5f2b10`, against which the M2 PASS was given, so it stays deferred for the operator.**
|
||||
- [ ] **B-redfix-6 — `canonical_ns()` docstring says "the 15 existing canonicals"; the real number is 17,
|
||||
and the invariant is not a count** (COSMETIC; docs-only; no behaviour change). At
|
||||
`redfix-m2-harness`@`b5f2b10`, `runner/harness/canonical.py:52` reads "zero blast radius on the 15
|
||||
|
||||
@ -1,57 +0,0 @@
|
||||
# BUILDER-INBOX
|
||||
|
||||
## From Adversary — wake #44 (2026-07-09T08:47Z) — I was wrong, and `f64d102` inherited it
|
||||
|
||||
Not a gate message. **Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10`
|
||||
unchanged.** This is a correction to operator-facing STATUS text — including text you correctly took on my
|
||||
authority. The error was mine, not yours.
|
||||
|
||||
**What I got wrong.** A-redfix-2 told you the wedge persists because there is **no re-trigger**
|
||||
(`warm-keycloak.service`: oneshot, no timer, no `Restart=`) and heals on the next reboot / `nixos-rebuild
|
||||
switch`. **False.** `warm_reconcile.py` has a second invoker:
|
||||
|
||||
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
||||
└─ nightly_sweep.py main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
||||
(WARM_APPS = ["keycloak","traefik"], nightly_sweep.py:39,57-61)
|
||||
|
||||
Proof that does not depend on reading code: `warm-keycloak.service` last ran **2026-06-17 17:29:31**, but
|
||||
`/var/lib/ci-warm/keycloak/last_good` was written **2026-07-05 03:04:51.2** — 1.2 s after the sweep's last
|
||||
trigger (03:04:50). `write_last_good()` is reachable only from `reconcile()`.
|
||||
|
||||
**The two probes now in STATUS cannot fail** — please delete them. The timer is named `nightly-sweep`, not
|
||||
`*keycloak*`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak -- nix/` are both empty **while a
|
||||
timer drives it weekly**. Sound probe greps the callee: `git grep warm_reconcile -- runner/ nix/`.
|
||||
|
||||
**Corrected behaviour (once armed).** Sweep N: undeploy → `snapshot()` guard raises → keycloak DOWN. Sweep N+1
|
||||
(7 d): `is_deployed`=False → fresh-deploy branch → UP on the old version, `warmsnap` never called. Sweep N+2:
|
||||
DOWN again. **Alternating weeks, indefinitely** — not "recovery is manual" (your `68b51d5`) and not "heals on
|
||||
reboot" (my `b358b7e`). Worse: it is **silent**. `roll_warm_infra()` ignores the subprocess rc
|
||||
(`nightly_sweep.py:59-63`) so `nightly-sweep.service` reports `Result=success`, and the raise at `:514` is
|
||||
outside the `try/except` at `:520-525` (**that is B-redfix-5**) and upstream of every `write_alert()`.
|
||||
|
||||
**The reassuring half — and I checked before alarming you.** I suspected Sunday's sweep (2026-07-12 03:00)
|
||||
could arm the trap by itself. It cannot. Arming needs the enrollment **without** the fix, and the deployed tree
|
||||
the sweep runs (`CCCI_REPO=/etc/cc-ci`, = `origin/main`) has `WARM_CANONICAL = False`; only `07fc6d4` and
|
||||
`b5f2b10` have `True`. **No autonomous node activity can arm it.** Slot verified clean (only `last_good`);
|
||||
keycloak deployed and healthy.
|
||||
|
||||
So the precondition's landmark is wrong: a `git merge` executes nothing. The binding rule is **never deploy
|
||||
`07fc6d4`** — enrollment-without-`canonical_ns()` is the only state that can arm it. `b5f2b10` ships both, so
|
||||
seeds land in `canon-keycloak/`. The precondition is **self-maintaining**; the merge story is calmer than
|
||||
STATUS currently tells it.
|
||||
|
||||
**Three asks — STATUS-redfix.md text only, no code:**
|
||||
1. Delete both "no re-trigger" probe commands (they are unfalsifiable).
|
||||
2. Replace "recovery is manual" / "heals on reboot / `nixos-rebuild switch`" with the weekly DOWN/UP
|
||||
oscillation, and say it is **silent** (`Result=success`, no `write_alert`).
|
||||
3. Re-anchor the precondition: "slot clean at merge time" → "**never deploy `07fc6d4`**".
|
||||
|
||||
Minor, same file: `nixos-rebuild switch` re-runs reconcile **iff `runner/**` changed** (`ExecStart` embeds
|
||||
`/nix/store/…-runner/warm_reconcile.py`, and `warmsnap.py` is in that same derivation). An unrelated switch
|
||||
heals nothing; the merge's own activation switch *is* a reconcile trigger.
|
||||
|
||||
Full evidence + repro in **A-redfix-4** (BACKLOG-redfix.md). A-redfix-2 is partially withdrawn there: its
|
||||
first half (a code path *does* redeploy it) stands. Remediation if ever armed is unchanged: **delete the
|
||||
foreign `snapshot/`**, do not redeploy keycloak.
|
||||
|
||||
Delete this file when consumed.
|
||||
@ -1815,3 +1815,63 @@ Neither is unblocked work, so there is no backlog item left to hold. Untracked `
|
||||
per the joint Builder/Adversary position: neither of us created it, it is inert (no `go.mod`, `go` not
|
||||
installed, nothing on `:8080`, in no commit on any ref), and the guardrails say I do not delete files I
|
||||
did not create.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-09T08:55Z — wake #44: the Adversary refuted itself, and I had inherited the error
|
||||
|
||||
Re-opened the loop on a watchdog ping. The Adversary withdrew half of A-redfix-2 and filed A-redfix-4:
|
||||
`warm_reconcile.py` has a *second* invoker — `nightly-sweep.timer` (`OnCalendar=Sun`, `Persistent=true`)
|
||||
→ `nightly_sweep.roll_warm_infra()` → `subprocess warm_reconcile.py keycloak`. So the "no re-trigger /
|
||||
heals on reboot" mechanism I wrote into STATUS at `f64d102` is false.
|
||||
|
||||
**The part worth recording is how it got in.** At `f64d102` I accepted A-redfix-2 and copied its two probe
|
||||
commands into STATUS as the HOW-to-verify. Both probes grep the *app* name — `git grep warm-keycloak -- nix/`
|
||||
and `list-timers | grep -i keycloak`. The timer is named `nightly-sweep`. So both return empty *while a timer
|
||||
drives the thing weekly*: they return the all-clear for a reason unrelated to the property under test. I ran
|
||||
them, saw empty, and treated that as confirmation. It confirmed nothing. The Adversary named the class
|
||||
correctly — it is the same shape as its own `e3b0c44298fc1c14` empty-input tell on B-redfix-8.
|
||||
|
||||
The lesson is not "the Adversary was wrong" — it self-corrected. It is that *I re-ran probes handed to me
|
||||
rather than deriving probes from the property*. The property is "what invokes `reconcile()`", so the probe
|
||||
must grep the **callee** (`git grep warm_reconcile -- runner/ nix/`), which finds both invokers immediately.
|
||||
A probe supplied alongside the claim it is meant to test is not independent evidence.
|
||||
|
||||
So this wake I verified every leg of A-redfix-4 myself before editing:
|
||||
- `git grep warm_reconcile -- runner/ nix/` → `nix/modules/warm-keycloak.nix:26` **and**
|
||||
`runner/nightly_sweep.py:60`. Two invokers.
|
||||
- `systemctl cat nightly-sweep.timer` → `OnCalendar=Sun *-*-* 03:00:00`, `Persistent=true`; next
|
||||
`2026-07-12 03:05:34`, last `2026-07-05 03:04:50`.
|
||||
- The timing inference, which is the load-bearing one: `warm-keycloak.service` `ExecMainStartTimestamp` =
|
||||
`2026-06-17 17:29:31`, but `/var/lib/ci-warm/keycloak/last_good` mtime = `2026-07-05 03:04:51.209` — 18 days
|
||||
after the unit last ran, 1.2 s after the timer fired. `write_last_good()` has four call sites
|
||||
(`b5f2b10:478,484,491,527`), all inside `reconcile()`. Nothing else can write that file. ⇒ the sweep ran it.
|
||||
- Silence: `roll_warm_infra()` binds `rc` and only prints it (`:59-62`); every `write_alert()` site
|
||||
(`:493,500,503,539`) is downstream of the `snapshot()` raise at `:514`; `main():556` does not catch.
|
||||
- Cannot self-arm: `git show origin/main:tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = False`. Only
|
||||
`07fc6d4` and `b5f2b10` carry `True`. `/etc/cc-ci` is `origin/main`, so Sunday's sweep cannot arm the trap.
|
||||
- `ExecStart` → `/nix/store/fcqh…-runner/warm_reconcile.py`; `harness/warmsnap.py` is in that same
|
||||
derivation, so a switch re-runs reconcile iff `runner/**` changed.
|
||||
|
||||
I also caught a line-number slip while writing: the `write_alert` / `write_last_good` sites I first wrote were
|
||||
`main`'s coordinates, but every other ref in that STATUS block is `b5f2b10`'s (they sit two lines apart).
|
||||
Fixed to branch coordinates and said so in the text, since an operator will `git show` the branch.
|
||||
|
||||
Then I falsification-tested each *new* probe before committing it, which is the whole point: the sound probe
|
||||
must print both invokers, the fresh-deploy grep must return `0` warmsnap hits, the timer must print its
|
||||
schedule. All did.
|
||||
|
||||
**Severity re-recorded, scope unchanged.** B-redfix-5's consequence is worse than I had it — a silent weekly
|
||||
alternating SSO outage rather than a one-shot manual-recovery wedge — but its *reachability* is unchanged and
|
||||
still nil on this node. I did not fix it: the two-line `try/except` widening would move the merge target off
|
||||
`b5f2b10`, which is the sha the M2 PASS was given against, and the Adversary's asks were explicitly
|
||||
"STATUS text only, no code". Fixing it is the operator's call at merge time; the remedy sketch in
|
||||
BACKLOG-redfix.md now also notes `roll_warm_infra()` should propagate the rc so the failure is visible at all.
|
||||
|
||||
Re-anchored the merge precondition per ask 3. "Slot clean at merge time" named the wrong landmark — a
|
||||
`git merge` executes nothing, and the slot can only go foreign when a canonical seed *runs* from a tree with
|
||||
the enrollment but without `canonical_ns()`. That tree is `07fc6d4`, and only `07fc6d4`. `b5f2b10` ships both
|
||||
halves, so the precondition is self-maintaining. The operator-facing rule is now "never deploy `07fc6d4`",
|
||||
and the slot check is kept as a cheap not-armed assertion rather than as the rule itself.
|
||||
|
||||
Nothing reopens: M1 + M2 PASS stand, `## DONE` stands, no VETO, merge target `b5f2b10` unchanged.
|
||||
|
||||
@ -19,16 +19,27 @@ there is **no standing VETO**:
|
||||
|
||||
Adversary findings F-redfix-1/2/3/4 are all **CLOSED**; no open blocking finding.
|
||||
|
||||
The three `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
|
||||
The four `A-redfix-*` findings are also non-blocking and are accounted for here, so nothing dangles:
|
||||
- **A-redfix-1** (`/etc/cc-ci/.git/config` is 0644 and embeds the Gitea bot password) — out of redfix scope,
|
||||
pre-existing undeclared server state, no DoD item. Mirrors my B-redfix-7. Not created by this phase.
|
||||
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **ACCEPTED**; the mechanism and remediation
|
||||
sentences in the B-redfix-5 precondition below are corrected. The precondition itself is unchanged.
|
||||
- **A-redfix-2** (the B-redfix-5 wedge's self-heal mechanism) — **PARTIALLY WITHDRAWN by the Adversary at
|
||||
wake #44.** Its first half stands (a code path *does* redeploy keycloak — the fresh-deploy branch). Its
|
||||
second half ("no re-trigger; heals on reboot") is **false** and is superseded by A-redfix-4.
|
||||
- **A-redfix-3** (the `main.go` "both clones" evidence) — **ACCEPTED**; that evidence is retracted below.
|
||||
`/srv/cc-ci` is a symlink to `/srv/cc-ci-orch`. The file's origin remains unexplained; risk inert.
|
||||
- **A-redfix-4** (a weekly `nightly-sweep.timer` *does* re-trigger reconcile; the wedge is **silent** and
|
||||
**recurring**) — **ACCEPTED and independently re-verified against the node**, not taken on authority. The
|
||||
B-redfix-5 precondition below is rewritten accordingly: two unfalsifiable probes deleted, the wedge
|
||||
described as a weekly alternating DOWN/UP outage that raises no alert, and the precondition re-anchored
|
||||
from "slot clean at merge time" to **"never deploy `07fc6d4`"**. The trap is **not armed** and **cannot be
|
||||
armed by autonomous node activity** (`origin/main`, the tree the sweep runs, has `WARM_CANONICAL = False`).
|
||||
|
||||
A-redfix-2/3 corrected operator-facing *descriptions*, not DoD items; the Adversary confirmed no VETO at
|
||||
wake #43 (`b358b7e`, 2026-07-09T08:31Z).
|
||||
All four corrected operator-facing *descriptions*, not DoD items. No VETO at wake #43 (`b358b7e`) or wake #44
|
||||
(`84a1666`, 2026-07-09T08:47Z). **Note on provenance:** A-redfix-2's false half entered this file at `f64d102`
|
||||
because I adopted the Adversary's stated mechanism *and its two probe commands* without re-deriving them. The
|
||||
probes returned the all-clear for a reason unrelated to the property under test (they grep the app name;
|
||||
the timer is named `nightly-sweep`). Both are now deleted, and every claim in the rewritten block below was
|
||||
re-checked first-hand.
|
||||
|
||||
## ⚠ OPERATOR ACTION REQUIRED — B-redfix-8 (HIGH, open, outside DoD)
|
||||
|
||||
@ -83,72 +94,108 @@ and the still-open non-blocking B-redfix-5, are below.
|
||||
One deferred, non-blocking item remains recorded (NOT part of any DoD item, NOT a standing finding):
|
||||
**B-redfix-5** in BACKLOG-redfix.md — in `warm_reconcile.py`, TWO post-`abra.undeploy()` calls sit outside
|
||||
the upgrade's `try/except`: `warmsnap.snapshot()` on the upgrade path (`:514`) and `warmsnap.restore()` on
|
||||
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**. Pre-dates the
|
||||
enrollment (present at `07fc6d4`). Adversary concurred it is not a VETO.
|
||||
the rollback path (`:536`). If either raises, live keycloak is left **undeployed**, and because both sit
|
||||
upstream of every `write_alert()` call site the failure is **silent**. Pre-dates the enrollment (present at
|
||||
`07fc6d4`). Adversary concurred it is not a VETO. Its *consequence* is worse than first recorded — see the
|
||||
merge precondition below: reached only via a foreign live-slot meta, which nothing on the node can create
|
||||
today, but if ever reached it produces a weekly alternating SSO outage that reports `Result=success`.
|
||||
|
||||
**MERGE PRECONDITION for `b5f2b10` (verify at merge time).** The F-redfix-4 fix *adds* a raise path to both
|
||||
of those calls — `warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in
|
||||
`restore()`). It raises iff the live slot's `snapshot/meta.json` records a `domain` other than the one
|
||||
passed. Composed with B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path
|
||||
this fires on *every* stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today**
|
||||
because the live slot holds no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free
|
||||
to claim"). It is unreachable for that reason — **not** because F-redfix-4 is closed.
|
||||
**MERGE PRECONDITION for `b5f2b10`.** The F-redfix-4 fix *adds* a raise path to both of those calls —
|
||||
`warmsnap._assert_slot_not_foreign()` (`warmsnap.py:158` in `snapshot()`, `:209` in `restore()`). It raises
|
||||
iff the live slot's `snapshot/meta.json` records a `domain` other than the one passed. Composed with
|
||||
B-redfix-5, a foreign live-slot meta wedges live keycloak: on the **upgrade** path this fires on *every*
|
||||
stateful auto-upgrade, not only on rollback. It is **unreachable on cc-ci today** because the live slot holds
|
||||
no snapshot at all (`read_meta` → `None` → "a slot with no snapshot yet is free to claim"). It is unreachable
|
||||
for that reason — **not** because F-redfix-4 is closed.
|
||||
|
||||
The only state that creates a foreign live-slot meta is a **pre-fix (`07fc6d4`) canonical seed**, which
|
||||
wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. The seed is a real, not
|
||||
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
|
||||
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
|
||||
**THE BINDING RULE: never deploy `07fc6d4`.** (Re-anchored 2026-07-09 per Adversary **A-redfix-4**; the
|
||||
earlier "verify the slot is clean at merge time" framing named the wrong landmark — *a `git merge` executes
|
||||
nothing*.) The only state that creates a foreign live-slot meta is a **canonical keycloak seed executed from
|
||||
a tree that carries the enrollment but not `canonical_ns()`**. Exactly one such tree exists: `07fc6d4`
|
||||
(`tests/keycloak/recipe_meta.py` → `WARM_CANONICAL = True`, no `canonical_ns()`). `b5f2b10` ships enrollment
|
||||
**and** fix together, so its canonical seeds land in `canon-keycloak/` and the live slot never goes foreign.
|
||||
The precondition is therefore **self-maintaining across the merge** — deploying the merge target cannot arm
|
||||
the trap; only deploying the intermediate `07fc6d4` can.
|
||||
|
||||
**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
|
||||
(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
|
||||
it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
|
||||
**No autonomous node activity can arm it.** The tree the weekly sweep executes is `CCCI_REPO=/etc/cc-ci` =
|
||||
`origin/main`, where `WARM_CANONICAL = False`. Only `07fc6d4` and `b5f2b10` have `True`.
|
||||
|
||||
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
|
||||
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
|
||||
with live keycloak already undeployed by the preceding `abra.undeploy()`.
|
||||
**If it is ever armed, the wedge is SILENT and RECURRING — a weekly alternating outage of the shared SSO
|
||||
provider.** (Corrected 2026-07-09 per **A-redfix-4**, which withdrew the "no re-trigger / heals on reboot"
|
||||
half of A-redfix-2. My own earlier "recovery is manual" (`68b51d5`) was also wrong. Re-derived independently
|
||||
against the node — see HOW below.) `warm_reconcile.py` has **two** invokers, and the second is a weekly timer:
|
||||
|
||||
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
|
||||
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
|
||||
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
|
||||
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
|
||||
block it.
|
||||
nightly-sweep.timer (OnCalendar=Sun *-*-* 03:00:00, Persistent=true)
|
||||
└─ nightly_sweep.main():147 → roll_warm_infra() → subprocess warm_reconcile.py keycloak
|
||||
(WARM_APPS = ["keycloak", "traefik"], nightly_sweep.py:39,57-61)
|
||||
|
||||
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
|
||||
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
|
||||
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
|
||||
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
|
||||
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
|
||||
Once armed, with `current != latest`: **sweep N** → `abra.undeploy()` → `snapshot()` guard raises → keycloak
|
||||
**DOWN**. **Sweep N+1** (7 days later) → `is_deployed` is `False` → the **fresh-deploy branch**
|
||||
(`:471-479`), which **never calls `warmsnap`** → keycloak **UP** on the old version. **Sweep N+2** → upgrade
|
||||
path → **DOWN** again. **Alternating weeks, indefinitely**, because the foreign `snapshot/meta.json` is never
|
||||
removed. (`abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n` and does **not** remove the
|
||||
app `.env`, so `current_version` stays resolvable — this is why the fresh-deploy branch is the one taken.)
|
||||
|
||||
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
|
||||
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
|
||||
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
|
||||
upgrade re-wedges.
|
||||
**And it raises no alert.** `roll_warm_infra()` captures the subprocess rc and discards it
|
||||
(`nightly_sweep.py:59-62` — `rc` is only ever printed), so `nightly-sweep.service` still reports
|
||||
`Result=success`. The raise at `:514` is outside the only `try/except` (`:520-525`, which wraps
|
||||
`deploy_version` + `wait_healthy` — *that gap is B-redfix-5*) and upstream of **every** `write_alert()` call
|
||||
site (`:493,500,503,539`). `reconcile()`'s sole caller `main():556` does not catch either. A weekly SSO
|
||||
outage would surface **nowhere**. (All line numbers here are `b5f2b10` coordinates, matching the rest of this
|
||||
block; on `main` the same sites sit two lines earlier.)
|
||||
|
||||
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
|
||||
**REMEDIATION if ever armed: delete the foreign `snapshot/` from the slot** (or fix B-redfix-5).
|
||||
**Redeploying keycloak does NOT fix it** — the next sweep re-wedges. This is the one part of the old text
|
||||
that was right, and it is the part that matters operationally.
|
||||
|
||||
**HOW to verify** (from a clone at `b5f2b10`). Note the probe greps the **callee**, never the app name: the
|
||||
invoking timer is called `nightly-sweep`, so `list-timers | grep -i keycloak` and `git grep warm-keycloak --
|
||||
nix/` are both empty **while a timer drives it weekly**. Those two probes were in this file until 2026-07-09
|
||||
and are unfalsifiable by construction; they have been deleted, not fixed.
|
||||
|
||||
git grep -n warm_reconcile -- runner/ nix/ # SOUND probe: finds both invokers
|
||||
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
||||
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
||||
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
|
||||
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
|
||||
ssh cc-ci 'systemctl cat nightly-sweep.timer | grep -iE "OnCalendar|Persistent"'
|
||||
ssh cc-ci 'systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||||
|
||||
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
|
||||
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
|
||||
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
|
||||
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
|
||||
**EXPECTED:** `git grep warm_reconcile` prints both `nix/modules/warm-keycloak.nix:26` (the oneshot unit) and
|
||||
`runner/nightly_sweep.py:60` (the weekly sweep's subprocess); `undeploy()` is a single
|
||||
`_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; `:472` is `if not deployed:` returning
|
||||
`deployed-fresh:{target}` at `:479` with no `warmsnap` between; the timer prints
|
||||
`OnCalendar=Sun *-*-* 03:00:00` + `Persistent=true`.
|
||||
|
||||
**Observational proof the sweep — not the unit — drives reconcile** (independent of reading any code;
|
||||
observed 2026-07-09):
|
||||
|
||||
ssh cc-ci 'systemctl show warm-keycloak.service -p ExecMainStartTimestamp
|
||||
stat -c "%y %n" /var/lib/ci-warm/keycloak/last_good
|
||||
systemctl list-timers --all nightly-sweep.timer --no-pager'
|
||||
|
||||
**EXPECTED:** unit last ran `Wed 2026-06-17 17:29:31 UTC`; `last_good` mtime `2026-07-05 03:04:51.209`; timer
|
||||
last trigger `Sun 2026-07-05 03:04:50 UTC`. The slot was written **18 days after the unit last ran and 1.2 s
|
||||
after the timer fired**, and `write_last_good()` is reachable **only** from `reconcile()` (call sites
|
||||
`:478,484,491,527`, all inside it) ⇒ `reconcile("keycloak")` executed from the sweep.
|
||||
|
||||
**Also:** `warm-keycloak.service`'s `ExecStart` resolves to `/nix/store/…-runner/warm_reconcile.py`, and
|
||||
`harness/warmsnap.py` lives in that **same** derivation. So `nixos-rebuild switch` re-runs reconcile **iff
|
||||
`runner/**` changed** — an unrelated switch heals nothing, while the merge's own activation switch *is* a
|
||||
reconcile trigger (alongside the Sunday sweep).
|
||||
|
||||
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
||||
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
||||
warm unit reaches either call site.
|
||||
|
||||
**HOW to verify the precondition** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
|
||||
**HOW to verify the trap is not armed** (on cc-ci; `python3` is absent there, so `ls`/`cat`):
|
||||
|
||||
ssh cc-ci 'ls -A /var/lib/ci-warm/keycloak/; cat /var/lib/ci-warm/keycloak/snapshot/meta.json 2>&1'
|
||||
|
||||
**EXPECTED (observed 2026-07-09T08:2xZ):** `last_good` only, and `cat` reports
|
||||
`No such file or directory` — no `snapshot/`, hence no `meta.json`. **A `meta.json` whose `"domain"` is
|
||||
anything other than `warm-keycloak.ci.commoninternet.net` means the precondition is violated:** fix B-redfix-5
|
||||
(or delete the foreign slot meta) BEFORE merging `b5f2b10`. A `meta.json` recording
|
||||
`warm-keycloak.ci.commoninternet.net` is self-consistent and passes the guard.
|
||||
**EXPECTED (observed 2026-07-09T08:5xZ):** `last_good` only, and `cat` reports `No such file or directory` —
|
||||
no `snapshot/`, hence no `meta.json`; no `/var/lib/ci-warm/canon-keycloak` either; live keycloak `200`. **A
|
||||
`meta.json` whose `"domain"` is anything other than `warm-keycloak.ci.commoninternet.net` means the trap is
|
||||
armed:** delete that foreign `snapshot/` (or fix B-redfix-5) **before the next Sunday sweep**, not merely
|
||||
before the merge. A `meta.json` recording `warm-keycloak.ci.commoninternet.net` is self-consistent and passes
|
||||
the guard.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user