status(redfix): accept A-redfix-2 + A-redfix-3 — correct the self-heal mechanism and retract the "both clones" evidence

Both claims re-derived independently before editing; both of mine were wrong.

A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.

A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).

Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
This commit is contained in:
2026-07-09 08:35:51 +00:00
parent 10576574c5
commit 8276ecdfe0
2 changed files with 84 additions and 8 deletions

View File

@ -1748,3 +1748,37 @@ Both loops have now independently derived the same structure from source and agr
deferred, correctly scoped to both call sites; it still needs the operator's safety call (redeploy
`last_good` on unrestored data vs. die loudly), which is exactly why it was filed rather than fixed inline.
No VETO, no gate, no DoD item touched. Phase remains `## DONE`. B-redfix-8 untouched — operator rotation.
## Wake #43 (Builder) — 2026-07-09T08:3xZ — I accept A-redfix-2 and A-redfix-3; both of my claims were wrong
The Adversary probed two operator-facing claims I wrote (`68b51d5`, `42bc4e4`) and refuted both. I re-derived
each from source/node before editing — not taking the verdict on faith — and both hold up.
**A-redfix-2 (mechanism wrong, conclusion right).** I wrote "no code path redeploys it. Recovery is manual."
The conclusion (keycloak stays down) is right; the *reason* I gave is false. I reasoned from the absence of a
`try/except` around the `abra.undeploy()` sites and stopped there, never asking what the *next* invocation of
`reconcile()` does. It takes the fresh-deploy branch (`:471-479`): `abra.undeploy()` doesn't remove the app
`.env`, so `current_version` still resolves while `is_deployed` is False. That branch redeploys and never calls
`warmsnap`, so the foreign meta can't block it. What actually stops the healing is that nothing *re-runs* the
unit — `warm-keycloak.service` is `Type=oneshot`/`RemainAfterExit=true` with no timer and no `Restart=`
(confirmed on the node). So it heals on the next reboot/`nixos-rebuild switch` and **re-wedges on the next due
upgrade**, because the foreign `snapshot/meta.json` is still in the slot.
Why this correction matters and is not cosmetic: my wording implied the remedy was to *redeploy keycloak*. An
operator who did that would see it come back, conclude it was fixed, and get re-wedged at the next upgrade. The
actual remedy is **deleting the foreign `snapshot/`**. My text hid the recurrence and pointed at the wrong fix.
It also mis-describes the failure's *shape*: intermittent flake, not permanent outage.
**A-redfix-3 (evidence void).** I claimed `main.go` was in "both clones", inferring it was written by something
outside git. But `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch` (`ls -la /srv/`) — the two paths are the same
inode (`3254604`, `links=1`). One file, seen twice. The inference carries zero information; the file's origin is
still unexplained. Lesson: I treated two *paths* as two *clones* without stat-ing them. Cheap check, skipped.
Both edits are docs-only, in STATUS-redfix.md. No DoD item, gate, or verdict is affected: `## DONE` stands,
M1 + M2 PASS stand, merge target `b5f2b10` unchanged, no VETO. The B-redfix-5 merge precondition itself is
unchanged and still correct — A-redfix-2 corrects only the mechanism + remediation sentences beneath it.
Also re-probed B-redfix-8 this wake with the sound (local, not `ssh`) probe: `3fcea78925015fc9` → **still
unrotated**, and the empty-input control printed `e3b0c44298fc1c14`, distinct — so the probe read the real file
rather than silently hashing nothing. B-redfix-8 remains OPEN/HIGH/operator-rotation-only. Class-A1 credential;
I must not rotate it.

View File

@ -88,10 +88,42 @@ wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`.
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps
`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514`
or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding
`abra.undeploy()` — no code path redeploys it. Recovery is manual.
**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
with live keycloak already undeployed by the preceding `abra.undeploy()`.
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
block it.
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
upgrade re-wedges.
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
@ -467,13 +499,23 @@ Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones,
identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation
cannot leave the same untracked file in two clones).
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **one file in one clone.**
**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
file in two clones" inference therefore carries **no information** about the file's origin, which remains
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`.
`git status --short``?? main.go`.
`ls -la /srv/ | grep cc-ci``cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.