status(redfix): accept A-redfix-2 + A-redfix-3 — correct the self-heal mechanism and retract the "both clones" evidence

Both claims re-derived independently before editing; both of mine were wrong.

A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.

A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).

Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
This commit is contained in:
2026-07-09 08:35:51 +00:00
parent 10576574c5
commit 8276ecdfe0
2 changed files with 84 additions and 8 deletions

View File

@ -1748,3 +1748,37 @@ Both loops have now independently derived the same structure from source and agr
deferred, correctly scoped to both call sites; it still needs the operator's safety call (redeploy deferred, correctly scoped to both call sites; it still needs the operator's safety call (redeploy
`last_good` on unrestored data vs. die loudly), which is exactly why it was filed rather than fixed inline. `last_good` on unrestored data vs. die loudly), which is exactly why it was filed rather than fixed inline.
No VETO, no gate, no DoD item touched. Phase remains `## DONE`. B-redfix-8 untouched — operator rotation. No VETO, no gate, no DoD item touched. Phase remains `## DONE`. B-redfix-8 untouched — operator rotation.
## Wake #43 (Builder) — 2026-07-09T08:3xZ — I accept A-redfix-2 and A-redfix-3; both of my claims were wrong
The Adversary probed two operator-facing claims I wrote (`68b51d5`, `42bc4e4`) and refuted both. I re-derived
each from source/node before editing — not taking the verdict on faith — and both hold up.
**A-redfix-2 (mechanism wrong, conclusion right).** I wrote "no code path redeploys it. Recovery is manual."
The conclusion (keycloak stays down) is right; the *reason* I gave is false. I reasoned from the absence of a
`try/except` around the `abra.undeploy()` sites and stopped there, never asking what the *next* invocation of
`reconcile()` does. It takes the fresh-deploy branch (`:471-479`): `abra.undeploy()` doesn't remove the app
`.env`, so `current_version` still resolves while `is_deployed` is False. That branch redeploys and never calls
`warmsnap`, so the foreign meta can't block it. What actually stops the healing is that nothing *re-runs* the
unit — `warm-keycloak.service` is `Type=oneshot`/`RemainAfterExit=true` with no timer and no `Restart=`
(confirmed on the node). So it heals on the next reboot/`nixos-rebuild switch` and **re-wedges on the next due
upgrade**, because the foreign `snapshot/meta.json` is still in the slot.
Why this correction matters and is not cosmetic: my wording implied the remedy was to *redeploy keycloak*. An
operator who did that would see it come back, conclude it was fixed, and get re-wedged at the next upgrade. The
actual remedy is **deleting the foreign `snapshot/`**. My text hid the recurrence and pointed at the wrong fix.
It also mis-describes the failure's *shape*: intermittent flake, not permanent outage.
**A-redfix-3 (evidence void).** I claimed `main.go` was in "both clones", inferring it was written by something
outside git. But `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch` (`ls -la /srv/`) — the two paths are the same
inode (`3254604`, `links=1`). One file, seen twice. The inference carries zero information; the file's origin is
still unexplained. Lesson: I treated two *paths* as two *clones* without stat-ing them. Cheap check, skipped.
Both edits are docs-only, in STATUS-redfix.md. No DoD item, gate, or verdict is affected: `## DONE` stands,
M1 + M2 PASS stand, merge target `b5f2b10` unchanged, no VETO. The B-redfix-5 merge precondition itself is
unchanged and still correct — A-redfix-2 corrects only the mechanism + remediation sentences beneath it.
Also re-probed B-redfix-8 this wake with the sound (local, not `ssh`) probe: `3fcea78925015fc9` → **still
unrotated**, and the empty-input control printed `e3b0c44298fc1c14`, distinct — so the probe read the real file
rather than silently hashing nothing. B-redfix-8 remains OPEN/HIGH/operator-rotation-only. Class-A1 credential;
I must not rotate it.

View File

@ -88,10 +88,42 @@ wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`.
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4` hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge. too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps **The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514` (Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
`abra.undeploy()` — no code path redeploys it. Recovery is manual.
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
with live keycloak already undeployed by the preceding `abra.undeploy()`.
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
block it.
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
upgrade re-wedges.
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry **Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
@ -467,13 +499,23 @@ Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item. repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it. No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, **WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **one file in one clone.**
identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation
cannot leave the same untracked file in two clones). **RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
file in two clones" inference therefore carries **no information** about the file's origin, which remains
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit). **HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`.
`git status --short``?? main.go`. `git status --short``?? main.go`.
`ls -la /srv/ | grep cc-ci``cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create; **EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict. flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.