status(redfix): accept A-redfix-2 + A-redfix-3 — correct the self-heal mechanism and retract the "both clones" evidence

Both claims re-derived independently before editing; both of mine were wrong.

A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.

A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).

Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
This commit is contained in:
2026-07-09 08:35:51 +00:00
parent 10576574c5
commit 8276ecdfe0
2 changed files with 84 additions and 8 deletions

View File

@ -88,10 +88,42 @@ wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`.
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps
`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514`
or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding
`abra.undeploy()` — no code path redeploys it. Recovery is manual.
**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
with live keycloak already undeployed by the preceding `abra.undeploy()`.
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
block it.
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
upgrade re-wedges.
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
@ -467,13 +499,23 @@ Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones,
identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation
cannot leave the same untracked file in two clones).
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **one file in one clone.**
**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
file in two clones" inference therefore carries **no information** about the file's origin, which remains
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`.
`git status --short``?? main.go`.
`ls -la /srv/ | grep cc-ci``cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.