status(redfix): accept A-redfix-2 + A-redfix-3 — correct the self-heal mechanism and retract the "both clones" evidence
Both claims re-derived independently before editing; both of mine were wrong.
A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.
A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).
Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
This commit is contained in:
@ -88,10 +88,42 @@ wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`.
|
||||
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
|
||||
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
|
||||
|
||||
**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps
|
||||
`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514`
|
||||
or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding
|
||||
`abra.undeploy()` — no code path redeploys it. Recovery is manual.
|
||||
**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
|
||||
(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
|
||||
it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
|
||||
|
||||
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
|
||||
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
|
||||
with live keycloak already undeployed by the preceding `abra.undeploy()`.
|
||||
|
||||
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
|
||||
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
|
||||
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
|
||||
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
|
||||
block it.
|
||||
|
||||
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
|
||||
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
|
||||
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
|
||||
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
|
||||
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
|
||||
|
||||
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
|
||||
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
|
||||
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
|
||||
upgrade re-wedges.
|
||||
|
||||
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
|
||||
|
||||
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
||||
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
||||
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
|
||||
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
|
||||
|
||||
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
|
||||
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
|
||||
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
|
||||
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
|
||||
|
||||
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
||||
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
||||
@ -467,13 +499,23 @@ Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left
|
||||
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
|
||||
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
|
||||
|
||||
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones,
|
||||
identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation
|
||||
cannot leave the same untracked file in two clones).
|
||||
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.**
|
||||
|
||||
**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
|
||||
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
|
||||
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
|
||||
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
|
||||
file in two clones" inference therefore carries **no information** about the file's origin, which remains
|
||||
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
|
||||
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
|
||||
|
||||
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
|
||||
`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`.
|
||||
`git status --short` → `?? main.go`.
|
||||
`ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
|
||||
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
|
||||
|
||||
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
|
||||
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
|
||||
|
||||
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
|
||||
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.
|
||||
|
||||
Reference in New Issue
Block a user