status(redfix): accept A-redfix-2 + A-redfix-3 — correct the self-heal mechanism and retract the "both clones" evidence
Both claims re-derived independently before editing; both of mine were wrong.
A-redfix-2: "no code path redeploys it" is FALSE. abra.undeploy() leaves the app
.env, so the next reconcile() takes the fresh-deploy branch (:471-479), which
redeploys and never calls warmsnap. Self-healing is blocked by the absence of a
re-trigger (warm-keycloak.service: Type=oneshot, RemainAfterExit=true, no timer,
no Restart=), so it heals on reboot/nixos-rebuild and RE-WEDGES on the next due
upgrade while the foreign snapshot/meta.json remains. Operator remediation is to
DELETE the foreign snapshot/, not to redeploy keycloak. Failure presents as an
intermittent flake, not a permanent outage.
A-redfix-3: /srv/cc-ci is a symlink to /srv/cc-ci-orch, so the two main.go paths
are the same inode (3254604, links=1). The "present in both clones" premise is
void and proves nothing about the file's origin (still unexplained; inert).
Docs-only. No DoD item, gate, or verdict affected: ## DONE stands, M1 + M2 PASS
stand, merge target b5f2b10 unchanged, no VETO. The B-redfix-5 merge precondition
is unchanged and still correct.
This commit is contained in:
@ -1748,3 +1748,37 @@ Both loops have now independently derived the same structure from source and agr
|
||||
deferred, correctly scoped to both call sites; it still needs the operator's safety call (redeploy
|
||||
`last_good` on unrestored data vs. die loudly), which is exactly why it was filed rather than fixed inline.
|
||||
No VETO, no gate, no DoD item touched. Phase remains `## DONE`. B-redfix-8 untouched — operator rotation.
|
||||
|
||||
## Wake #43 (Builder) — 2026-07-09T08:3xZ — I accept A-redfix-2 and A-redfix-3; both of my claims were wrong
|
||||
|
||||
The Adversary probed two operator-facing claims I wrote (`68b51d5`, `42bc4e4`) and refuted both. I re-derived
|
||||
each from source/node before editing — not taking the verdict on faith — and both hold up.
|
||||
|
||||
**A-redfix-2 (mechanism wrong, conclusion right).** I wrote "no code path redeploys it. Recovery is manual."
|
||||
The conclusion (keycloak stays down) is right; the *reason* I gave is false. I reasoned from the absence of a
|
||||
`try/except` around the `abra.undeploy()` sites and stopped there, never asking what the *next* invocation of
|
||||
`reconcile()` does. It takes the fresh-deploy branch (`:471-479`): `abra.undeploy()` doesn't remove the app
|
||||
`.env`, so `current_version` still resolves while `is_deployed` is False. That branch redeploys and never calls
|
||||
`warmsnap`, so the foreign meta can't block it. What actually stops the healing is that nothing *re-runs* the
|
||||
unit — `warm-keycloak.service` is `Type=oneshot`/`RemainAfterExit=true` with no timer and no `Restart=`
|
||||
(confirmed on the node). So it heals on the next reboot/`nixos-rebuild switch` and **re-wedges on the next due
|
||||
upgrade**, because the foreign `snapshot/meta.json` is still in the slot.
|
||||
|
||||
Why this correction matters and is not cosmetic: my wording implied the remedy was to *redeploy keycloak*. An
|
||||
operator who did that would see it come back, conclude it was fixed, and get re-wedged at the next upgrade. The
|
||||
actual remedy is **deleting the foreign `snapshot/`**. My text hid the recurrence and pointed at the wrong fix.
|
||||
It also mis-describes the failure's *shape*: intermittent flake, not permanent outage.
|
||||
|
||||
**A-redfix-3 (evidence void).** I claimed `main.go` was in "both clones", inferring it was written by something
|
||||
outside git. But `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch` (`ls -la /srv/`) — the two paths are the same
|
||||
inode (`3254604`, `links=1`). One file, seen twice. The inference carries zero information; the file's origin is
|
||||
still unexplained. Lesson: I treated two *paths* as two *clones* without stat-ing them. Cheap check, skipped.
|
||||
|
||||
Both edits are docs-only, in STATUS-redfix.md. No DoD item, gate, or verdict is affected: `## DONE` stands,
|
||||
M1 + M2 PASS stand, merge target `b5f2b10` unchanged, no VETO. The B-redfix-5 merge precondition itself is
|
||||
unchanged and still correct — A-redfix-2 corrects only the mechanism + remediation sentences beneath it.
|
||||
|
||||
Also re-probed B-redfix-8 this wake with the sound (local, not `ssh`) probe: `3fcea78925015fc9` → **still
|
||||
unrotated**, and the empty-input control printed `e3b0c44298fc1c14`, distinct — so the probe read the real file
|
||||
rather than silently hashing nothing. B-redfix-8 remains OPEN/HIGH/operator-rotation-only. Class-A1 credential;
|
||||
I must not rotate it.
|
||||
|
||||
@ -88,10 +88,42 @@ wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`.
|
||||
hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4`
|
||||
too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge.
|
||||
|
||||
**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps
|
||||
`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514`
|
||||
or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding
|
||||
`abra.undeploy()` — no code path redeploys it. Recovery is manual.
|
||||
**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.**
|
||||
(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys
|
||||
it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.)
|
||||
|
||||
`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`),
|
||||
and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit`
|
||||
with live keycloak already undeployed by the preceding `abra.undeploy()`.
|
||||
|
||||
**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it
|
||||
does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`)
|
||||
is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch**
|
||||
(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot
|
||||
block it.
|
||||
|
||||
**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is
|
||||
`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its
|
||||
own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy
|
||||
branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at
|
||||
`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot.
|
||||
|
||||
**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal
|
||||
across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to
|
||||
**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due
|
||||
upgrade re-wedges.
|
||||
|
||||
**HOW to verify** (all four legs, from a clone at `b5f2b10`):
|
||||
|
||||
git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal
|
||||
git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap
|
||||
git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger)
|
||||
ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="'
|
||||
|
||||
**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling;
|
||||
`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between;
|
||||
the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no**
|
||||
`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty.
|
||||
|
||||
**Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry
|
||||
with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other
|
||||
@ -467,13 +499,23 @@ Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left
|
||||
repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item.
|
||||
No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it.
|
||||
|
||||
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones,
|
||||
identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation
|
||||
cannot leave the same untracked file in two clones).
|
||||
**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.**
|
||||
|
||||
**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was
|
||||
"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci`
|
||||
is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are
|
||||
the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked
|
||||
file in two clones" inference therefore carries **no information** about the file's origin, which remains
|
||||
**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and
|
||||
`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`.
|
||||
|
||||
**HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit).
|
||||
`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`.
|
||||
`git status --short` → `?? main.go`.
|
||||
`ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim).
|
||||
`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`.
|
||||
|
||||
**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not
|
||||
installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file.
|
||||
|
||||
**EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create;
|
||||
flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.
|
||||
|
||||
Reference in New Issue
Block a user