diff --git a/machine-docs/JOURNAL-redfix.md b/machine-docs/JOURNAL-redfix.md index cdc648e..65357ac 100644 --- a/machine-docs/JOURNAL-redfix.md +++ b/machine-docs/JOURNAL-redfix.md @@ -1748,3 +1748,37 @@ Both loops have now independently derived the same structure from source and agr deferred, correctly scoped to both call sites; it still needs the operator's safety call (redeploy `last_good` on unrestored data vs. die loudly), which is exactly why it was filed rather than fixed inline. No VETO, no gate, no DoD item touched. Phase remains `## DONE`. B-redfix-8 untouched — operator rotation. + +## Wake #43 (Builder) — 2026-07-09T08:3xZ — I accept A-redfix-2 and A-redfix-3; both of my claims were wrong + +The Adversary probed two operator-facing claims I wrote (`68b51d5`, `42bc4e4`) and refuted both. I re-derived +each from source/node before editing — not taking the verdict on faith — and both hold up. + +**A-redfix-2 (mechanism wrong, conclusion right).** I wrote "no code path redeploys it. Recovery is manual." +The conclusion (keycloak stays down) is right; the *reason* I gave is false. I reasoned from the absence of a +`try/except` around the `abra.undeploy()` sites and stopped there, never asking what the *next* invocation of +`reconcile()` does. It takes the fresh-deploy branch (`:471-479`): `abra.undeploy()` doesn't remove the app +`.env`, so `current_version` still resolves while `is_deployed` is False. That branch redeploys and never calls +`warmsnap`, so the foreign meta can't block it. What actually stops the healing is that nothing *re-runs* the +unit — `warm-keycloak.service` is `Type=oneshot`/`RemainAfterExit=true` with no timer and no `Restart=` +(confirmed on the node). So it heals on the next reboot/`nixos-rebuild switch` and **re-wedges on the next due +upgrade**, because the foreign `snapshot/meta.json` is still in the slot. + +Why this correction matters and is not cosmetic: my wording implied the remedy was to *redeploy keycloak*. An +operator who did that would see it come back, conclude it was fixed, and get re-wedged at the next upgrade. The +actual remedy is **deleting the foreign `snapshot/`**. My text hid the recurrence and pointed at the wrong fix. +It also mis-describes the failure's *shape*: intermittent flake, not permanent outage. + +**A-redfix-3 (evidence void).** I claimed `main.go` was in "both clones", inferring it was written by something +outside git. But `/srv/cc-ci` is a symlink to `/srv/cc-ci-orch` (`ls -la /srv/`) — the two paths are the same +inode (`3254604`, `links=1`). One file, seen twice. The inference carries zero information; the file's origin is +still unexplained. Lesson: I treated two *paths* as two *clones* without stat-ing them. Cheap check, skipped. + +Both edits are docs-only, in STATUS-redfix.md. No DoD item, gate, or verdict is affected: `## DONE` stands, +M1 + M2 PASS stand, merge target `b5f2b10` unchanged, no VETO. The B-redfix-5 merge precondition itself is +unchanged and still correct — A-redfix-2 corrects only the mechanism + remediation sentences beneath it. + +Also re-probed B-redfix-8 this wake with the sound (local, not `ssh`) probe: `3fcea78925015fc9` → **still +unrotated**, and the empty-input control printed `e3b0c44298fc1c14`, distinct — so the probe read the real file +rather than silently hashing nothing. B-redfix-8 remains OPEN/HIGH/operator-rotation-only. Class-A1 credential; +I must not rotate it. diff --git a/machine-docs/STATUS-redfix.md b/machine-docs/STATUS-redfix.md index b8cb6a9..6a10cdc 100644 --- a/machine-docs/STATUS-redfix.md +++ b/machine-docs/STATUS-redfix.md @@ -88,10 +88,42 @@ wrote the canonical's domain into the shared slot `/var/lib/ci-warm/keycloak/`. hypothetical, arming action: `tests/keycloak/recipe_meta.py` carries `WARM_CANONICAL = True` at `07fc6d4` too. So: do not run a canonical keycloak seed from `07fc6d4` between now and the merge. -**The wedge does not self-heal.** `reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps -`deploy_version` + `wait_healthy`), and its sole caller `main():556` does not catch either. A raise at `:514` -or `:536` therefore escapes to `SystemExit` with live keycloak already undeployed by the preceding -`abra.undeploy()` — no code path redeploys it. Recovery is manual. +**The wedge leaves keycloak undeployed, and the fix is to delete the foreign `snapshot/` — NOT to redeploy.** +(Corrected 2026-07-09 per Adversary **A-redfix-2**, which refuted the earlier claim "no code path redeploys +it"; the conclusion held but the mechanism was wrong. Re-derived independently — see below.) + +`reconcile()`'s only `try/except` is `warm_reconcile.py:520-525` (it wraps `deploy_version` + `wait_healthy`), +and its sole caller `main():556` does not catch either. A raise at `:514` or `:536` escapes to `SystemExit` +with live keycloak already undeployed by the preceding `abra.undeploy()`. + +**A code path *does* redeploy it.** `abra.undeploy()` (`abra.py:295-297`) runs only `abra app undeploy -n`; it +does **not** remove the app `.env`. So on the next `reconcile()`, `current_version` (`:286`, reads the `.env`) +is still resolvable while `is_deployed` (`:304`, reads docker services) is `False` → the **fresh-deploy branch** +(`:471-479`) redeploys `current or latest`. **That branch never calls `warmsnap`**, so the foreign meta cannot +block it. + +**What actually prevents self-healing is the absence of a re-trigger.** `warm-keycloak.service` is +`Type=oneshot`, `RemainAfterExit=true`, with **no timer and no `Restart=`**. So the unit does not re-run on its +own: keycloak stays down until the next **reboot / `nixos-rebuild switch`**, which heals it via the fresh-deploy +branch — and then the *following* reconcile sees `current != latest`, takes the upgrade path, and **re-wedges at +`:514`**, because the foreign `snapshot/meta.json` is still sitting in the slot. + +**Consequence for the operator:** the failure presents as an **intermittent flake** (alternating wedge / heal +across reboots), not a permanent outage — and **redeploying keycloak does not fix it.** The remediation is to +**delete the foreign `snapshot/` from the slot** (or fix B-redfix-5); until that meta is gone, every due +upgrade re-wedges. + +**HOW to verify** (all four legs, from a clone at `b5f2b10`): + + git show redfix-m2-harness:runner/harness/abra.py | sed -n '295,297p' # undeploy: no .env removal + git show redfix-m2-harness:runner/warm_reconcile.py | sed -n '471,479p' # fresh-deploy branch, no warmsnap + git grep -n warm-keycloak -- nix/ | grep -iE 'timer|OnCalendar|Restart=' # → EMPTY (no re-trigger) + ssh cc-ci 'systemctl cat warm-keycloak.service | grep -iE "Type=|RemainAfterExit|Restart="' + +**EXPECTED:** `undeploy()` body is a single `_run(["app","undeploy",domain,"-n"], …)` with no `.env` handling; +`:472` is `if not deployed:` returning `deployed-fresh:{target}` at `:479` with no `warmsnap` call in between; +the `git grep` prints **nothing**; `systemctl cat` prints `Type=oneshot` + `RemainAfterExit=true` and **no** +`Restart=` (observed on cc-ci 2026-07-09), and `systemctl list-timers --all | grep -i keycloak` is empty. **Blast radius: keycloak only.** `warm.WARM_DOMAINS == {"keycloak"}` and `keycloak` is the only `SPECS` entry with `stateful: True` (`traefik` is `stateful: False` — version-rollback-only, never snapshots). No other @@ -467,13 +499,23 @@ Reasoning/dead-ends in JOURNAL-redfix.md. Node left clean (see "Node state left repo root. It is not in any commit on any ref, is not gitignored, and is unrelated to every redfix DoD item. No `go.mod` accompanies it. I did not create it and have not committed, modified, or deleted it. -**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, -identical size, identical mtime `2026-07-09 00:01`, i.e. written by something outside git (a git operation -cannot leave the same untracked file in two clones). +**WHERE.** `/srv/cc-ci-orch/cc-ci/main.go` — **one file in one clone.** + +**RETRACTED (2026-07-09, Adversary A-redfix-3):** an earlier version of this entry claimed the file was +"present in **both** clones … i.e. written by something outside git." **That evidence is void.** `/srv/cc-ci` +is a **symlink to `/srv/cc-ci-orch`**, so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are +the **same inode** — one file seen twice, not two clones. The "a git operation cannot leave the same untracked +file in two clones" inference therefore carries **no information** about the file's origin, which remains +**unexplained**. Independently re-derived: `ls -la /srv/` shows `cc-ci -> /srv/cc-ci-orch`, and +`stat -c '%i %h %n'` on both paths prints the same inode `3254604` with `links=1`. **HOW to verify.** `git log --all --oneline -- main.go` → empty (in no commit). -`ls -la /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go` → both 281 bytes, mtime `Jul 9 00:01`. `git status --short` → `?? main.go`. +`ls -la /srv/ | grep cc-ci` → `cc-ci -> /srv/cc-ci-orch` (the symlink that voids the "both clones" claim). +`stat -c '%i %h %n' /srv/cc-ci/cc-ci/main.go /srv/cc-ci-orch/cc-ci/main.go` → same inode, `links=1`. + +**Risk: inert** (Adversary-confirmed). 281-byte hello-world `net/http` listener; no `go.mod`; `go` is **not +installed**; **nothing listening on `:8080`**; in no commit on any ref; unreferenced by any tracked file. **EXPECTED.** Left in place, untracked. Per guardrails I do not delete or commit files I did not create; flagging for the operator rather than acting. It affects no gate, no DoD item, and no verdict.