From 10576574c537db68b507bc91a2f9dfdd17ba494f Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Thu, 9 Jul 2026 08:32:49 +0000 Subject: [PATCH] =?UTF-8?q?review(redfix):=20wake=20#43=20=E2=80=94=20A-re?= =?UTF-8?q?dfix-2=20REFUTES=20"no=20code=20path=20redeploys=20it";=20A-red?= =?UTF-8?q?fix-3=20voids=20the=20"both=20clones"=20evidence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Probed the two operator-facing claims in the Builder's new commits (cbeceea, d5dc547). No inbox, no gate. A-redfix-2 (LOW-MED) — "The wedge does not self-heal ... no code path redeploys it. Recovery is manual." Conclusion right, mechanism wrong. abra.undeploy() does not remove the app .env, so the next reconcile() sees current_version resolvable + is_deployed False and takes the fresh-deploy branch (warm_reconcile.py:471-479), which redeploys and NEVER calls warmsnap -- the guard is unreachable there. What actually blocks self-healing is the lack of a re-trigger: warm-keycloak.service is Type=oneshot with no timer and no Restart=. So it heals on the next reboot / nixos-rebuild switch, then RE-WEDGES on the next due upgrade because the foreign snapshot/meta.json is still in the slot. The failure therefore presents as an intermittent flake, and the correct remediation is removing the foreign snapshot/, not redeploying keycloak. Asked the Builder to correct two sentences; the precondition itself is unchanged and still right. A-redfix-3 (INFO) — "main.go present in both clones => written outside git". Evidence void: /srv/cc-ci is a SYMLINK to /srv/cc-ci-orch, so both paths are the same inode (2049:3254604, links=1). One file, one clone. Risk bounded: 281-byte hello-world net/http listener, no go.mod, go not installed, nothing on :8080, in no commit, unreferenced, inert. Concur with leaving it in place. No VETO. DONE stands, M1+M2 PASS stand, merge target b5f2b10 unchanged. B-redfix-8 unchanged. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01SoraVF2hkEMExHswFX1g4M --- machine-docs/BACKLOG-redfix.md | 81 ++++++++++++++++++++++++++++++++++ machine-docs/REVIEW-redfix.md | 38 ++++++++++++++++ 2 files changed, 119 insertions(+) diff --git a/machine-docs/BACKLOG-redfix.md b/machine-docs/BACKLOG-redfix.md index 26f23e7..433292c 100644 --- a/machine-docs/BACKLOG-redfix.md +++ b/machine-docs/BACKLOG-redfix.md @@ -517,3 +517,84 @@ reachability from all refs.) a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha, or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci `redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch. + +### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent + +Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred* +item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the +wrong mechanism points the operator at the wrong remediation. + +**What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:** +> "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with +> live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it. +> Recovery is manual.**" + +**The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that +redeploys it, and it runs automatically. + +**Repro (static, from any clone; no node access needed):** + + git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p' # the fresh-deploy branch + git show b5f2b10:runner/harness/abra.py | sed -n '295,298p' # undeploy does NOT remove the .env + sed -n '38,45p' nix/modules/warm-keycloak.nix # oneshot, RemainAfterExit, no Restart= + git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart=" # -> empty + +**Chain, each leg verified:** +1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/.env`. +2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that + `.env`, while `is_deployed(domain)` is `False` (no docker services for the stack). +3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` → + `deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:`. + **This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign + meta cannot block it. Live keycloak comes back up. + +**What actually prevents self-healing** is not the absence of a redeploy path but the absence of a +**re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`, +with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing +re-runs it between activations. + +**Why this matters operationally — the failure is intermittent, not sticky.** +- The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit + re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will + mislead an operator into hand-redeploying something that a routine rebuild already fixed. +- But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next + time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises + → `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*. +- Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**, + not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade. + +**Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between +activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or +`nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`) +does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation +is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still +correct**; only the mechanism/remediation sentences are wrong. + +### A-redfix-3 [adversary] — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two + +Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase, +leave it alone) is fine; the supporting evidence is an artifact. + +**What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:** +> "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical +> size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same +> untracked file in two clones**)." + +**Repro:** + + ls -la /srv/ # cc-ci -> /srv/cc-ci-orch (symlink) + stat -c '%d:%i %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go + # -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1 + +Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the +Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones" +observation carries **zero** information about the writer, and the identical size/mtime are tautological. + +**Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256 +bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**; +**nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any +tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary +clone paths**, which is consistent with a single stray write into one working tree. + +**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed +on not deleting it: neither of us created it. diff --git a/machine-docs/REVIEW-redfix.md b/machine-docs/REVIEW-redfix.md index 30a3935..ea64a8c 100644 --- a/machine-docs/REVIEW-redfix.md +++ b/machine-docs/REVIEW-redfix.md @@ -1321,3 +1321,41 @@ B-redfix-5 stays deferred/open (now correctly scoped to both call sites). B-redf (OPEN/HIGH/operator-rotation-only; not re-probed, by design — Builder concurs). Inbox consumed and deleted. Did not read JOURNAL-redfix.md before forming this verdict. + +--- + +## Wake #43 — 2026-07-09T08:31Z — two new Builder claims probed; **one REFUTED** (A-redfix-2), one's evidence void (A-redfix-3). Still no VETO. + +Autonomous tick. No inbox, no gate claimed. Two new Builder commits (`68b51d5`, `42bc4e4`) each assert an +operator-facing claim I had never verified. Probed both cold. + +**1. `68b51d5` — "The wedge does not self-heal … no code path redeploys it. Recovery is manual."** +The *conclusion* holds; the *mechanism* is wrong → filed **A-redfix-2**. +`abra.undeploy()` (`abra.py:295-297`) does not remove the app `.env`, so the next `reconcile()` finds +`current_version` resolvable and `is_deployed` False, and takes the **fresh-deploy branch** +(`warm_reconcile.py:471-479`) which redeploys `current or latest`. That branch **never calls `warmsnap`**, so +the guard is never reached and the foreign meta cannot block it. A code path **does** redeploy it. +What actually blocks self-healing is the absence of a **re-trigger**: `warm-keycloak.service` is +`Type=oneshot`, `RemainAfterExit=true`, **no timer, no `Restart=`** (`git grep warm-keycloak -- nix/ | grep -iE +'timer|OnCalendar|Restart='` → empty). So it heals on the next reboot / `nixos-rebuild switch`, then +**re-wedges on the next due upgrade** because the foreign `snapshot/meta.json` is still in the slot. +Consequence the Builder's wording hides: **the failure presents as an intermittent flake, and the correct +remediation is deleting the foreign `snapshot/`, not redeploying keycloak.** I checked myself before filing +(wake-#38 taught me to): confirmed no timer, confirmed `.env` survives undeploy, read the branch verbatim. + +**2. `42bc4e4` — untracked `main.go` "present in both clones ⇒ written by something outside git".** +Evidence void → filed **A-redfix-3**. `/srv/cc-ci` is a **symlink to `/srv/cc-ci-orch`** +(`ls -la /srv/`), so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are the **same inode** +(`2049:3254604`, `links=1`). One file, one clone. The "two clones" premise — and with it the "a git operation +cannot leave the same untracked file in two clones" inference — carries no information. Same symlink applies +to my own clones (`cc-ci-adv` → inode `3206945` via both paths), which is why I saw the file in neither: it +sits in the Builder's tree only. +Risk bounded read-only: 281-byte hello-world `net/http` listener, `sha256 bdbc3bf1…`, no `go.mod`, `go` **not +installed**, **nothing listening on :8080**, in no commit on any ref, not gitignored, unreferenced by any +tracked file. Inert. Concur with leaving it in place — neither of us created it. + +**Verdict: no VETO. `## DONE` stands; M1 + M2 PASS stand; merge target `b5f2b10` unchanged.** Both findings +are LOW-MED/INFO and concern operator-facing *descriptions*, not DoD items. The B-redfix-5 **precondition +itself is unchanged and still correct** — A-redfix-2 corrects only the mechanism + remediation sentences. +B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design). +Did not read JOURNAL-redfix.md before forming these verdicts.