review(redfix): wake #43 — A-redfix-2 REFUTES "no code path redeploys it"; A-redfix-3 voids the "both clones" evidence
Probed the two operator-facing claims in the Builder's new commits (cbeceea,d5dc547). No inbox, no gate. A-redfix-2 (LOW-MED) — "The wedge does not self-heal ... no code path redeploys it. Recovery is manual." Conclusion right, mechanism wrong. abra.undeploy() does not remove the app .env, so the next reconcile() sees current_version resolvable + is_deployed False and takes the fresh-deploy branch (warm_reconcile.py:471-479), which redeploys and NEVER calls warmsnap -- the guard is unreachable there. What actually blocks self-healing is the lack of a re-trigger: warm-keycloak.service is Type=oneshot with no timer and no Restart=. So it heals on the next reboot / nixos-rebuild switch, then RE-WEDGES on the next due upgrade because the foreign snapshot/meta.json is still in the slot. The failure therefore presents as an intermittent flake, and the correct remediation is removing the foreign snapshot/, not redeploying keycloak. Asked the Builder to correct two sentences; the precondition itself is unchanged and still right. A-redfix-3 (INFO) — "main.go present in both clones => written outside git". Evidence void: /srv/cc-ci is a SYMLINK to /srv/cc-ci-orch, so both paths are the same inode (2049:3254604, links=1). One file, one clone. Risk bounded: 281-byte hello-world net/http listener, no go.mod, go not installed, nothing on :8080, in no commit, unreferenced, inert. Concur with leaving it in place. No VETO. DONE stands, M1+M2 PASS stand, merge targetb5f2b10unchanged. B-redfix-8 unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoraVF2hkEMExHswFX1g4M
This commit is contained in:
@ -517,3 +517,84 @@ reachability from all refs.)
|
||||
a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
|
||||
or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
|
||||
`redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.
|
||||
|
||||
### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
|
||||
|
||||
Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
|
||||
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
|
||||
wrong mechanism points the operator at the wrong remediation.
|
||||
|
||||
**What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:**
|
||||
> "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with
|
||||
> live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it.
|
||||
> Recovery is manual.**"
|
||||
|
||||
**The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that
|
||||
redeploys it, and it runs automatically.
|
||||
|
||||
**Repro (static, from any clone; no node access needed):**
|
||||
|
||||
git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p' # the fresh-deploy branch
|
||||
git show b5f2b10:runner/harness/abra.py | sed -n '295,298p' # undeploy does NOT remove the .env
|
||||
sed -n '38,45p' nix/modules/warm-keycloak.nix # oneshot, RemainAfterExit, no Restart=
|
||||
git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart=" # -> empty
|
||||
|
||||
**Chain, each leg verified:**
|
||||
1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/<domain>.env`.
|
||||
2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that
|
||||
`.env`, while `is_deployed(domain)` is `False` (no docker services for the stack).
|
||||
3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` →
|
||||
`deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:<target>`.
|
||||
**This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign
|
||||
meta cannot block it. Live keycloak comes back up.
|
||||
|
||||
**What actually prevents self-healing** is not the absence of a redeploy path but the absence of a
|
||||
**re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`,
|
||||
with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing
|
||||
re-runs it between activations.
|
||||
|
||||
**Why this matters operationally — the failure is intermittent, not sticky.**
|
||||
- The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit
|
||||
re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will
|
||||
mislead an operator into hand-redeploying something that a routine rebuild already fixed.
|
||||
- But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next
|
||||
time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises
|
||||
→ `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*.
|
||||
- Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**,
|
||||
not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.
|
||||
|
||||
**Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between
|
||||
activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or
|
||||
`nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`)
|
||||
does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation
|
||||
is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still
|
||||
correct**; only the mechanism/remediation sentences are wrong.
|
||||
|
||||
### A-redfix-3 [adversary] — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two
|
||||
|
||||
Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase,
|
||||
leave it alone) is fine; the supporting evidence is an artifact.
|
||||
|
||||
**What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:**
|
||||
> "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical
|
||||
> size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same
|
||||
> untracked file in two clones**)."
|
||||
|
||||
**Repro:**
|
||||
|
||||
ls -la /srv/ # cc-ci -> /srv/cc-ci-orch (symlink)
|
||||
stat -c '%d:%i %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
|
||||
# -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1
|
||||
|
||||
Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the
|
||||
Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones"
|
||||
observation carries **zero** information about the writer, and the identical size/mtime are tautological.
|
||||
|
||||
**Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256
|
||||
bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**;
|
||||
**nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any
|
||||
tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary
|
||||
clone paths**, which is consistent with a single stray write into one working tree.
|
||||
|
||||
**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
|
||||
on not deleting it: neither of us created it.
|
||||
|
||||
@ -1321,3 +1321,41 @@ B-redfix-5 stays deferred/open (now correctly scoped to both call sites). B-redf
|
||||
(OPEN/HIGH/operator-rotation-only; not re-probed, by design — Builder concurs).
|
||||
|
||||
Inbox consumed and deleted. Did not read JOURNAL-redfix.md before forming this verdict.
|
||||
|
||||
---
|
||||
|
||||
## Wake #43 — 2026-07-09T08:31Z — two new Builder claims probed; **one REFUTED** (A-redfix-2), one's evidence void (A-redfix-3). Still no VETO.
|
||||
|
||||
Autonomous tick. No inbox, no gate claimed. Two new Builder commits (`68b51d5`, `42bc4e4`) each assert an
|
||||
operator-facing claim I had never verified. Probed both cold.
|
||||
|
||||
**1. `68b51d5` — "The wedge does not self-heal … no code path redeploys it. Recovery is manual."**
|
||||
The *conclusion* holds; the *mechanism* is wrong → filed **A-redfix-2**.
|
||||
`abra.undeploy()` (`abra.py:295-297`) does not remove the app `.env`, so the next `reconcile()` finds
|
||||
`current_version` resolvable and `is_deployed` False, and takes the **fresh-deploy branch**
|
||||
(`warm_reconcile.py:471-479`) which redeploys `current or latest`. That branch **never calls `warmsnap`**, so
|
||||
the guard is never reached and the foreign meta cannot block it. A code path **does** redeploy it.
|
||||
What actually blocks self-healing is the absence of a **re-trigger**: `warm-keycloak.service` is
|
||||
`Type=oneshot`, `RemainAfterExit=true`, **no timer, no `Restart=`** (`git grep warm-keycloak -- nix/ | grep -iE
|
||||
'timer|OnCalendar|Restart='` → empty). So it heals on the next reboot / `nixos-rebuild switch`, then
|
||||
**re-wedges on the next due upgrade** because the foreign `snapshot/meta.json` is still in the slot.
|
||||
Consequence the Builder's wording hides: **the failure presents as an intermittent flake, and the correct
|
||||
remediation is deleting the foreign `snapshot/`, not redeploying keycloak.** I checked myself before filing
|
||||
(wake-#38 taught me to): confirmed no timer, confirmed `.env` survives undeploy, read the branch verbatim.
|
||||
|
||||
**2. `42bc4e4` — untracked `main.go` "present in both clones ⇒ written by something outside git".**
|
||||
Evidence void → filed **A-redfix-3**. `/srv/cc-ci` is a **symlink to `/srv/cc-ci-orch`**
|
||||
(`ls -la /srv/`), so `/srv/cc-ci/cc-ci/main.go` and `/srv/cc-ci-orch/cc-ci/main.go` are the **same inode**
|
||||
(`2049:3254604`, `links=1`). One file, one clone. The "two clones" premise — and with it the "a git operation
|
||||
cannot leave the same untracked file in two clones" inference — carries no information. Same symlink applies
|
||||
to my own clones (`cc-ci-adv` → inode `3206945` via both paths), which is why I saw the file in neither: it
|
||||
sits in the Builder's tree only.
|
||||
Risk bounded read-only: 281-byte hello-world `net/http` listener, `sha256 bdbc3bf1…`, no `go.mod`, `go` **not
|
||||
installed**, **nothing listening on :8080**, in no commit on any ref, not gitignored, unreferenced by any
|
||||
tracked file. Inert. Concur with leaving it in place — neither of us created it.
|
||||
|
||||
**Verdict: no VETO. `## DONE` stands; M1 + M2 PASS stand; merge target `b5f2b10` unchanged.** Both findings
|
||||
are LOW-MED/INFO and concern operator-facing *descriptions*, not DoD items. The B-redfix-5 **precondition
|
||||
itself is unchanged and still correct** — A-redfix-2 corrects only the mechanism + remediation sentences.
|
||||
B-redfix-8 unchanged (OPEN/HIGH/operator-rotation-only; not re-probed, by design).
|
||||
Did not read JOURNAL-redfix.md before forming these verdicts.
|
||||
|
||||
Reference in New Issue
Block a user