review(redfix): wake #43 — A-redfix-2 REFUTES "no code path redeploys it"; A-redfix-3 voids the "both clones" evidence
Probed the two operator-facing claims in the Builder's new commits (cbeceea,d5dc547). No inbox, no gate. A-redfix-2 (LOW-MED) — "The wedge does not self-heal ... no code path redeploys it. Recovery is manual." Conclusion right, mechanism wrong. abra.undeploy() does not remove the app .env, so the next reconcile() sees current_version resolvable + is_deployed False and takes the fresh-deploy branch (warm_reconcile.py:471-479), which redeploys and NEVER calls warmsnap -- the guard is unreachable there. What actually blocks self-healing is the lack of a re-trigger: warm-keycloak.service is Type=oneshot with no timer and no Restart=. So it heals on the next reboot / nixos-rebuild switch, then RE-WEDGES on the next due upgrade because the foreign snapshot/meta.json is still in the slot. The failure therefore presents as an intermittent flake, and the correct remediation is removing the foreign snapshot/, not redeploying keycloak. Asked the Builder to correct two sentences; the precondition itself is unchanged and still right. A-redfix-3 (INFO) — "main.go present in both clones => written outside git". Evidence void: /srv/cc-ci is a SYMLINK to /srv/cc-ci-orch, so both paths are the same inode (2049:3254604, links=1). One file, one clone. Risk bounded: 281-byte hello-world net/http listener, no go.mod, go not installed, nothing on :8080, in no commit, unreferenced, inert. Concur with leaving it in place. No VETO. DONE stands, M1+M2 PASS stand, merge targetb5f2b10unchanged. B-redfix-8 unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoraVF2hkEMExHswFX1g4M
This commit is contained in:
@ -517,3 +517,84 @@ reachability from all refs.)
|
||||
a sha alone is not durable evidence. Record the *content assertion* (file → expected line) alongside the sha,
|
||||
or push a tag. The other three redfix fixes pinned exactly (`4ca7f418`, `a0f2db88`, `4987ba91`), as did cc-ci
|
||||
`redfix-m2-harness`@`07fc6d4a` — discourse drifted only because a later phase reused its branch.
|
||||
|
||||
### A-redfix-2 [adversary] — STATUS's "no code path redeploys it / recovery is manual" is REFUTED: the B-redfix-5 wedge **does** self-heal on the next unit activation, which makes it look intermittent
|
||||
|
||||
Severity **LOW-MED**, **no VETO, does not reopen the phase** (operator-facing description of a *deferred*
|
||||
item, not a DoD item). Merge target `b5f2b10` unchanged. Filed because the stated mechanism is wrong and the
|
||||
wrong mechanism points the operator at the wrong remediation.
|
||||
|
||||
**What STATUS-redfix.md (`68b51d5`, lines ~91-94) claims:**
|
||||
> "**The wedge does not self-heal.** … A raise at `:514` or `:536` therefore escapes to `SystemExit` with
|
||||
> live keycloak already undeployed by the preceding `abra.undeploy()` — **no code path redeploys it.
|
||||
> Recovery is manual.**"
|
||||
|
||||
**The conclusion "does not self-heal" is right; the reason given is wrong.** There *is* a code path that
|
||||
redeploys it, and it runs automatically.
|
||||
|
||||
**Repro (static, from any clone; no node access needed):**
|
||||
|
||||
git show b5f2b10:runner/warm_reconcile.py | sed -n '468,479p' # the fresh-deploy branch
|
||||
git show b5f2b10:runner/harness/abra.py | sed -n '295,298p' # undeploy does NOT remove the .env
|
||||
sed -n '38,45p' nix/modules/warm-keycloak.nix # oneshot, RemainAfterExit, no Restart=
|
||||
git grep -n "warm-keycloak" -- nix/ | grep -iE "timer|OnCalendar|Restart=" # -> empty
|
||||
|
||||
**Chain, each leg verified:**
|
||||
1. `abra.undeploy()` runs only `abra app undeploy` — it does **not** remove `~/.abra/servers/default/<domain>.env`.
|
||||
2. So on the next `reconcile("keycloak")`: `current_version(domain)` still resolves from `TYPE=` in that
|
||||
`.env`, while `is_deployed(domain)` is `False` (no docker services for the stack).
|
||||
3. `warm_reconcile.py:471-479` therefore takes the **fresh-deploy branch**: `target = current or latest` →
|
||||
`deploy_version(...)` → `wait_healthy` → `write_last_good` → `return deployed-fresh:<target>`.
|
||||
**This branch never calls `warmsnap`**, so `_assert_slot_not_foreign` is never reached and the foreign
|
||||
meta cannot block it. Live keycloak comes back up.
|
||||
|
||||
**What actually prevents self-healing** is not the absence of a redeploy path but the absence of a
|
||||
**re-trigger**: `warm-keycloak.service` is `Type=oneshot`, `RemainAfterExit=true`, `wantedBy=multi-user.target`,
|
||||
with **no timer and no `Restart=`** (the module comment says it "converges every activation/boot"). Nothing
|
||||
re-runs it between activations.
|
||||
|
||||
**Why this matters operationally — the failure is intermittent, not sticky.**
|
||||
- The wedge persists only until the next **reboot or `nixos-rebuild switch`**, at which point the unit
|
||||
re-activates, hits the fresh-deploy branch and silently restores live keycloak. "Recovery is manual" will
|
||||
mislead an operator into hand-redeploying something that a routine rebuild already fixed.
|
||||
- But the foreign `snapshot/meta.json` is **still in the slot**. So the box heals, then **re-wedges the next
|
||||
time an upgrade is actually due** (`current != latest` → upgrade path → `:514` `snapshot()` → guard raises
|
||||
→ `abra.undeploy()` at `:512` has already run). Heal → re-wedge → heal: it will read as a *flake*.
|
||||
- Therefore the correct remediation is **remove the foreign `snapshot/` from `/var/lib/ci-warm/keycloak/`**,
|
||||
not "redeploy keycloak". Redeploying treats the symptom and the wedge returns on the next due upgrade.
|
||||
|
||||
**Ask of the Builder:** correct the two sentences in STATUS. Suggested: *"The wedge does not self-heal between
|
||||
activations — `warm-keycloak.service` is a `oneshot` with no timer and no `Restart=`. The next reboot or
|
||||
`nixos-rebuild switch` re-activates it and the fresh-deploy branch (`:471-479`, which never calls `warmsnap`)
|
||||
does restore live keycloak — but the foreign meta remains, so it re-wedges on the next due upgrade. Remediation
|
||||
is to remove the foreign `snapshot/`, not to redeploy."* The **precondition itself is unchanged and still
|
||||
correct**; only the mechanism/remediation sentences are wrong.
|
||||
|
||||
### A-redfix-3 [adversary] — STATUS's `main.go` "present in both clones ⇒ written outside git" evidence is void: `/srv/cc-ci` is a **symlink** to `/srv/cc-ci-orch`, so there is one file, not two
|
||||
|
||||
Severity **INFO**. No VETO, no gate impact, no DoD impact. The *conclusion* (untracked, not from this phase,
|
||||
leave it alone) is fine; the supporting evidence is an artifact.
|
||||
|
||||
**What STATUS-redfix.md (`42bc4e4`, lines ~470-472) claims:**
|
||||
> "`/srv/cc-ci-orch/cc-ci/main.go` **and** `/srv/cc-ci/cc-ci/main.go` — present in **both** clones, identical
|
||||
> size, identical mtime … i.e. written by something outside git (**a git operation cannot leave the same
|
||||
> untracked file in two clones**)."
|
||||
|
||||
**Repro:**
|
||||
|
||||
ls -la /srv/ # cc-ci -> /srv/cc-ci-orch (symlink)
|
||||
stat -c '%d:%i %n' /srv/cc-ci-orch/cc-ci/main.go /srv/cc-ci/cc-ci/main.go
|
||||
# -> 2049:3254604 for BOTH; stat -c 'links=%h' -> links=1
|
||||
|
||||
Same device, same inode, link count 1: **one file, one clone, seen through a symlink.** The same holds for the
|
||||
Adversary clones (`/srv/cc-ci-orch/cc-ci-adv` and `/srv/cc-ci/cc-ci-adv` are inode `3206945`). The "two clones"
|
||||
observation carries **zero** information about the writer, and the identical size/mtime are tautological.
|
||||
|
||||
**Risk bounded (read-only):** contents are a 281-byte hello-world `net/http` listener; `sha256
|
||||
bdbc3bf167cd20f30c00880005f4f994f17f3722660973c9c65c7bf33e81ffaf`; no `go.mod`; **`go` is not installed**;
|
||||
**nothing is listening on `:8080`**; `git log --all -- main.go` is empty; not gitignored; unreferenced by any
|
||||
tracked file. It cannot execute. Present in the **Builder's** clone only — **absent from both Adversary
|
||||
clone paths**, which is consistent with a single stray write into one working tree.
|
||||
|
||||
**Ask of the Builder:** drop the "both clones ⇒ outside git" inference (keep the file, keep the flag). Agreed
|
||||
on not deleting it: neither of us created it.
|
||||
|
||||
Reference in New Issue
Block a user