cc-ci/machine-docs/REVIEW-dstamp.md

# REVIEW-dstamp.md — Adversary verdicts for phase `dstamp`

Phase: investigate & solve the discourse abra-stamp drift (upgrade-HC1 stamps the
prev-base tag commit instead of the PR-head version, harness-neutral, since ~06-10).
SSOT: `/srv/cc-ci/cc-ci-plan/plan-phase-dstamp-discourse-drift.md`. Gates M1, M2.

Verdict log is append-only. `review(...)`-prefixed commits carry verdicts (load-bearing
watchdog signal). Findings filed under `## Adversary findings` in BACKLOG-dstamp.md.

---

## Prep notes (NOT a verdict — no gate claimed yet) @2026-06-11T15:5x

Recon done cold before any Builder claim, to make M1/M2 verification fast and independent.
Anti-anchoring: formed only from the plan (SSOT), the harness code, and direct host evidence
— no dstamp JOURNAL exists yet; none read.

**Stamp mechanism (from code):** HC1's "stamp" = the `coop-cloud.<stack>.chaos-version`
docker service label abra writes on a `--chaos` deploy = the deployed recipe git commit
(`runner/harness/lifecycle.py:468 deployed_identity`, `runner/harness/generic.py:146
assert_upgraded`). Upgrade flow (`generic.py:226 perform_upgrade`): deploy prev-published
base → `recipe_checkout_ref(recipe, head_ref)` (git checkout -f head) → `chaos_redeploy`
(`abra app deploy --chaos`). HC1 asserts `chaos_commit == head_ref` (after stripping the
`+U` untracked-overlay marker). PASS requires the chaos-version to equal the PR head.

**Cold observable facts (from `/var/lib/cc-ci-runs/m2p-discourse/abra/recipes/discourse`
snapshot + live `~/.abra/recipes/discourse` on cc-ci, 2026-06-11):**
- Recipe HEAD `7ae7b0f` = "chore: upgrade to 0.9.0+3.5.0"; `git describe --tags` =
  `0.7.0+3.3.1-9-g7ae7b0f` → HEAD is **9 commits past the newest annotated tag**
  `0.7.0+3.3.1` (commit `eb96de9`). No `0.8.x`/`0.9.x` tag exists.
- The drift symptom (per plan): chaos-version stamped `eb96de94+U` = the **prev-base tag
  commit** (= the upgrade base `0.7.0+3.3.1`), NOT the PR-head `7ae7b0f`.
- abra is **nix-pinned**: `abra version 0.13.0-beta-06a57de`, store path under
  `/run/current-system` → binary drift requires a flake.lock/nixos-generation bump between
  06-05 and 06-10 (verify against generations, don't assume).

**Open question I'll independently re-derive when M1 is claimed:** why the `--chaos`
redeploy after checkout-to-HEAD stamps the BASE commit (eb96de9), not HEAD (7ae7b0f).
Candidates to test cold: (a) re-checkout to head silently reverted (abra fetch/reset during
deploy); (b) abra chaos resolves the version from the app's recorded `.env` RECIPE/version
(= the base) rather than the working-tree HEAD; (c) the "env drift" since 06-10 = recipe/
mirror git state moved (unreleased commits pushed past last tag) or a tag re-pointed.

**Guardrail teeth I will enforce at M2:** HC1 must still FAIL on a genuinely wrong stamp
(synthesize a wrong-version deploy and show RED). Any "fix" that derives EXPECTED from
"what makes the test pass" rather than abra's documented behavior = automatic FAIL.

Status: idle, awaiting Builder to seed STATUS-dstamp.md and claim M1. Watchdog will ping
on the `claim(...)` commit.

---

## Independent probe findings @2026-06-11T17:3x (NOT a verdict — no M1 claim yet)

Anti-anchoring preserved: JOURNAL-dstamp NOT read. Root cause derived independently from
harness code, per-run artifacts (repro1/repro2 console logs), and direct docker service
inspect on cc-ci. Independently arrived at the same attribution as the Builder.

**Causal chain derived from code + direct evidence:**

1. `provide_ccci_overlay` (rcust-era addition) copies `compose.ccci.yml` into the per-run
   recipe dir as an UNTRACKED file. Absent in run 184 (2026-06-05, which used the old
   `install_steps.sh` path writing to canonical `~/.abra`) — consistent with run 184 having
   no `+U` suffix and passing. The `+U` itself is stripped by HC1's `chaos_commit.split("+",1)[0]`
   and is NOT the cause of drift.

2. abra reads `git HEAD = 7ae7b0f` and computes `chaos-version = 7ae7b0f7+U` CORRECTLY.
   Confirmed via three bail-at-secrets manual repros + repro2 debug line
   `taking chaos version: 7ae7b0f7+U`. abra and the per-run git checkout are EXONERATED.

3. `chaos_redeploy` passes `-c` (no_converge_checks) → `docker stack deploy` returns
   immediately; Swarm rolling update runs asynchronously.

4. Discourse `compose.yml` (BOTH base `eb96de94` AND PR-head `7ae7b0f`) sets
   `deploy.update_config: { failure_action: rollback, order: start-first, monitor: 5s }`
   on the `app` service. Confirmed by direct `docker service inspect disc-ae10f0_..._app`.

5. With `order: start-first`, OLD + NEW task co-reside (~2× memory). Discourse's
   Rails/Sidekiq precompile is memory-heavy; under the heavier host load since ~06-10
   (warm keycloak and other rcust-phase stacks), the NEW task intermittently fails swarm's
   5s update monitor → `failure_action: rollback` fires → Swarm REVERTS the app service
   spec to PreviousSpec (base deploy, `chaos-version=eb96de94+U`).

6. `services_converged` blind spot: after rollback `UpdateStatus.State = "rollback_completed"`,
   NOT in the blocking set `("updating", "rollback_started")` → returns True as if converged.
   Under start-first the OLD task kept serving → `wait_healthy` also passes on the
   rolled-back spec.

7. `deployed_identity` reads `.Spec.Labels` → rolled-back spec → `chaos-version=eb96de94+U`.
   HC1 asserts head_ref `7ae7b0f76efb` ≠ `eb96de94` → FAIL with misleading "re-checkout failed".

**Key disproving evidence (independent route):** repro1 was isolated (no concurrent discourse
run, domain `disc-ae10f0` used for the first time) and STILL showed the drift. This refuted
the pure-concurrency hypothesis BEFORE reading the Builder's evidence or JOURNAL.

**Intermittency explained (run 184 ✓ solo 06-05; clustered/repro1/repro4 ✗; repro2 ✓):**
Whether the new start-first task survives the 5s monitor depends on momentary memory pressure.
Run 184: solo + lighter host load + pre-rcust overlay path → new task survived. repro2: warm
volumes/containers from repro1 → faster Rails precompile → task survived. The "since ~06-10
on every run" pattern = heavier baseline load from warm rcust-phase stacks after run 184.

**Fix analysis (Builder commit 0cc31a5 — read before JOURNAL):**

*Part 1 — overlay `order: stop-first`*: Old task stops before new starts → new boots with full
host memory → no OOM under the 5s monitor → no spurious rollback. `failure_action: rollback`
intentionally preserved so a genuinely broken head still rolls back and is caught.
ASSESSMENT: **CORRECT AND SUFFICIENT** for eliminating the spurious-rollback trigger.

*Part 2 — `lifecycle.assert_upgrade_converged`*: Called in `perform_upgrade` immediately after
`chaos_redeploy`, before `wait_healthy`. Polls `docker service inspect
--format '{{if .UpdateStatus}}{{.UpdateStatus.State}}{{else}}none{{end}}'` until terminal.
Returns on `""|"none"|"completed"`; raises on `"rollback_completed"|"rollback_paused"|"paused"`;
polls on `"updating"|"rollback_started"`; times out at `meta.DEPLOY_TIMEOUT`.
ASSESSMENT: **CORRECT** — closes the wait_healthy-masking blind spot. Makes a swarm rollback
an HONEST upgrade failure ("head did not stay healthy") rather than a misreported stamp mismatch.
HC1 commit-match logic is unchanged; this only makes the rollback visible before HC1 runs.

**One concern flagged (not a blocker — defense-in-depth covers it):**
`assert_upgrade_converged` has a theoretical race window: on the very first poll, Docker may
not yet have transitioned from a prior `"completed"` state to `"updating"` (tiny gap between
`docker stack deploy` returning and the Swarm manager scheduling the roll). If the race fires,
the function returns OK on `"none"`, then the rollback happens silently afterward.
Mitigation: with `stop-first` (fix part 1), a post-assert-converged rollback leaves NO serving
task during the rollback → `wait_healthy` also FAILS → the test result is still FAIL, just
with a less specific error ("wait_healthy timeout" rather than "swarm rolled back"). HC1 is
NOT weakened even if the race fires. No action required unless a recipe uses `start-first`
where a post-race rollback could masquerade as a clean upgrade.

**UPDATE — race concern CLOSED by Builder (commit e9c26c7 `harden(dstamp)`):**
Builder addressed the race with a 2-phase protocol:
- **Pre-redeploy**: `update_status_started(domain)` snapshots `UpdateStatus.StartedAt`.
- **Phase 1**: polls until `StartedAt` advances past the snapshot (new update scheduled) OR
  state is `"updating"/"rollback_started"`. 30s grace: if no new update appears → no-op
  redeploy, nothing to converge.
- **Phase 2**: now that the NEW update is confirmed in flight, waits for terminal state
  (same logic as before, but with confidence it's the right update).
Assessment: **CORRECT AND COMPLETE**. Phase 1 deterministically distinguishes the new update
from stale base-deploy terminal state. No new failure modes introduced. The grace period (30s)
is generous relative to Docker's near-immediate scheduling. Race concern fully closed.

**Status:** no `claim(dstamp)` commit yet. Awaiting M1 claim to issue formal verdict.

---

## M1: PASS @2026-06-11T17:36Z

Cold verification from `/srv/cc-ci/cc-ci-adv`. JOURNAL-dstamp not read before verdict (anti-anchoring).

**Check 1 — Recipe policy at 7ae7b0f76efb:** PASS
`cd ~/.abra/recipes/discourse && git checkout -q 7ae7b0f76efb && grep -nA3 update_config compose.yml`
→ `failure_action: rollback`, `order: start-first` confirmed present at lines 33-35. Direct evidence the
discourse app service is configured to rollback+start-first at the PR-head.

**Check 2 — abra CONSTANT (no binary change 06-05→06-10):** PASS
`for g in $(ls -d /nix/var/nix/profiles/system-*-link); do ...readlink -f $g/sw/bin/abra; done`
→ Gens 2-11 all `/nix/store/bf6azhpi8bi5491n8i4bhjm1z7fva7pb-abra-0.13.0-beta/bin/abra`.
Gen1 differs (pre-bootstrap), gens 4-11 (2026-06-01 onward) identical. abra version change as
cause of drift definitively ruled out by direct evidence.

**Check 3 — Direct rollback evidence (repro4):** PASS
`grep -E 'DSTAMP|UpdateStatus|PreviousSpec|chaos-version' /var/lib/cc-ci-runs/dstamp-repro4.console.log`
→ Line immediately after chaos_redeploy:
- `UpdateStatus.State="updating"` (in flight)
- `Spec.Labels chaos-version="7ae7b0f7+U"` (abra correctly applied HEAD)
- `PreviousSpec.Labels chaos-version="eb96de94+U"` (the base, what swarm reverts to)
→ HC1 line: `chaos-version=eb96de94+U` (AFTER rollback completed) → mismatch → FAIL

Causal chain proven in a single artifact: abra stamped correctly, swarm rolled back, label reverted.
Mechanism confirmed: start-first co-residency → OOM under monitor → failure_action:rollback → PreviousSpec.

**Check 4 — Fix present:** PASS
- `runner/harness/lifecycle.py`: `update_status_started` (line 511) + `assert_upgrade_converged` (line 526).
  Phase-1 polls until StartedAt advances past prev_started (or in-flight state seen) → closes race.
  Phase-2 terminal: `completed`=OK; `rollback_completed`/`rollback_paused`/`paused`=FAIL with honest message.
- `runner/harness/generic.py:268-278`: `prev_started = update_status_started(domain)` called BEFORE
  `chaos_redeploy`, then `assert_upgrade_converged(domain, timeout=DEPLOY_TIMEOUT, prev_started=prev_started)`
  called immediately after — BEFORE `wait_healthy`. Correct call order.
- `tests/discourse/compose.ccci.yml:54-55`: `deploy.update_config.order: stop-first` with full WHY
  comment citing direct evidence (dstamp-repro1/4) and stating `failure_action: rollback` is LEFT INTACT.
  Both commits 0cc31a5 + e9c26c7 verified present (git log --oneline).

**Check 5 — Fix works (dstamp-fix1 and dstamp-fix2):** PASS
- `dstamp-fix1`: `upgrade-converged: disc-ae10f0_ci_commoninternet_net_app swarm UpdateStatus=completed`
  + `upgrade→PR-head: head_ref=7ae7b0f7 chaos-version=7ae7b0f7+U version=0.7.0+3.3.1→0.9.0+3.5.0`
  + `test_upgrade_reconverges PASSED`. Level=2 (install+upgrade only, backup/functional not in STAGES).
- `dstamp-fix2`: same params, same domain, same result — second reliability run confirms.
  Both runs: chaos-version=7ae7b0f7+U (head), NOT eb96de94+U (base). Fix is deterministic.

**Check 6 — Blast-radius:** PASS
- n8n: runs 162 (level=4, upgrade=pass) and 47 (level=4, upgrade=pass). Run 162 dated post-06-10
  (when discourse was failing) → n8n not affected despite same rollback+start-first policy.
- keycloak: runs 155 (level=4, upgrade=pass) and 187 (level=4, upgrade=pass). Same conclusion.
- `assert_upgrade_converged` now provides a general harness backstop for all rollback-policy recipes.
  No overlay change needed for keycloak/n8n (lighter apps, no OOM symptom in evidence).
- drone/traefik: infra, no recipe-CI upgrade tier. No action needed.

**HC1 teeth preserved (code inspection):** `generic.py:174-175` — `assert_upgraded` logic is UNCHANGED:
`chaos_commit = chaos.split("+",1)[0]`; assertion `head_ref.startswith(chaos_commit) or
chaos_commit.startswith(head_ref)`. `assert_upgrade_converged` runs BEFORE `assert_upgraded`; if a
rollback occurs it raises FIRST with the honest "head did not stay healthy" message; if no rollback occurs,
HC1 commit-match assertion still runs unmodified. A deliberately wrong stamp (e.g. deploying eb96de94
as the chaos version) would still fail HC1 exactly as before. M2 will demonstrate this with a live negative test.

**One nuance (not a blocker):** The "06-05→06-10 change" being specifically "heavier resident load from
rcust-phase stacks" is circumstantially supported by the timeline, but repro1 (isolated, no concurrent apps)
also showed drift — the mechanism fires under general memory pressure during discourse's precompile, not
only when other apps are warm. The exact delta between run 184 (06-05, passed) and subsequent runs is
intermittency of memory pressure, proven by repro2 (warm volumes → faster precompile → task survived) vs
repro4 (fresh boot → slower precompile → task failed). The ROOT CAUSE mechanism is proven by direct
evidence; the specific "what changed between 06-05 and 06-10" reduces to: heavier/more-variable memory
pressure, the mechanism was always latent. This doesn't weaken M1 — the fix eliminates the exposure.

**Verdict: M1 PASS.** Root cause attributed by direct evidence; minimal reproducible demonstration
confirmed; fix (stop-first overlay + assert_upgrade_converged) implemented and working; HC1 unweakened;
blast-radius sweep complete. Builder cleared to proceed to M2.

---

## M2: PASS @2026-06-11T17:58Z

Cold verification from `/srv/cc-ci/cc-ci-adv`. JOURNAL-dstamp not read before verdict (anti-anchoring).

**Check 1 — Build 450 results (level, tiers, flags):** PASS
`cat /var/lib/cc-ci-runs/450/results.json`:
- `"level": 5` ✓
- `"recipe": "discourse"`, `"ref": "7ae7b0f76efb"`, `"pr": "2"` ✓
- All tiers: `"install": "pass"`, `"upgrade": "pass"`, `"backup": "pass"`, `"restore": "pass"`, `"custom": "pass"` ✓
- All rungs: `"install": "pass"`, `"upgrade": "pass"`, `"backup_restore": "pass"`, `"functional": "pass"`, `"lint": "pass"` ✓
- `"clean_teardown": true`, `"no_secret_leak": true` ✓
- Timestamp: `"finished": 1781199631.4...` (2026-06-11 ~17:40 UTC) ✓
- `screenshot.png` present (discourse functional screenshot)

**Check 2 — JUnit XML: test_upgrade_reconverges PASS (HC1 satisfied):** PASS
`grep -c '<failure\|<error' upgrade__generic__test_upgrade.xml` → 0
Full XML: `<testcase classname="tests._generic.test_upgrade" name="test_upgrade_reconverges" time="0.260"/>`
(no `<failure>` child). `test_upgrade_reconverges` directly calls `generic.assert_upgraded(live_app, meta)`.
`assert_upgraded` at `generic.py:174-175` does the HC1 commit-match: `chaos_commit == head_ref`.
Test PASSED → `chaos_commit = 7ae7b0f7` matched `head_ref = 7ae7b0f7` ✓

**Check 3 — PR comment 14347 (!testme path):** PASS
Comment 14346 body = `!testme` (the trigger).
Comment 14347 body (bot response):
`<!-- cc-ci:testme -->\n🌻 **cc-ci** — \`discourse\` @ \`7ae7b0f7\` ✅ **passed**\n[...links to run 450 summary.png + badge + drone build 450...]`
Confirmed via Gitea API. Run directory `/var/lib/cc-ci-runs/450/` exists with full contents.
!testme → bridge ack → drone build 450 → run 450 results → PR comment ✅ passed. Path verified.

**Check 4 — DEFERRED entry closed:** PASS
`machine-docs/DEFERRED.md` lines 346-366: ✅ RESOLVED @2026-06-11 (phase dstamp, Builder) with:
- Root cause narrative (rollback mechanism)
- Direct evidence pointer (dstamp-repro4.console.log)
- Fix commits (0cc31a5 + e9c26c7)
- Real CI proof (drone build #450, LEVEL 5)
- Blast-radius note (only discourse; harness guard covers all rollback-policy recipes)
- Cross-references (STATUS/JOURNAL/REVIEW-dstamp)

**Check 5 — HC1 teeth (wrong stamp still FAILs):** PASS
*Negative control (pre-fix, existing run):* `m2p-discourse/results.json` shows HC1 caught wrong stamp:
`AssertionError: upgrade deployed chaos commit 'eb96de94+U', not the intended PR-head '7ae7b0f76efb'
— the re-checkout to the code under test failed, so the upgrade is not exercising the PR's changes (HC1)`
This is HC1 raising on `eb96de94 ≠ 7ae7b0f7`. HC1 commit-match assertion WORKS.

*Code unchanged (from M1):* `generic.py:174-175` commit-match assertion unmodified. The fix adds
`assert_upgrade_converged` BEFORE `assert_upgraded` — it catches rollback EARLIER with an honest message
but does NOT bypass HC1. If a non-rollback wrong stamp were deployed (e.g. abra bug stamping wrong commit),
`assert_upgrade_converged` would see `completed` and pass, then HC1 would FAIL on the commit mismatch.

*Post-fix rollback path:* `assert_upgrade_converged` raises `RuntimeError` on `rollback_completed` →
upgrade FAILS with honest "head did not stay healthy" → HC1 doesn't even run but test is RED.
Both paths (rollback → caught by assert_upgrade_converged; wrong stamp without rollback → caught by HC1)
still FAIL. The pre-fix negative controls (m2p-discourse, repro1, repro4) demonstrate the wrong-stamp
path is always caught; the fix only changes HOW it's reported and at which point.

**Blast-radius (confirmed at M1, still valid):** Only discourse affected. keycloak/n8n PASS L4
in 06-10/06-11 era. General `assert_upgrade_converged` guard now covers all rollback-policy recipes.

**Phase DoD summary:**
- ✅ Drift mechanism attributed with reproducible evidence (repro4 direct evidence)
- ✅ Fixed at the true root (stop-first overlay + assert_upgrade_converged)
- ✅ Discourse back at real level in real CI via drone !testme (build 450, LEVEL 5)
- ✅ No other recipe silently affected (blast-radius sweep, keycloak/n8n PASS)
- ✅ HC1 unweakened and adversarially re-proven (m2p-discourse negative control + code inspection)
- ✅ DEFERRED closed with pointers

**Verdict: M2 PASS. All phase dstamp DoD items satisfied. Builder cleared for ## DONE.**