cc-ci/machine-docs/JOURNAL-dstamp.md

# JOURNAL — phase `dstamp` (Builder, reasoning/private)

## 2026-06-11 — Bootstrap + investigation

Read the phase plan, plan.md §6.1/§7/§9, the Adversary's REVIEW-dstamp prep notes, and the
stamp-relevant harness code (`abra.py`, `lifecycle.py:deployed_identity/recipe_checkout_ref/
chaos_redeploy/prepull_images`, `generic.py:perform_upgrade/assert_upgraded`, run_recipe_ci
upgrade op + fetch_recipe).

### Mechanism (from abra source @06a57de = the pinned binary)
chaos-version label is set in `cli/app/deploy.go`: for a `-C` deploy, `getDeployVersion` (l.365)
returns `Recipe.ChaosVersion()` (l.367-373) and `SetChaosVersionLabel(compose, stack, toDeployVersion)`
(l.168). `ChaosVersion` (`pkg/recipe/git.go:300`) = `formatter.SmallSHA(Head().String())` + `+U`
if dirty. `Head` (l.483) = go-git `repo.Head()`. Crucially, `app.Recipe.Ensure(ctx)` (deploy.go:86)
calls into git.go:38 which **early-returns on `ctx.Chaos`** (l.41-43) — so a chaos deploy does NOT
re-checkout the .env version. `GetEnsureContext` (cli/internal/ensure.go) wires `EnsureContext{Chaos,
Offline, IgnoreEnvVersion=DeployLatest}` from the CLI flags. So `-C` ⇒ Ensure no-op ⇒ chaos version
= whatever git HEAD the harness left checked out.

### The contradiction that drove the dig
The m2p failure message is `chaos commit 'eb96de94+U', not the intended PR-head '7ae7b0f76efb'`.
`eb96de9` = tag `0.7.0+3.3.1` (the upgrade base); `7ae7b0f` = PR head (9 commits past that tag,
and there is NO 0.8/0.9 tag despite HEAD's "upgrade to 0.9.0+3.5.0" message). The harness
`perform_upgrade` does `recipe_checkout_ref(head_ref=7ae7b0f)` then `chaos_redeploy`, with only
`env_set` + `prepull_images` (pure docker compose, no git) in between — and the run's recipe
**snapshot HEAD = 7ae7b0f**. So at deploy time HEAD *should* be 7ae7b0f ⇒ stamp 7ae7b0f. Yet it
stamped eb96de9. abra's source says chaos = Head(); so for eb96de9 to be stamped, HEAD had to be
eb96de9 at the chaos deploy — which the isolated flow never produces.

### Reproductions (all on cc-ci, scratch ABRA_DIR, deploys bail at `secret not generated`
### which is deploy.go:140, AFTER the chaos version is computed+logged at deploy.go:372)
1. cp -a canonical recipe, checkout head→base(tag)→head, `abra app deploy -C` → `taking chaos
   version: 7ae7b0f7`. HEAD stays 7ae7b0f. NO drift.
2. real non-chaos base deploy (exercises go-git `EnsureVersion` which checks out tag via
   `Branch: refs/tags/0.7.0+3.3.1`, leaving HEAD=eb96de9), then CLI `git checkout -f head`, then
   `-C` deploy → `taking chaos version: 7ae7b0f7`. NO drift.
3. mirror-faithful: `git clone <recipe-maintainers/discourse>` + `git checkout 7ae7b0f` +
   `git fetch <coop-cloud/discourse> refs/tags/*:refs/tags/*` (exact `fetch_recipe`), then base
   deploy → re-checkout head → `-C` deploy → `taking chaos version: 7ae7b0f7`. NO drift.

Conclusion: the isolated git/abra version-resolution path is **correct** in the current host
state. The drift is not in that path.

### Timeline / differentiator
- abra binary: constant since 2026-06-01 (system-4). Not abra.
- Same ref 7ae7b0f: run 184 (06-05 02:17, **solo**) was L4 upgrade-PASS. The drift runs
  (m2b 06-10 20:54, m2p 06-11 00:44, ab 06-11 00:48) are **clustered** (m2p & ab 4 min apart →
  overlapping for a multi-tier discourse run that takes ≫4 min).
- `app_domain` hashes (recipe|pr|ref) ⇒ all three drift runs, same ref, **collide on one swarm
  stack**. The upgrade `chaos_redeploy` does NOT take `deploy_app`'s app-domain flock, so two
  concurrent runs can interleave deploys on the shared stack and the `<stack>_app` service label
  read by `deployed_identity` reflects whichever deploy last wrote it.

**Leading hypothesis:** the "harness-neutral env drift" is actually a **concurrency artifact** of
the rcust-phase M2 A/B discourse experiments running near-simultaneously on the shared stack — not
an abra/recipe/environment regression. Run 184 solo = green; clustered 06-11 = drift; isolated
re-reproduction now = green. Testing with one clean isolated real run (install,upgrade) before
committing to this attribution — direct evidence required by the plan, not inference alone.

Open: must still explain *exactly* how a concurrent peer produces an `eb96de9+U` (dirty CHAOS)
label on the shared stack — a base deploy is pinned/non-chaos (no chaos label), so the +U chaos
label must come from some chaos deploy with HEAD=eb96de9. The isolated real run + (if needed) a
deliberate 2-run concurrency repro will nail the mechanism. Will NOT claim M1 on inference.

## 2026-06-11 (cont.) — REAL runs: concurrency REFUTED, true root cause = swarm rollback

Three real install+upgrade runs of discourse @7ae7b0f (CCCI_RUN_ID=dstamp-repro{1,2,3}), each
SOLO/isolated (no concurrent discourse run):

- **base deploy is CHAOS** (not pinned): `compose.ccci.yml` overlay is present ⇒
  `deploy_app` takes the `has_ccci_overlay` auto-chaos branch (`lifecycle.py:291-298`). So the
  base stamps `chaos-version = eb96de9+U` on the shared stack. (My earlier bail-at-secrets repros
  used a non-chaos/manual base → that's why they didn't expose it.)
- **repro1 (unpatched): upgrade FAIL** — `chaos commit 'eb96de94+U', not 7ae7b0f76efb`. The
  per-run tree reflog + snapshot prove HEAD = **7ae7b0f** at the upgrade deploy (last checkout
  16:39:03, no checkout-back), yet the deployed `.Spec` chaos label was eb96de9+U.
- **repro2 (instrumented: abra deploy `--debug` + a HEAD-print subprocess before the redeploy):
  upgrade PASS** — `[DSTAMP] taking chaos version: 7ae7b0f7+U`, HEAD=7ae7b0f,
  `deployed_identity = {version 0.9.0+3.5.0, image bitnamilegacy/discourse:3.3.1, chaos 7ae7b0f7+U}`.

So the SAME solo config is **intermittent** (184✓ 06-05, m2b/m2p/ab✗ 06-10/11, repro1✗, repro2✓);
flipping with a tiny timing change ⇒ **NOT a concurrency artifact, NOT abra version-resolution**
(abra computes 7ae7b0f7 correctly — proven by repro2's debug line AND all 3 bail-at-secrets repros).

**TRUE ROOT CAUSE (recipe deploy policy + heavy/flaky new task):** discourse `compose.yml` app
service sets `deploy.update_config: { failure_action: rollback, order: start-first }` with a
`healthcheck.start_period: 20m`. The upgrade chaos deploy applies the head spec
(`chaos-version=7ae7b0f7+U`) start-first (old + new task co-resident = ~2× memory for a
precompile-heavy Rails app). When the NEW task intermittently fails swarm's update monitor,
swarm executes **failure_action: rollback ⇒ reverts the app service to its PreviousSpec (the
base: `chaos-version=eb96de9+U`)**. Under `start-first` the OLD task keeps serving, so the
harness `wait_healthy` still passes — but `deployed_identity` reads `.Spec.Labels` of the
ROLLED-BACK spec and sees the base commit. The "since ~06-10 on every run" pattern = the
rcust-phase runs happened under heavier host load (warm keycloak etc.), so the new task reliably
failed the monitor ⇒ rollback every time; the solo 06-05 run (184) didn't roll back. Harness- and
abra-neutral, exactly as observed.

repro3 (UpdateStatus + PreviousSpec capture, NO --debug to preserve failing timing) running to
get the swarm rollback in the act (expect `UpdateStatus.State = rollback_*`, `PreviousSpec.Labels`
chaos=eb96de9+U == the read `.Spec.Labels` after revert). That is the direct-evidence smoking gun.

### DIRECT EVIDENCE — captured (repro4, solo/isolated, upgrade FAIL)
repro3 base deploy FATA'd (abra convergence monitor gave up — discourse is genuinely flaky/heavy
under load, which is the very premise). repro4 reached the upgrade and the post-`chaos_redeploy`
`docker service inspect <stack>_app` capture is the smoking gun:
- `UpdateStatus = {"State":"updating","Message":"update in progress"}`
- `.Spec.Labels`  chaos-version = **7ae7b0f7+U**, version = 0.9.0+3.5.0  (HEAD spec applied OK)
- `.PreviousSpec.Labels` chaos-version = **eb96de94+U**, version = 0.7.0+3.3.1 (the base)
- `deployed_identity` (same instant) = chaos **7ae7b0f7+U**  (reads Spec, correct)
Then `wait_healthy` ran (old task serving under start-first → passes); the new task failed swarm's
monitor → `failure_action: rollback` reverted `.Spec` → `.PreviousSpec` (eb96de94+U); the
assertion-phase read saw eb96de94+U → HC1 FAIL. The ONLY operation that turns `.Spec.Labels` from
7ae7b0f7+U into the exact `.PreviousSpec` eb96de94+U is a swarm rollback. abra+harness exonerated;
the head was really deployed and then swarm-reverted. Attribution complete, by direct evidence.

Note the app image is `bitnamilegacy/discourse:3.3.1` for BOTH base and head spec (head only bumps
the version label + db image), so the new task isn't failing on a missing image — it's the
start-first 2× co-residency of the precompile/Rails-heavy app under host memory pressure (a real
new-task failure, intermittent), which trips `failure_action: rollback`.

### Fix plan (HC1 teeth preserved)
- Reliability: `tests/discourse/compose.ccci.yml` overlay → app `deploy.update_config.order:
  stop-first` (old stops before new starts → new boots with full memory → genuinely healthy → no
  spurious rollback). Upgrade-to-head still really deployed+asserted; not a weakening. WHY in header.
  Risk to weigh: stop-first = brief real downtime during the CI upgrade (covered by DEPLOY_TIMEOUT
  3600). Alternative `failure_action: pause` REJECTED — it would let a genuinely-failed new task
  pass HC1 (start-first keeps old serving) = test-weakening.
- Correctness: harness upgrade path asserts the redeploy converged to the head spec (UpdateStatus
  not rollback*/paused / `.Spec` not reverted to `.PreviousSpec`) → honest failure message on a
  real rollback, instead of the misleading "re-checkout failed". General (all rollback-policy
  recipes). HC1 teeth intact: a head that truly can't stay healthy still fails.
- Will validate stop-first actually eliminates the rollback with a full real run before claiming.

## 2026-06-11 (cont.) — fix validated + blast-radius

**Fix implemented** (commit 0cc31a5): (1) `tests/discourse/compose.ccci.yml` app service
`deploy.update_config.order: stop-first`; (2) `lifecycle.assert_upgrade_converged()` + call in
`generic.perform_upgrade` right after `chaos_redeploy` (before wait_healthy) — waits for swarm's
app-service rolling update to reach a TERMINAL state and FAILs honestly on rollback*/paused.
Unit tests: 253 passed (no regression).

**fix1 validation** (run `dstamp-fix1`, fresh checkout @0cc31a5, install+upgrade, solo): UPGRADE
**PASS** — `upgrade-converged: …UpdateStatus=completed`, `upgrade→PR-head: head_ref=7ae7b0f7
chaos-version=7ae7b0f7+U version=0.7.0+3.3.1→0.9.0+3.5.0`. The head is deployed, the update
converges (no rollback), HC1 reads 7ae7b0f7+U. (Bug was intermittent — running more to show
reliability, since repro2 passed unpatched.)

**Blast-radius sweep** — recipes with `failure_action: rollback` + `order: start-first`:
`discourse, drone, keycloak, n8n, traefik`. Evidence check of the upgrade tier across many runs
(incl. the rcust-era m2r-* runs under the same heavy load):
- keycloak: runs 155/186/187/m2r/shot-proof → upgrade PASS L4 (HC1 pass ⇒ chaos==head). NOT affected.
- n8n: runs 47/54/61/162/197/m2r/shot-proof → upgrade PASS L4. NOT affected.
- drone, traefik: cc-ci INFRA (warm-reconciled), NOT enrolled in the recipe-CI upgrade tier.
⇒ **Only discourse actually exhibits the drift** — its app is uniquely heavy (Rails asset
precompile, 2.4GB image) so the start-first 2× co-residency OOMs the new task; the lighter
keycloak/n8n new tasks survive swarm's monitor, so no rollback. The general harness guard
(`assert_upgrade_converged`) now protects ALL rollback-policy recipes from a silent future
rollback (honest failure), and discourse additionally gets stop-first to converge reliably.

### Hardening (commit e9c26c7) + fix2 validation
Adversary independently confirmed the root cause + assessed the fix CORRECT (REVIEW-dstamp probe),
flagging one non-blocking race: assert_upgrade_converged's first poll could read a STALE terminal
`completed` (from the install/base deploy) before swarm schedules the new roll → return OK
prematurely → miss a later rollback. Hardened with a two-phase wait: phase 1 confirms the NEW
update is scheduled (`UpdateStatus.StartedAt` advances past the pre-redeploy value, captured via
`update_status_started`, or state is in-flight `updating`/`rollback_started`), with a 30s grace for
a genuine no-op redeploy; phase 2 then waits for the terminal verdict. fix2 (hardened, fresh
checkout @e9c26c7, install+upgrade): UPGRADE **PASS** — `upgrade-converged: …UpdateStatus=completed`,
`chaos-version=7ae7b0f7+U version=0.7.0+3.3.1→0.9.0+3.5.0`. Two consecutive green fixed runs
(fix1+fix2) vs intermittent unpatched failures (repro1✗ repro4✗ repro2✓). Unit tests 253 pass.

### M1 claimed
Attribution + minimal repro + 06-05→06-10 change + fix + blast-radius all complete and
Adversary-pre-confirmed → claiming M1 (verification recipe in STATUS-dstamp). Next: M2 — full
all-stages discourse green at true level via the drone `!testme` path (the recipe-CI pipeline runs
`cc-ci-run runner/run_recipe_ci.py` from the drone-cloned cc-ci workspace, so e9c26c7 is live for
!testme — no nixos-rebuild needed for the harness), other recipes re-proven (none affected), HC1
teeth shown (wrong stamp still FAILs), DEFERRED closed.

Fix direction (HC1 must keep its teeth — do NOT relax the commit match): the upgrade chaos redeploy
must assert against the *intended* applied spec, not a silently rolled-back one — i.e. the harness
must DETECT a swarm rollback (UpdateStatus.State rollback*) and treat it as an upgrade FAILURE with
a clear message (the deploy did not converge to the head spec), AND/OR make the upgrade redeploy not
subject to silent rollback masking (e.g. assert UpdateStatus completed before reading identity).
The recipe's rollback policy is legitimate for prod; the harness bug is that a rollback is invisible
to HC1 and masquerades as "stamped the wrong commit". Will finalise the fix after repro3 confirms.