cc-ci/machine-docs/JOURNAL-samever.md

# JOURNAL — phase `samever` (Builder reasoning; Adversary does not read before verdict)

## 2026-06-17 — M1 design + implementation

**Root cause (confirmed against `runner/run_recipe_ci.py`):** the warm-canonical path of
`resolve_upgrade_base` returned `BasePlan("version", rec["version"], …)` unconditionally — it was
never given the head's *version*, only `head_ref` (a commit sha), so it could not detect the
canonical==head collision. The ref (main-tip) path was already guarded (`main_tip == head_ref →
skip`); the version path was not. In the nightly steady state a green cold-on-latest run promotes
`canonical → latest`, so the *next* night finds `canonical == latest == version-under-test` and the
upgrade tier deploys base==head: a vacuous same-version "upgrade."

**Why pass `head_version` as a param rather than read compose inside the resolver:** keeps the
resolver pure/unit-testable (the existing 8 tests inject `canonical.read_registry` /
`lifecycle.recipe_branch_commit` via monkeypatch and never touch the filesystem). The call site
(`main()`) reads it once via `abra.head_compose_version(recipe)` from the head checkout that already
exists on disk. Tests pass `head_version=` directly.

**Why `version_key`-based equality instead of raw string `==`:** the canonical record version and the
compose label *should* be byte-identical when equal, but routing both through the existing coop-cloud
ordering key (`warm_reconcile.version_key`) means a re-published or incidentally-reformatted equal
version still compares equal, and the step-back's "strictly older" uses the *same* single ordering
source — no hand-rolled semver (plan §2 constraint). `version_key` is the inner key of the existing
`sort_versions`, lifted out so `sort_versions`/`newest_older_version` share it (no behavior change to
`sort_versions` — verified by the unchanged existing warm_reconcile tests).

**Why the step-back inherits F1d-2 automatically:** it returns `kind="version"` exactly like the
normal canonical base, so it flows through the same deploy path (`abra.recipe_checkout` pins the tag
on disk, non-chaos deploy) — the chosen older base genuinely deploys that pinned version, never
LATEST. No new deploy code; the protection is structural.

**Skip only when genuinely no older predecessor:** `newest_older_version` returns None only when the
head version is the oldest (or only) published tag — then, and only then, a declared skip
(`"base == head … and no older published predecessor"`), never a same-version no-op.

**`head_version is None` (compose unreadable / no label):** cannot compare → `same=False` →
preserves prevb behavior exactly (canonical is primary). No regression for any caller that omits
`head_version`; the existing `test_last_green_warm_canonical_is_primary` still passes unchanged.

**Pre-existing unrelated failures** (confirmed failing on clean `279d84d` with my changes stashed,
so NOT introduced here): `tests/unit/test_meta.py::test_generated_doc_table_in_sync` and
`tests/unit/test_warm_reconcile.py::test_traefik_spec_is_stateless_with_setup` (KeyError
'health_domain'). Out of scope for samever.

## 2026-06-17T04:25Z — M1 claimed; M2 prep (no gate runs until M1 PASS)

M1 claimed (c5a0d20). Parked at gate; doing read-only M2 prep:
- Trigger mechanism (from prevb M2): `!testme` on a recipe PR → bridge (polls 30s) → Drone build of
  cc-ci@main (now = samever code) → artifacts at `/var/lib/cc-ci-runs/<N>/` (junit/results.json,
  Adversary-readable). Local full-pipeline runs on cc-ci de-risk before posting.
- Enrolled (WARM_CANONICAL=True) recipes: only **custom-html** currently. No canonical registries on
  cc-ci right now (`/var/lib/cc-ci-canonical/` empty).
- M2 plan shape: (1) nightly steady state — seed custom-html canonical registry version = its LATEST
  published tag, run cold-on-latest → assert upgrade tier `kind=version`, base_version < latest
  (step-back, genuine delta, not no-op/skip). (2) PR form — non-version-bump PR, head==canonical, same
  step-back. (3) discourse #4 version-bump → UNAFFECTED (canonical→head). (4) spot-check ≥1 other
  enrolled recipe (only custom-html enrolled today — resolve during M2: enroll/seed a 2nd, or use the
  registry mechanism on another recipe). Need ≥2 published tags on the step-back recipe for an older
  target to exist — verify custom-html tag count before run.

## 2026-06-17T04:40Z — M2 real-CI evidence captured (custom-html + discourse)

Two-run authentic nightly simulation on cc-ci (/root/samever-deploy @ cc-ci main, samever code):
- **Run A** (cold-on-latest, no canonical): upgrade base kind=skip (head==main tip); green 5 tiers;
  WC5 promote → canonical custom-html = 1.13.0+1.31.1 (the "first nightly").
- **Run B** = THE HEADLINE (2nd consecutive nightly, canonical==latest==head):
  `upgrade base: kind=version version=1.11.0+1.29.0 (step-back: last-green canonical (1.13.0+1.31.1)
  == head version 1.13.0+1.31.1; newest older published base)`. Upgrade tier deployed base 1.11.0+1.29.0
  then chaos-upgraded to head: `version=1.11.0+1.29.0→1.13.0+1.31.1` (label MOVED, base<head, REAL
  delta — not a no-op, not a skip). All 5 tiers green. Proves F1d-2: the older base actually deployed
  the pinned 1.11.0 then upgraded to 1.13.0.
- **Run C** (version-bump UNAFFECTED, enrolled): re-seeded canonical→OLDER 1.11.0+1.29.0, cold-on-latest
  head 1.13.0 → `kind=version version=1.11.0+1.29.0 (last-green (warm canonical, status=idle))` —
  reason "last-green", NOT "step-back": the unchanged prevb path. Upgrade 1.11.0→1.13.0 green. The
  step-back never engages when canonical≠head.
- **discourse #4** (non-enrolled version-bump, REF=ae5a8180): `kind=ref ref=f87c612d71b4 (target-branch
  (main) tip)` — byte-identical to prevb run 717; discourse never enters the canonical branch, so samever
  cannot perturb it. (Full install,upgrade migration running to green for completeness.)

Artifacts preserved on cc-ci: /root/samever-run{A,B,C}.log, /root/samever-disc4.log; run B/C results
copied to /var/lib/cc-ci-runs/samever-run{B,C}/ (Adversary-readable).

## 2026-06-17T04:55Z — M2 complete (PR form + spot-check), claiming

- **Run D (PR form):** ran custom-html with REF=2b82ebab PR=999 (a PR head whose compose version is
  still 1.13.0 == canonical). Resolver stepped back to 1.11.0+1.29.0 even with the ref present —
  confirming the step-back is ref-independent (the canonical branch precedes the main-tip/ref path).
  Upgrade 1.11.0→1.13.0 green.
- **Spot-check (hedgedoc):** only custom-html is WARM_CANONICAL-enrolled, so to exercise the resolver on
  a SECOND recipe + different tag ordering I hand-seeded hedgedoc's canonical record to its latest
  (3.0.10+1.10.8) — the resolver reads canonical.read_registry regardless of enrollment, so this is the
  same production code path. cold-on-latest → step-back to 3.0.9+1.10.7, upgrade green. Removed the
  seeded record afterward (`rm -rf /var/lib/ci-warm/hedgedoc`) to leave clean state; hedgedoc is not
  enrolled and would be pruned anyway.
- **State hygiene:** custom-html canonical left at the legitimately-promoted 1.13.0+1.31.1 (its real
  enrolled steady state). No leftover run stacks (clean teardown verified). Pre-existing warm-keycloak
  orphan untouched.

Design B (canonical history) is already recorded out-of-scope in cc-ci-plan/IDEAS.md (per plan §5) —
verify before DONE.