cc-ci/REVIEW-rcust.md

# REVIEW-rcust.md — Adversary ledger for the recipe-customization restructure phase

SSOT for this phase: `/srv/cc-ci/cc-ci-plan/recipe-custom-restructure-full-plan.md`.
Gates: **M1** (implementation verified — branch `restructure/recipe-custom`, unit+concurrency+lint
green on cold clone, resolved-customization diff clean for all 21 recipes, adversarial diff review)
and **M2** (merged + real-CI regression sweep matching baseline matrix). DONE requires fresh PASS
for both with no open VETO.

I own this file and the `## Adversary findings` section of BACKLOG-rcust.md only.

---

## Standing watch items (what I will hunt at M1/M2)

- **Coverage loss** (cardinal risk): for every migrated recipe, old loaders' effective customization
  values must equal new `meta.load()` values. Throwaway diff script over all 21 recipe dirs; any
  delta = finding.
- **Assertion weakening** in `tests/<recipe>/` diffs — migrations must be mechanical only (signatures,
  fixture/key renames, underscore prefixes). Any changed assert/expected value = VETO.
- **Deleted-code fallout** — dangling refs to `_recipe_meta`, `_load_meta`, `_recipe_extra_env`,
  `_recipe_meta_flag`, `declared_deps`, `is_canonical_enrolled`, `OIDC_AT_INSTALL`,
  `CHAOS_BASE_DEPLOY`, `SKIP_GENERIC`, `setup_custom_tests`, `deps_apps`, `deps_creds`, `deployed_app`.
- **Validation gaps** — typo'd key / wrong type / callable-on-data-key must raise MetaError, not pass.
- **R2 fixed end-to-end** — orchestrator load path delivers SCREENSHOT to screenshot.py.
- **HC2 / F2-11 integrity** — repo-local default-deny, requires_deps skip-report, generic floor
  semantics all unchanged.

---

## Verdicts

_(no GATE verdict yet — M1 is not claimed. M1 only claims after P1–P6 are all on the branch;
Builder has landed P1 (472a68b) + P2 (8cd72fd) and is mid-P3. The interim pre-review below is
front-loaded break-it work on the FROZEN P1/P2 commits — NOT an M1 PASS.)_

### Interim pre-review of frozen P1+P2 (branch @ 8cd72fd) — @2026-06-10, cold from upstream clone

Done as idle-time break-it work while no gate is pending. P1/P2 phase commits won't be rewritten
(Builder adds P3+ on top), so reviewing them now is non-wasted and front-loads M1. Cold clone of
`origin/restructure/recipe-custom` into `/tmp/rcust-verify` from the true upstream remote.

**No defects found so far.** Results:

1. **Deleted-code fallout — CLEAN.** Grepped `runner/ tests/ scripts/` for live refs to every deleted
   symbol (`_recipe_meta`, `_load_meta`, `_recipe_extra_env`, `_recipe_meta_flag`, `declared_deps`,
   `is_canonical_enrolled`, `OIDC_AT_INSTALL`, `CHAOS_BASE_DEPLOY`, `SKIP_GENERIC`,
   `setup_custom_tests`, `deps_apps`, `deps_creds`, `deployed_app`). All hits are comments/docstrings
   explaining the deletion, test names, or the intentionally-RETAINED `CCCI_SKIP_GENERIC*` env form
   (kept per P2c). Zero live call-sites. `setup_custom_tests.sh` files gone.
2. **All-recipes-load-clean (typo gate) — PASS, independently.** Ran `meta.load()` (pure stdlib) over
   all 21 recipe dirs cold via plain python3 (did NOT trust the Builder's test_meta.py). All 21 load;
   non-default key sets sane. Every ALL-CAPS key used in any recipe_meta.py is in the 14-key registry.
3. **Coverage-loss diff (CARDINAL check) — ZERO deltas on data keys + hook presence.** Throwaway
   harness (`/tmp/diff_meta.py`) reproduces main's six-loader effective resolution (`_load_meta`,
   `declared_deps`, `is_enrolled`, `_recipe_extra_env`) from MAIN's recipe_meta files and diffs vs the
   BRANCH's `meta.load()` for all 21 recipes. After correcting one harness artifact (EXTRA_ENV default
   is `{}` not None), **0/21 recipes show any delta** for HEALTH_PATH/HEALTH_OK/DEPLOY_TIMEOUT/
   HTTP_TIMEOUT/BACKUP_CAPABLE/EXPECTED_NA/UPGRADE_BASE_VERSION/DEPS/WARM_CANONICAL + presence of
   READY_PROBE/BACKUP_VERIFY/UPGRADE_EXTRA_ENV/EXTRA_ENV/SCREENSHOT.
4. **Validation gaps — CLOSED.** Crafted tmp recipe_metas: typo'd key → MetaError (with "did you mean
   DEPLOY_TIMEOUT?"); wrong type (`DEPLOY_TIMEOUT="str"`) → MetaError; callable on data key
   (`DEPLOY_TIMEOUT=lambda ctx:...`) → MetaError; `_PRIVATE`/lowercase-helper → loads clean (exemption
   works). All four behave per the locked decision.
5. **meta.py read** — single `exec()`, frozen `RecipeMeta` generated from `KEYS`, `_coerce` rejects
   bool-as-int and callable-on-data-key; `non_default` compares vs registry default. No issues.

**Still UNVERIFIED for M1 (do NOT treat above as M1 PASS):** full `pytest tests/unit -q` +
`pytest tests/concurrency -q` + `scripts/lint.sh` cold on the cc-ci host; R2 end-to-end through the
real orchestrator screenshot path; P3 ctx-hook signature migration (assert byte-identical, legacy
`lambda domain:` raises clear MetaError); P4/P5/P6; re-run the coverage diff on the FINAL branch
(P3 changes hook signatures); recipe-test diffs are mechanical-only (no assertion weakening);
HC2/F2-11/generic-floor integrity. These wait for the `claim(rcust): M1`.

### Interim pre-review of frozen P3 (branch @ fd02d9f) — @2026-06-10, cold from upstream clone

Builder landed P3 (uniform ctx hook convention) and moved to P4, so P3 is frozen. Pre-reviewed it.
**No defects found.**

1. **Mechanical-migration discipline — HELD (no VETO trigger).** `git diff 8cd72fd..fd02d9f` over
   `tests/*/` shows ZERO changed assert/expected literals. Every hook change is purely
   `def HOOK(domain[, meta])` → `def HOOK(ctx)` + `domain` → `ctx.domain` in the body. Spot-checked
   cryptpad/mumble/ghost/lasuite-drive recipe_meta.py + lasuite-drive ops.py: seeded values, return
   dicts, paths, status codes, and the `pre_restore` `assert _psql(...) in (...)` are byte-identical
   apart from the `ctx.` deref.
2. **HookCtx — present + complete.** `meta.HookCtx` frozen dataclass has all 5 documented fields
   (`.domain`, `.base_url`, `.meta`, `.deps`, `.op`); `meta.hook_ctx(domain, meta, op=…)` factory
   builds it and pulls `deps` from `$CCCI_DEPS_FILE`. All call sites migrated: run_recipe_ci
   `pre_<op>`, BACKUP_VERIFY; lifecycle `extra_env` + READY_PROBE; screenshot `SCREENSHOT(page, ctx)`.
   (NB my first pass falsely flagged "no HookCtx" — that was a STALE WORKTREE at P2; corrected by
   checking out fd02d9f. Logged here for honesty.)
3. **Legacy-signature guard (P3.4) — PRESENT + works, live-probed.** `meta.check_hook_signature`
   exact-matches positional params and raises a CLEAR MetaError naming the P3 migration + HookCtx
   fields. Wired into both `load()` (recipe_meta hooks; SCREENSHOT expects `(page, ctx)`, rest
   `(ctx)`) and the orchestrator (ops.py `pre_<op>`). Crafted tmp metas: legacy `READY_PROBE(domain)`,
   `SCREENSHOT(page, domain, meta)`, `EXTRA_ENV(domain)` all → MetaError at load; `READY_PROBE(ctx)`
   loads clean. No silent mid-run TypeError path.
4. **Coverage diff re-run at P3 head — still 0/21 deltas** (hook presence + all data keys unchanged).

Net: P1+P2+P3 all clean under cold adversarial probing. M1 still gated on full unit+concurrency+lint
on the cc-ci host, P4–P6, R2 end-to-end via the real screenshot orchestrator path, and a final
coverage re-diff. No findings filed; no VETO.

### Interim pre-review of frozen P4 (branch @ 29a28e2) — @2026-06-10T18:55Z, cold from fresh host clone

Builder landed P4 (custom-test ergonomics) and moved to P5, so P4 is frozen. Pre-reviewed it cold.
**No defects found.** NOT an M1 verdict — M1 stays gated (see "Still UNVERIFIED" below).

Cold acceptance (fresh `git clone` on cc-ci host at 29a28e2, my own checkout — not the Builder's):
- `cc-ci-run -m pytest tests/unit -q` → **184 passed** (exact match to claim; full suite, no
  cross-fixture pollution from the session-scoped `deps` fixture).
- `cc-ci-run -m pytest tests/unit/test_discovery.py test_discovery_phase2.py
  test_conftest_fixtures.py -q` → 14 passed.
- `nix develop .#lint --command scripts/lint.sh` → **lint: PASS** (ruff format/check, deadnix,
  shfmt, shellcheck, yamllint all clean).

Correctness probes:
1. **Placement-rule claim ("zero in-repo users of top-level custom tests") — HOLDS.** Filesystem
   sweep of every `tests/<recipe>/test_*.py`: ALL are lifecycle names (test_{install,upgrade,
   backup,restore}.py). No top-level non-lifecycle custom exists in-repo, so dropping the top-level
   glob in `discovery.custom_tests` loses ZERO coverage. The lifecycle-name exclusion is retained
   inside functional/playwright as the double-run safety net.
2. **Discovery diff — clean.** Top-level `glob(test_*.py)` branch removed; functional/ + playwright/
   subdir globs retained with `basename not in lifecycle_names` guard. Docstring + module header
   updated to state the placement RULE.
3. **Test changes are adaptation + strengthening, NOT weakening (no VETO trigger).**
   - `test_discovery_phase2`: renamed to `..._placement_rule_...`; now ASSERTS the top-level
     `test_sso_smoke.py` is `not in names` (new negative assertion proving the behavior change),
     while functional/playwright customs are still `in names` and lifecycle name excluded.
   - `test_discovery::test_custom_tests_repo_local_gated`: repo-local custom moved from top-level
     into `functional/`; HC2 default-deny (`== []` when unapproved) and approved-case
     (`functional/test_sso.py in names`, `test_install.py` excluded) both INTACT. HC2 integrity
     preserved.
4. **op_state fixture — correct.** Skips with clear reason on unset env / missing file / non-JSON
   (`except ValueError` catches JSONDecodeError); reads & returns parsed dict otherwise. Tests
   cover 3 of 4 paths (the non-JSON skip path is untested — minor coverage gap, not a defect; the
   branch is trivially correct by inspection).

Net: P1+P2+P3+P4 all clean under cold adversarial probing; both halves of every phase claim
(unit count + lint) reproduced cold on a fresh clone. No findings filed; no VETO.

**Still UNVERIFIED for M1 (do NOT treat above as M1 PASS):** P5 (manifest) + P6 (docs);
`pytest tests/concurrency -q` cold; R2 end-to-end through the real orchestrator screenshot path;
final coverage re-diff on the COMPLETE branch (P1–P6, all 21 recipes, effective customization set
unchanged); recipe-test diffs mechanical-only across the whole branch; HC2/F2-11/generic-floor
integrity at the final head. These wait for `claim(rcust): M1`.

### Interim pre-review of frozen P5 (branch @ 68954be) — @2026-06-10T19:06Z, cold from fresh host clone

Builder landed P5 (customization manifest) and moved to P6, so P5 is frozen. Pre-reviewed it cold.
**No blocking defect; one secret-SURFACE observation raised (heads-up to Builder, NOT a VETO, NOT
an M1 secret-leak failure).** NOT an M1 verdict.

Cold acceptance (fresh `git clone` on cc-ci host at 68954be, my own checkout):
- `cc-ci-run -m pytest tests/unit -q` → **191 passed** (exact match to claim).
- `nix develop .#lint --command scripts/lint.sh` → **lint: PASS**.

Primary adversarial target — SECRET LEAKAGE via the new manifest surface (D-gate: published logs +
dashboard contain NO secrets, incl. generated app passwords):
1. **Generated/runtime secrets — NOT exposed (gate holds).** `manifest.build` collects only:
   `meta_non_default` (static recipe_meta), hook NAMES (pre-ops/install_steps.sh/compose.ccci.yml),
   overlay FILENAMES, custom-test COUNTS, and env-override KEY names (printed `KEY=1`, value never
   rendered). It never touches `deps` (client_secret), `op_state`, abra-generated app passwords, or
   any env VALUE. The cardinal concern — generated app passwords on the dashboard — is structurally
   absent from this surface.
2. **Cold all-recipes sweep.** Built+rendered the manifest for all 21 recipes on the host; grepped
   the rendered blocks AND the results.json `customization` payload for secret/password/token/key/
   credential and for any 32+ char high-entropy string. The ONLY hit, across every recipe, is
   plausible's `EXTRA_ENV.SECRET_KEY_BASE` =
   `"ccciplausibletestkeybase64charsexactlyforCIephemeral4567890123"`.
3. **OBSERVATION (not a leak):** that value is a HARDCODED, committed, PUBLIC dummy CI constant
   (tests/plausible/recipe_meta.py, in the open-source repo) — not a generated or real secret.
   `meta_non_default` dumps EXTRA_ENV literal dicts verbatim into the log AND results.json (→
   dashboard), so a field literally named `SECRET_KEY_BASE` with a value now appears on the
   dashboard. No real secret is exposed (it's public), so this is NOT a D-gate failure and does NOT
   block P5. BUT it's a standing surface: (a) a dashboard secret-scan gets a true-positive-shaped
   hit on a public dummy (noise that could mask a real leak), and (b) if any recipe ever set a real
   secret-ish literal in a meta dict, the manifest would surface it unredacted. Flagged to Builder
   via BUILDER-INBOX as a heads-up to consider redacting values of sensitive-named meta keys before
   M1. Will re-examine on the real dashboard at the M1 cold-verify.
4. **HC2-honoring — confirmed.** Manifest routes ALL repo-local reads through `discovery._gated`
   (ops.py loop direct; `install_steps`/`resolve_overlay_op`/`custom_tests` each call `_gated`
   internally). An unapproved repo-local recipe contributes nothing to the manifest.
5. **Pure presentation — holds.** `build()` only reads files/env and returns a dict; `render()`
   formats a string. Called at run_recipe_ci.py:889-890 (print) + embedded at :1261 into results;
   no state mutation, no verdict influence. `_jsonable` renders callables as `'<hook>'` (so a
   callable EXTRA_ENV/READY_PROBE never leaks closure internals) and tuples→lists for JSON.

Net: P1–P5 all clean under cold adversarial probing; every phase claim (unit count + lint)
reproduced cold. No findings filed; no VETO. One non-blocking secret-surface heads-up sent.

**Still UNVERIFIED for M1:** P6 (docs); `pytest tests/concurrency -q` cold; R2 end-to-end via the
real orchestrator screenshot path; final coverage re-diff on the COMPLETE branch (all 21 recipes,
effective customization unchanged); recipe-test diffs mechanical-only across the whole branch;
HC2/F2-11/generic-floor integrity at final head; AND — at the M1 dashboard check — confirm the
SECRET_KEY_BASE-named field on the real dashboard is the accepted public dummy (or redacted).
These wait for `claim(rcust): M1`.

## M1 — implementation verified: **PASS** @2026-06-10T19:27Z (branch `restructure/recipe-custom` @ 858e0f5)

Cold-verified from TWO fresh clones on the cc-ci host (NEW=858e0f5, OLD=main pre-restructure;
merge-base 49fb818 confirmed → `main..858e0f5` is exactly P1–P6). Verdict formed from the phase plan
(SSOT), the code/git history, the STATUS verification facts, and my own cold re-runs — NOT from
JOURNAL rationale (isolation discipline; I did not need to consult JOURNAL).

**All M1 Definition-of-Done items PASS:**

1. **Cold test suites — match claim exactly.** Fresh clone @858e0f5:
   `cc-ci-run -m pytest tests/unit -q` → **192 passed**; `tests/concurrency -q` → **23 passed**
   (untouched by this plan, proven); `nix develop .#lint --command scripts/lint.sh` → **lint: PASS**.

2. **Coverage diff (cardinal risk) — 0 REAL deltas / 21 recipes.** Wrote throwaway extractors that
   resolve EVERY recipe's effective customization in BOTH worlds — OLD via the legacy loaders
   (`_load_meta` + `lifecycle._recipe_extra_env` + `deps.declared_deps` + `_recipe_meta_flag`),
   NEW via `meta.load()` + `meta.extra_env/upgrade_extra_env` — for the common keys (HEALTH_*,
   timeouts, DEPS, EXTRA_ENV resolved at a fixed domain, UPGRADE_EXTRA_ENV, BACKUP_CAPABLE,
   EXPECTED_NA, UPGRADE_BASE_VERSION, READY_PROBE/BACKUP_VERIFY presence). Diff = **0 behavioral
   deltas**; the only raw diffs were 20× `UPGRADE_EXTRA_ENV: None→{}` (unset default representation,
   behaviorally identical) and mumble (most-customized: callable EXTRA_ENV→dict, UPGRADE_EXTRA_ENV,
   READY_PROBE) is **byte-identical** old↔new.
   Deleted keys accounted for (no silent loss): `SKIP_GENERIC` (0 recipe users); `CHAOS_BASE_DEPLOY`
   → overlay-presence (discourse+ghost, exactly the two shipping compose.ccci.yml — perfect 1:1, no
   change either direction); `OIDC_AT_INSTALL` → install-time made universal (drive+meet were
   already install-time). **lasuite-docs** declared DEPS but NOT OIDC_AT_INSTALL → OLD post-install,
   NEW install-time: an INTENTIONAL P2b consolidation, not a drop — flagged below for M2 validation.

3. **Assertion weakening (VETO-class) — NONE.** Full branch diff over all recipe test files
   (excl. harness unit/concurrency/regression): 18 removed asserts, 18 added. After mechanical
   normalization (`domain`→`ctx.domain`, `deps_creds`→`deps`, `MAX_USERS`→`_MAX_USERS`, whitespace)
   the removed and added assert sets are **IDENTICAL** — zero unmatched in either direction. Every
   change is a pure signature/fixture/constant rename; no expected value altered, no assert deleted.
   Spot-confirmed discourse/ghost `_psql(domain,…ci_marker…) in (…)` → `ctx.domain` only (expected
   tuple + SQL byte-identical). **No VETO.**

4. **Deleted-code fallout — clean.** No dangling LIVE refs to any of the 13 deleted symbols
   (`_recipe_meta`/`_load_meta`/`_recipe_extra_env`/`_recipe_meta_flag`/`declared_deps`/
   `is_canonical_enrolled`/`OIDC_AT_INSTALL`/`CHAOS_BASE_DEPLOY`/`SKIP_GENERIC`/`setup_custom_tests`/
   `deps_apps`/`deps_creds`/`deployed_app`). Only residue: stale DOC/comment mentions of
   `OIDC_AT_INSTALL` + `setup_custom_tests.sh` in PARITY.md files (non-blocking P6 cosmetic nit).

5. **Validation gaps — closed.** Cold-probed `meta.load()` with synthetic bad metas: typo'd key,
   str-on-int, bool-as-int, callable-on-data-key, legacy hook sig `READY_PROBE(domain)`, and unknown
   key ALL → `MetaError` (clear, names the offending file/key). Clean + underscore-private-helper
   metas load fine (no false positives). No silent pass.

6. **R2 fixed end-to-end.** Cold proof through the REAL load path: a recipe declaring
   `def SCREENSHOT(page, ctx)` is surfaced by `meta.load()` and resolved callable by
   `screenshot._load_screenshot_hook` (old L1 allowlist dropped it — now arrives); orchestrator wires
   it `run_recipe_ci.py:1029 capture(…, recipe_meta=meta)` → `hook(page, hook_ctx(domain, meta))`.
   Absent recipe → None (default landing-page path). Legacy `SCREENSHOT(page, domain, meta)` sig
   rejected at load.

7. **HC2 / F2-11 / generic-floor integrity — preserved.** Cold-probed `discovery.custom_tests` +
   `install_steps`: UNAPPROVED repo-local → `[]` / `None` (default-deny holds); APPROVED → surfaced.
   `sso_dep_unverified` (F2-11) logic UNCHANGED (only a comment edited) — a deps-not-ready run that
   skips ≥1 `requires_deps` test still suppresses the green signal. Generic floor `_skip_generic`
   default = run (additive); opt-out now env-only (same env vars as before; the 0-user meta key
   removed) and surfaced LOUDLY in CI + flagged `!!` in the manifest — strictly stronger, never
   silent.

8. **(Bonus) P5 secret-surface heads-up RESOLVED + verified.** The Builder landed `858e0f5`
   redacting secret-named meta values in the manifest (my P5 BUILDER-INBOX ask). Cold-verified:
   `plausible.EXTRA_ENV.SECRET_KEY_BASE` → `<redacted>` in BOTH the log block and results.json;
   recursive into nested dict keys; word-segment `(^|_)KEY(_|$)` regex avoids over-match
   (KEYCLOAK_* passes). All-21-recipe sweep: exactly 1 redaction, ZERO over-redaction, ZERO
   under-redaction (no secret-shaped value remains). Regression test
   `test_manifest_redacts_sensitive_named_values` present.

**Verdict: M1 PASS.** No findings filed, no VETO.

**This does NOT clear `## DONE`.** Per the phase DoD, DONE requires a fresh Adversary PASS for BOTH
M1 *and* M2. M2 (merged-main real-CI regression sweep vs the committed baseline matrix) is still
unverified. M2 watch-items I will specifically re-check from run logs:
- **lasuite-docs OIDC is now install-time** (post→install change above) — must pass a real run with
  OIDC wired at install (skip-count 0 on its `requires_deps` tests).
- the customization spot-checks the plan §M2.4 enumerates (mumble READY_PROBE tcp lines, cryptpad
  SANDBOX_DOMAIN, ghost/discourse BACKUP_VERIFY + overlay copy + auto-chaos base deploy, lasuite-*
  deps provisioning + OIDC tests ran, immich ops.py seeds, manifest block present in every log,
  screenshot.png where capture succeeded).
- canary suite (RED canaries still caught at intended tier) + per-recipe level == baseline matrix.
- zero leaked apps after teardown.

### M2-prep — independent hook-port audit (shell→python / best-effort↔fatal drift) @2026-06-10T20:55Z

Triggered by the lasuite-drive regression (below), which my M1 PASS MISSED: my M1 coverage diff
compared recipe_meta KEYS (resolved values), not ops.py hook BODIES, and my assertion scan matched
`assert ` not `raise AssertionError`. So a hook that flipped best-effort→fatal was invisible to my
M1 method. M2 (real-CI sweep) caught it — the safety net working as designed. I then audited ALL
hook ports cold (`git diff c2508c7..origin/main` per recipe ops.py + the 2 setup_custom_tests.sh
ports), filtering for non-mechanical error-handling (raise/assert/except/exit/timeout/poll changes):

- **lasuite-drive `pre_install`** — GENUINE rcust regression (Builder-disclosed, I confirmed):
  OLD setup_custom_tests.sh bucket poll fell through on 90s timeout (best-effort, no failure; the
  custom-tier `test_minio_storage.py` upload→list→download is the real gate); NEW port added a
  terminal `raise AssertionError` → deterministic install RED when the bucket appears just after
  90s. Fix-forward APPROVED (restore best-effort print+return, scoped to line-54 only; conditioned
  on an L5 re-run + my diff re-verify). See approval entry in BUILDER-INBOX history (commit 57c66ad).
- **lasuite-docs `install_steps.sh`** — INTENTIONAL P2b change, NOT a defect: OLD setup_custom_tests
  did `exit 1` on missing deps/null KC creds; NEW does `exit 0` (no-op) for missing-deps (gated now
  by F2-11: the `@requires_deps` OIDC test skips → `sso_dep_unverified` suppresses green) BUT
  preserves `exit 1` on secret-insert failure. Consistent with the install-time-deps redesign.
  WATCH-ITEM (residual): the missing-deps path now relies entirely on F2-11; the sweep didn't
  exercise it (deps were ready, skip-count 0). Mechanism verified present at M1; not blocking.
- **All other ops.py** (cryptpad, discourse, ghost, immich, keycloak, lasuite-meet, matrix-synapse,
  mattermost-lts, mumble, n8n, plausible, custom-html) — pure mechanical ctx migration
  (`domain`→`ctx.domain`, `meta`→`ctx.meta`); expected tuples/strings byte-identical (spot-checked
  keycloak 201/409 + 204/200, discourse/ghost _psql ci_marker). No error-handling drift.

Net: exactly ONE accidental hook-port regression (lasuite-drive), now under approved fix. No other
best-effort↔fatal flips. This audit closes the M1-method gap for the hook bodies.

---

### M2 proof-run independent analysis (cold, Adversary) @2026-06-10T23:53Z

M2 is NOT yet claimed by the Builder; this is my independent read of the proof runs sitting on
cc-ci (`/var/lib/cc-ci-runs/{m2b-*,ab-*-oldmain}`), parsed myself via jq (NOT trusting Builder
narrative). The 6 first-sweep mismatches break down as follows.

**Confirmed root fact — REF MISMATCH is real (I verified, not taken on faith).** Every baseline
matrix run used a *PR-head* ref; the first M2.3 sweep used each mirror's *default-branch head* — a
different commit. Independently confirmed via `results.json.ref`:
| recipe | baseline run/ref/level | sweep ref/level |
|---|---|---|
| discourse | 184 / 7ae7b0f76efb / L4 | 7d53d4ec390f / L2 |
| plausible | 308 / 13458fac56a1 / L4 | da159375d89a / L2 |
| mattermost-lts | 196 / a333e31a6002 / L4 | 41c9eb8e5f34 / L2 |
| immich | 307 / 107d7220adce / L4 | 7eb3937a82d0 / L2 |
| lasuite-drive | 189 / ffa7d585afa2 / L5 | f4135d78201e / L0 |
So the sweep was NOT apples-to-apples vs the baseline matrix. Reconciliation requires either
(a) re-run at the baseline ref on new main == baseline level, or (b) A/B same-ref old-vs-new main
== same level. Status per recipe:

- **immich** — m2b-immich (new main, baseline ref 107d7220adce) = **L4 == baseline L4. CLEAN.**
- **mattermost-lts** — m2b (new main, a333e31a6002) = **L4 == baseline L4. CLEAN.**
- **plausible** — m2b (new main, 13458fac56a1) = **L4 == baseline L4. CLEAN.**
  → these three: restructure proven INNOCENT (baseline ref reproduces baseline level on merged main).
- **bluesky-pds** — ab-bluesky-pds-oldmain (OLD main, b2d86efba3f1) = L0 == new-main sweep L0 at
  same ref → restructure-NEUTRAL at the sweep ref. (Baseline is "L4-equiv, pre-results-era", no run
  id — softer baseline; A/B neutrality is the available evidence.)
- **discourse — NOT yet clean. OPEN.** Two *distinct* flake modes seen, and the A/B was run at the
  wrong ref to close the gap:
  - baseline 184 (OLD main, 7ae7b0f): all pass → L4.
  - m2b-discourse (NEW main, SAME ref 7ae7b0f): **upgrade FAILED**, HC1 guard fired —
    "upgrade deployed chaos commit 'eb96de94+U', not intended PR-head '7ae7b0f76efb' — re-checkout
    to code-under-test failed (HC1)" → L1.  ← same-ref old=L4 vs new=L1 discrepancy, UNexplained.
  - ab-discourse-oldmain (OLD main, 7d53d4ec): **restore FAILED** (ci_marker truncated-dump race)
    → L2 == new-main sweep L2 at that ref → neutrality proven, but for the RESTORE mode at the
    DEFAULT-head ref, NOT for the L1/upgrade-HC1 mode at the baseline ref.
  - Net: the clean A/B (ref 7ae7b0f on OLD main vs NEW main) that would explain L4→L1 was NOT run.
    The upgrade re-checkout/HC1 path lives in run_recipe_ci.py/lifecycle which the meta-param
    threading DID touch — so "pre-existing flake" is plausible but UNPROVEN here. To clear: run
    discourse @7ae7b0f on OLD main (does it deterministically reproduce L4, or also flake to L1?),
    and/or repeat @7ae7b0f on new main to characterise the HC1 re-checkout as a race. The HC1 guard
    FIRING (not silently passing the wrong commit) is the safety net working — good — but it means
    the upgrade did not exercise the PR code, so the run is inconclusive, not a clean baseline match.
- **lasuite-drive** — fix-forward 1357544 (restore best-effort bucket poll) landed; needs a fresh
  L5 run at the baseline ref ffa7d585afa2 on merged main to confirm baseline. m2rr/earlier runs
  predate or used the default head — NOT yet a clean baseline match. OPEN.

**M2 disposition: still OPEN — no PASS.** 3/6 cleanly reconciled (immich/mattermost/plausible);
bluesky neutral-at-sweep-ref; discourse + lasuite-drive NOT yet closed. I will require, at the M2
claim: (1) discourse same-ref A/B (or repeat) explaining L4→L1; (2) a clean lasuite-drive L5 at
baseline ref; (3) my own cold re-parse of every per-recipe level vs baseline; (4) the M2.4
customization-executed spot-greps; (5) zero leaked apps. Recorded a BUILDER-INBOX heads-up on the
discourse-HC1 gap so it is addressed in the claim, not glossed as "the restore flake".

### M2 proof-run progress + self-correction @2026-06-11T00:05Z

Builder is running (independently, matching my inbox ask) the decisive A/B serially on the box:
`m2-proof.sh` → lasuite-drive @ffa7d585afa2 PR=1 (post-fix-forward 1357544) on merged main 5c0676b,
then discourse @7ae7b0f76efb **PR=2** on merged main (m2p-discourse); `m2-proof2.sh` (queued) →
discourse @7ae7b0f76efb **PR=2** on OLD main (/root/m2-oldmain, ab-discourse-7ae7b0f-oldmain).

**Self-correction to my 23:53Z discourse analysis:** my m2b-discourse run used **PR=0**, but the
upgrade HC1 guard resolves the *PR head* for the re-checkout. The L1 failure message ("deployed
chaos commit 'eb96de94+U', not PR-head 7ae7b0f — re-checkout failed") is plausibly a **PR=0
artifact** (no real PR to resolve the head from), NOT a restructure regression. The Builder's proof
runs correctly use PR=2 (matching baseline run 184's pr=2). So the apples-to-apples comparison I
need is m2p-discourse (PR=2, new main) vs ab-discourse-7ae7b0f-oldmain (PR=2, old main) vs baseline
184 (PR=2, old main, L4). I will cold-verify those three when they land; my L4→L1 concern is on
hold pending the PR=2 result, not yet a confirmed regression. Live lasu-f68b63 stack = active
lasuite-drive proof run (expected, not a leak).

### M2 fix-forward APPROVE: be2026a (services_converged completed-one-shot rule) @2026-06-11T00:31Z

Builder proposed a 2nd lasuite-drive P2b fix on branch `fix/converged-oneshot @ be2026a` and asked
approval before merging to main (M2 "trivial fix-forward w/ Adversary approval" path). Cold-verified
independently (fresh clone of be2026a at /root/adv-be2026a on cc-ci, NOT the Builder's working tree):

- **Diff** (`git diff origin/main..be2026a runner/harness/lifecycle.py`, read myself): in
  `services_converged`, a `cur != want` deficit now passes ONLY if `docker service ps <svc>` shows
  ALL task states == `Complete`. Conservative: any Running/Preparing/Pending (spinning up) or
  Failed/Rejected (broken) in the deficit still returns False; no-tasks-yet still False; plain N/N
  and 0/0 unchanged. Targeted addition, not a rewrite.
- **False-green analysis (my own):** only `restart_policy:none` one-shots ever show `Complete`; a
  normal crashed service shows Failed/Running(restarting), never Complete. Even if converge passed
  on a completed-but-ineffective one-shot, two INDEPENDENT gates still catch it — the generic
  `test_serving` HTTP floor and the custom-tier functional test (lasuite-drive
  `test_minio_storage.py` upload→list→download is the real bucket gate). Defense-in-depth holds; I
  could not construct a false-green path.
- **Tests** `tests/unit/test_converged_oneshot.py` (read + cold-ran): 7 cases pin exactly the
  non-vacuity criteria — completed→converged, Failed→NOT, mixed Complete+Failed→NOT (covers the
  `docker service ps` history concern), Preparing→NOT, no-tasks→NOT, N/N→converged, 0/0→converged.
- **Cold suite+lint from fresh be2026a checkout:** `cc-ci-run -m pytest tests/unit -q` → **199
  passed**; the 7 new tests pass alone; `nix develop .#lint --command scripts/lint.sh` → **lint:
  PASS**. Matches Builder's claim.
- **Root cause judged genuine P2b regression** (hook moved into ops.py pre_install runs BEFORE the
  install assert; the completed one-shot's 0/1 then burns DEPLOY_TIMEOUT in the converge poll). The
  fix accepts a genuinely-healthy deploy (HTTP 200, all other services 1/1) the old `cur!=want`
  wrongly rejected — correction, not masking.
- **Not on main** — confirmed `all(s == "Complete")` absent from origin/main; Builder held the gate.
- **Disclosed semantic delta** (a failing one-shot now blocks install convergence earlier vs later
  at custom-tier): ACCEPTED — both paths RED, no false-green, no enrolled recipe has a
  baseline-failing one-shot.

**VERDICT: fix-forward be2026a APPROVED, conditional on:**
1. Post-merge lasuite-drive proof re-run @ffa7d585afa2 PR=1 lands **L5** (binding end-to-end proof
   the fix resolves the converge hang — if it doesn't, the diagnosis was wrong and approval voids).
2. I re-verify the MERGED diff == be2026a diff (no extra change sneaks in at merge).
3. discourse PR=2 A/B pair (m2p-discourse / ab-discourse-7ae7b0f-oldmain — no one-shots, unaffected
   by this fix) completes and I cold-verify those levels too.
This APPROVE does NOT clear M2; M2 still needs all per-recipe levels reconciled + my independent
sample re-check + zero-leak teardown.

### be2026a merge cold-verify — condition #2 SATISFIED @2026-06-11T00:42Z

Builder merged be2026a as 6cabbe7 (build 350 green, origin/main now b4505ac). Independently checked:
`diff origin/main:runner/harness/lifecycle.py be2026a:...` → **IDENTICAL**; the merged
`tests/unit/test_converged_oneshot.py` → **IDENTICAL** to be2026a. Clean merge, no extra change
slipped in — approval condition #2 met. m2p-lasuite-drive (pre-fix) landed L0 (install/converge
timeout) = the diagnosed symptom (Builder disclosed b4505ac it SIGINT-shortcut the doomed burn;
binding proof is the post-fix m2p2 re-run). REMAINING be2026a conditions: #1 post-fix lasuite-drive
L5, #3 discourse PR=2 A/B cold-check — both pending (m2p-discourse running, then ab-oldmain, then
m2p2-lasuite-drive).

### be2026a conditions CLEARED + SSO-baseline staleness finding (independent) @2026-06-11T01:12Z

Reached the conclusions below COLD (own git archaeology + run-dir jq) BEFORE reading the Builder's
01:10Z inbox — which then concurred. Anti-anchoring preserved (no JOURNAL read; inbox read after my
own derivation).

**be2026a fix-forward — ALL 3 CONDITIONS SATISFIED → fix-forward FULLY CLEARED:**
1. **Post-fix lasuite-drive (m2p2, merged main 6cabbe7, ffa7d585afa2, PR=1): L4, rc=0, 3m19s.**
   Independently verified: flags clean_teardown=true + no_secret_leak=true; all 4 essential rungs
   pass; `test_minio_storage::...object_roundtrip` PASSED; `test_oidc_..._keycloak` PASSED. The
   install converge no longer hangs — both fix-forwards (1357544 best-effort poll + 6cabbe7
   completed-one-shot converge) exercised in one run. The literal "L5" in my condition is
   **unmeetable on current code and NOT an rcust effect** — see staleness finding below; I accept
   the L4-equivalence. Fix works end-to-end.
2. **Merged diff == branch diff** — verified earlier (4428e76): lifecycle.py + test file
   byte-identical to be2026a.
3. **discourse A/B — restructure-NEUTRAL.** m2p-discourse (NEW main, 7ae7b0f, PR=2) = L1 and
   ab-discourse-7ae7b0f-oldmain (OLD main, SAME ref, SAME PR=2) = L1, SAME stage (upgrade), SAME
   message (`eb96de94+U` HC1 re-checkout). old==new byte-identical → rcust did NOT regress discourse.
   The L4(184)→L1 vs baseline is pre-existing env drift since 06-05 (filed below), not rcust.

**FINDING [adversary] — M2 baseline matrix has 3 STALE L5 entries (lasuite-docs/drive/meet).**
Independently established: the level ladder dropped 6-rung(L5)→4-rung(max L4, integration &
recipe-local now OPTIONAL/non-laddered) in mainline PR#6 (c51cd84 "4-rung ladder", + 46e2cdb),
which `git merge-base --is-ancestor c51cd84 01e6d49^` confirms is an ANCESTOR OF PRE-RCUST MAIN.
The rcust merge touches level.py NOT AT ALL and results.py by +4 cosmetic P5 lines; compute_level
+ derive_rungs are byte-identical old-main↔merged-main. So NO current-code run (rcust or pre-rcust)
can produce L5; baselines 188/189/204 (L5, integration:pass) were recorded under the OLD schema
(run 204 ran 06-09 hours before the refactor deployed). **rcust is INNOCENT of L4≠L5.** Integration
coverage is NOT lost: the requires_deps OIDC tests EXECUTE and PASS (skip-count 0) on current code —
verified in m2p2 AND the sweep's m2r-lasuite-docs (`test_oidc_login_via_keycloak` +
`test_oidc_password_grant_...` PASSED) and m2r-lasuite-meet (`...password_grant...` PASSED).
ACCEPTED equivalence for the M2 matrix: **old L5 ≡ new L4 (all 4 essential rungs pass) + requires_deps
OIDC test PASSED (skip-count 0)**. Under this, lasuite-docs (m2r L4) / lasuite-meet (m2r L4) /
lasuite-drive (m2p2 L4) all MATCH. (Note: this validates — but corrects the basis of — the Builder's
first-sweep "lasuite-docs/meet matched baseline"; they are L4+OIDC, not numeric L5.) This is a
matrix-staleness correction, NOT a rcust regression; no VETO.

**Still OPEN for the M2 verdict (my side):** (a) per-recipe levels reconciled vs the CORRECTED
baseline for all 21; (b) bluesky-pds is L0 on BOTH old & new main (upstream image
`Cannot find module index.js`) — restructure-neutral but also cannot match its L4-equiv baseline on
ANY current run → needs a DECISIONS/DEFERRED note as non-rcust upstream breakage, not a silent
mismatch; (c) the 2 drone-path !testme runs (immich#2/plausible#3); (d) zero-leak teardown sweep;
(e) my own independent re-check of ≥5 recipes' logs + ALL mismatches before any M2 PASS.

---

## M2 — merged-main real-CI regression sweep: **PASS** @2026-06-11T01:15Z

Cold-verified the M2 claim (STATUS gate "M2 CLAIMED ~01:30Z") from my own clone + direct on cc-ci,
re-running/ re-parsing rather than trusting Builder logs. Every M2.0–M2.4 item holds.

**M2.2 canaries — cold RE-RAN myself** from a fresh `origin/main` checkout (/root/adv-be2026a @
origin/main): `cc-ci-run -m pytest tests/regression/ -m canary -v` → **7/7 passed (301s)**, incl.
`bad-false-green` (the false-green detector) + all four RED canaries (bad-install/upgrade/backup/
restore) caught at their designed tier. The level system is NOT inflating. (log /root/adv-canary.log)

**M2.3 per-recipe — all 21 reconciled (cold jq on each run dir):**
- 13 clean: cryptpad/custom-html/ghost/hedgedoc/keycloak/matrix-synapse/n8n/uptime-kuma = L4;
  mailu/custom-html-tiny = L2 (backup_restore N/A); mumble = L4 (deploy-count=1) — all == baseline,
  clean_teardown=true.
- 2 designed-bad canaries genuinely exercised: bkp-bad rungs backup_restore=**fail** (backup=fail);
  rst-bad backup_restore=**fail** (backup=pass→restore=fail). The L1 cap is upgrade-N/A ladder
  semantics; the designed failure is recorded in the rung (verified — NOT a coincidental
  level-match).
- immich/mattermost-lts/plausible: **L4 @ exact baseline refs** (m2b-*) — baseline REPRODUCED on the
  restructured harness (cold-verified earlier this session).
- discourse: m2p-discourse (NEW main) == ab-discourse-7ae7b0f-oldmain (OLD main) — SAME ref/PR=2,
  SAME stage, SAME upgrade-HC1 message (`eb96de94+U`), SAME L1. **old==new ⇒ rcust-neutral**; the
  L4(184)→L1 is pre-existing env drift since 06-05 (DEFERRED.md), NOT caused by the restructure.
- lasuite-docs/-meet/-drive: L4 all-rungs-pass + requires_deps OIDC test PASSED (skip-count 0)
  [lasuite-drive m2p2 also MinIO PASSED, post-both-fixes, rc=0]. Their "L5" baselines are STALE:
  the 6→4-rung ladder landed in mainline c51cd84 (PR#6), which `git merge-base --is-ancestor
  c51cd84 01e6d49^` confirms PREDATES the rcust merge; level.py untouched by the merge, derive_rungs
  byte-identical old↔new. **rcust-innocent; integration coverage preserved** (OIDC tests execute &
  pass). Accepted equivalence old L5 ≡ new L4-all-pass + OIDC-pass.
- bluesky-pds: EXCLUDED — `Cannot find module /app/index.js` crash-loop on BOTH old & new main at
  every ref → upstream image breakage, rcust-neutral. DEFERRED.md note present.

**M2.3 drone→harness path:** drone builds **356 (immich) + 357 (plausible)** = `build_event=custom`
(bridge-triggered; distinct from push builds 358-361), trigger=autonomic-bot, both **success**
(verified in drone sqlite DB); run dirs 356/357 = immich L4 pr=2 / plausible L4 pr=3, customization
manifest present, clean_teardown=true.

**M2.4 customizations actually executed (cold-grep):** manifest block **21/21** logs; mumble
`ready-probe OK (tcp 3x) 127.0.0.1:64738`; ghost `ccci-overlay: provided compose.ccci.yml ...
base deploy auto-chaos` (P2a first-class path live); cryptpad `EXTRA_ENV='<hook>'`; immich
`ops.py[pre_backup,pre_restore,pre_upgrade]` + `pre-op seed` lines (migrated ctx hooks run).

**Teardown:** `docker stack ls` = infra (backups/bridge/dashboard/reports/drone/traefik) +
warm-keycloak ONLY, **zero leaked app stacks** (checked after ALL runs incl. drone-path).

**Fix-forwards (both Adversary-approved, additive):** 1357544 (lasuite-drive best-effort poll, appr
57c66ad) + be2026a/6cabbe7 (services_converged completed-one-shot, appr a531746) — merged diff ==
branch diff, all 3 be2026a conditions cleared (24a203a). Cold unit suite on post-fix main = 199
passed, lint PASS.

**VERDICT: M2 PASS.** No regression CAUSED BY the restructure: every deviation from the baseline
matrix is proven rcust-neutral by same-ref old-vs-new A/B (discourse, bluesky) or is a pre-rcust
stale-schema artifact with coverage preserved (3 lasuite), all documented in DEFERRED.md — not a
silent mismatch. The false-green detector is green on my own cold canary run. No findings filed,
no VETO.

**M1 PASS (01f9f70) + M2 PASS (this entry) both stand** → the phase DoD handshake is satisfied; the
Builder may write `## DONE` to STATUS-rcust.md. (M1's unit+lint acceptance still holds on post-fix
main: 199 passed / lint PASS, the fix-forwards being additive + separately approved.)