cc-ci/machine-docs/JOURNAL-rcust.md

# JOURNAL — sub-phase rcust (Builder)

## 2026-06-10 bootstrap

Read phase plan (recipe-custom-restructure-full-plan.md), plan.md §6.1/§7/§9, and the reference
spec docs/recipe-customization.md @ 76a4b6b in full. Created phase state files. Work branch will
be `restructure/recipe-custom` off main @ 76a4b6b. Starting P1: reading the six current loaders
(run_recipe_ci.py::_load_meta, conftest.py::_recipe_meta, lifecycle.py::_recipe_extra_env,
lifecycle.py::_recipe_meta_flag, deps.py::declared_deps, canonical.py::is_canonical_enrolled)
before writing harness/meta.py.

## 2026-06-10 P1 — single loader + registry (branch 472a68b)

Wrote runner/harness/meta.py: KEYS registry (14 keys + CHAOS_BASE_DEPLOY/OIDC_AT_INSTALL/
SKIP_GENERIC kept registered as deprecated=True so P1 lands green before P2 deletes them),
RecipeMeta generated from KEYS via dataclasses.make_dataclass (frozen; field set cannot drift from
the registry), load() = the only exec() of recipe_meta.py, MetaError on unknown ALL-CAPS/type
mismatch/callable-on-data-key, difflib suggestion in the unknown-key message. BACKUP_CAPABLE keeps
its tri-state via default None (None = auto-detect — preserves the old `"BACKUP_CAPABLE" in meta`
semantics in generic.backup_capable).

Migrations: orchestrator loads once + passes meta down (deploy_app/perform_upgrade/_perform_op/
run_lifecycle_tier all take the object); conftest meta fixture returns full RecipeMeta (R3 closed);
lifecycle._recipe_extra_env/_recipe_meta_flag and deps.declared_deps deleted; canonical.is_enrolled
+ enrolled_recipes go through meta.load (tests monkeypatch meta.TESTS_DIR now instead of
canonical.__file__); screenshot._load_screenshot_hook reads the attribute (R2 fixed — unit test
proves SCREENSHOT survives the real orchestrator load path). deploy_app keeps an optional
meta=None fallback (loads via the single loader) for fixture/manual callers — exec still happens
in exactly one function.

Effective-value safety check before committing: dumped non_default() for all 21 recipe dirs through
the new loader — every recipe's customized key set matches its recipe_meta.py source (e.g. mumble:
DEPLOY_TIMEOUT/EXTRA_ENV/HEALTH_OK/READY_PROBE/UPGRADE_EXTRA_ENV). One intentional delta class:
deps.deploy_deps' fallback timeouts for a MISSING dep meta change from literal 900/600 to loading
the dep's real meta (orchestrator path always supplied metas, so CI behavior is identical).

Verified on cc-ci (rsynced working tree before committing):
  cc-ci-run -m pytest tests/unit -q  -> 175 passed
  nix develop .#lint --command scripts/lint.sh -> lint: PASS
Three pre-existing f212 unit tests passed dicts to wait_ready_probes — updated mechanically to
construct RecipeMeta via dataclasses.replace (assertions untouched).

Next: P2a compose.ccci.yml first-class + auto-chaos.

## 2026-06-10 P2 — legacy keys & paths deleted (branch 8cd72fd)

P2a: lifecycle.provide_ccci_overlay copies tests/<recipe>/compose.ccci.yml into the per-run
checkout (after install_steps hook, before prepull/deploy); pinned base deploys auto-chaos on
overlay presence (has_ccci_overlay replaces the meta.CHAOS_BASE_DEPLOY elif). ghost/discourse
install_steps.sh were copy-only -> deleted whole; their metas keep COMPOSE_FILE in EXTRA_ENV
(unchanged wiring, the harness now owns the copy).

P2b: oidc_at_install condition removed — `if declared:` provisions before the single deploy,
legacy post-deploy block + _run_setup_custom_tests_hook deleted. lasuite-docs install_steps.sh is
the meet/drive hook with docs' exact env names (diffed against the deleted setup_custom_tests.sh:
same keys incl. OIDC_OP_DISCOVERY_ENDPOINT + scopes 'openid email profile'; secret-insert bump
identical; only the abra-redeploy step is gone — the single deploy reads the env instead).
lasuite-drive's MinIO bucket one-shot -> ops.py pre_install (runs at install-tier start, post-
deploy; bucket lives in the minio volume so it survives upgrade/restore; same scale --detach +
30x3s poll as the shell version). run_quick: deps still provision (realm/creds), hook call gone —
no quick-enrolled recipe declares DEPS today; noted inline.

P2c: SKIP_GENERIC out of the registry; _skip_generic(op) env-only; skip_generic_env_overrides()
prints a `!!` warning when active under DRONE (P5 will embed in the manifest).

P2d: conftest deps fixture = dict of _DepEntry (dict subclass w/ attribute sugar) — the 6 lasuite
files only ever used deps_creds, renamed param to deps, zero assertion changes. NOTE for Adversary:
some assert MESSAGE strings ('setup_custom_tests should have populated this.' -> 'dep
provisioning...') and docstrings updated — message text only, no assert logic/expected values.

Verified on cc-ci (rsync of working tree): cc-ci-run -m pytest tests/unit -q -> 175 passed;
nix develop .#lint --command scripts/lint.sh -> PASS. Doc table regenerated to the 14-key registry
(doc-sync unit test pins it).

Next: P3 — HookCtx + ctx-hook signatures everywhere.

## 2026-06-10 P3 — uniform ctx hook convention (branch fd02d9f)

HookCtx frozen dataclass + hook_ctx() constructor in harness/meta.py; ctx.deps read straight from
$CCCI_DEPS_FILE (json, both shapes) — meta.py stays import-cycle-free (deps.py imports lifecycle
which imports meta). Registry keys carry hook_params; meta.load() enforces the expected positional
names per hook key (READY_PROBE/BACKUP_VERIFY/EXTRA_ENV/UPGRADE_EXTRA_ENV=(ctx,),
SCREENSHOT=(page, ctx)); _run_pre_hook applies meta.check_hook_signature(fn, ("ctx",)) to ops.py
hooks before calling. Conversion of 17 ops.py + 8 recipe_meta hooks was scripted (def-line regex +
bare `domain` -> `ctx.domain` inside the pre_*/hook function bodies only) and diff-reviewed; the
only manual fixes: keycloak pre_restore passed `meta` -> `ctx.meta`, and two comment lines in
lasuite-drive/-meet metas that the regex over-replaced were restored. wait_ready_probes gained
op= (install/upgrade call sites pass it) so probes can know the phase.

Verified on cc-ci: cc-ci-run -m pytest tests/unit -q -> 180 passed; lint PASS.

Next: P4 — discovery placement rule + op_state/deps fixtures + migrate hand-parsers.

## 2026-06-10 P4 — custom-test ergonomics (branch 29a28e2)

Pre-change sweeps confirmed the plan's zero-users claims: no top-level non-lifecycle test_*.py in
any recipe dir; no recipe test file reads os.environ / CCCI_OP_STATE_FILE directly (the only
op-state consumers are the generic assertions via harness.generic.op_state — harness-side, fine).
So P4 = discovery glob removal + new op_state fixture + pinning tests; no test migrations needed.
test_discovery.py's HC2 gate test moved its repo-local custom fixture under functional/ (the rule);
test_discovery_phase2.py now asserts top-level custom is NOT discovered. op_state fixture skips
(clear reason) when env unset / file missing / unparseable; tested via request.getfixturevalue.

Verified on cc-ci: cc-ci-run -m pytest tests/unit -q -> 184 passed; lint PASS.

Next: P5 — customization manifest (print block + results.json key).

## 2026-06-10 P5 — customization manifest (branch 68954be)

(Resumed after a usage-limit pause mid-P5; working tree carried the in-flight manifest.py.)
New runner/harness/manifest.py: build() collects {meta_non_default, hooks, overlays, custom_tests,
env_overrides} via the SAME discovery/meta functions the run uses (so the manifest can never
disagree with what actually executes — incl. the HC2 _gated() repo-local gate), render() prints
the block. Orchestrator builds+prints right after meta load / repo-local snapshot, BEFORE the
quick-lane branch (both lanes get the block); the dict rides into build_results(customization=...)
verbatim. run_quick writes no results.json, so the single build_results call site covers all.
Hooks render as "<hook>", tuples as lists (JSON-clean); ops.py pre-ops listed by cheap source
scan (same approach as discovery._module_defines — no import at manifest time).

Lint flagged: C408 dict() literal, import-block order (manifest after deps), ruff-format on the
new test file — all fixed. Verified on cc-ci (rsync of working tree): cc-ci-run -m pytest
tests/unit -q -> 191 passed; nix develop .#lint --command scripts/lint.sh -> lint: PASS.

Next: P6 docs, then M1 prep (tests/concurrency proof run + 21-recipe baseline matrix).

## 2026-06-10 P6 — docs (branch da558ca) + inbox response (858e0f5)

Rewrote the three docs to the restructured end state; kept the generated §4 table byte-identical
(doc-sync test pins it). recipe-customization.md flipped from review spec to reference; §8 is now
the R1–R9 resolution ledger. Facts double-checked against code before writing: R2 proof lives in
test_screenshot.py::test_screenshot_reachable_through_real_load_path (not test_meta.py — fixed a
first-draft error); mumble's post-F2-14c shape has NO install_steps.sh/CHAOS_BASE_DEPLOY (base =
mumbleweb-only COMPOSE_FILE, host-ports added at head via UPGRADE_EXTRA_ENV); lasuite-docs now
ships install_steps.sh (P2b migration); deps file shape is dict recipe->entry; custom_tests
discovery is NON-recursive over functional/+playwright/ (old doc said recursive — corrected).

Adversary inbox (19:06Z, non-blocking): manifest dumps meta values verbatim -> dashboard shows a
field named SECRET_KEY_BASE (plausible's committed CI dummy — public, no real leak). Took the
redaction option: _jsonable masks values whose key NAME matches
SECRET|PASSWORD|TOKEN|CREDENTIAL|word-segment-KEY, recursing into dict values (the plausible case
is a NESTED key under EXTRA_ENV); names stay visible. KEYCLOAK_URL deliberately not matched
(word-segment KEY). Unit test pins redacted+passthrough both.

Verified on cc-ci (rsync of working tree): cc-ci-run -m pytest tests/unit -q -> 192 passed;
nix develop .#lint --command scripts/lint.sh -> lint: PASS.

Next: M1 prep — tests/concurrency proof run on the branch + the 21-dir baseline matrix.

## 2026-06-10 M1 prep + claim

Concurrency proof run on branch head 858e0f5 (rsynced tree on cc-ci): cc-ci-run -m pytest
tests/concurrency -q -> 23 passed in 11.46s (suite untouched by the restructure, as planned).

Baseline matrix: pulled every /var/lib/cc-ci-runs/*/results.json (141 files) and took the most
recent per recipe. 19/21 dirs covered by results.json; mumble's last full run predates the
results system (log ~/ccci-mumble-f214c.log, 5 tiers pass 05-31); bluesky-pds likewise
(Adversary Phase-2 cold verify e45e0ee). plausible's weekly-report RED was its PR branch
(pg13->14, build 200); its default-branch baseline is run 308 (06-10) L4 — runs 307/308 are
today's, from the conc-phase M2 sweep. Bad canaries recorded at their designed-fail tier.

Claimed M1. While waiting: nothing else unblocked in this phase (M2 is gated on M1) — will hold
with short fallback polls per §7 case 2.

## 2026-06-11 M2 reconciliation — discourse upgrade-HC1 root-cause hunt + bluesky re-characterization

Resumed after a loop stall (~21:18Z–23:50Z): the m2b/ab sweeps had finished but nothing processed
them. Adversary's 23:53Z inbox asked for (1) a same-ref A/B for the m2b-discourse upgrade-HC1 L1
and (2) a fresh post-fix lasuite-drive L5 at baseline ref — both now queued/running.

Discourse dig (why I don't yet have a mechanism): first hypothesis was my own invocation error —
m2b ran PR=0 where baseline 184 ran PR=2, and I guessed the PR-head sha was unreachable without
the PR fetch. WRONG: fetch_recipe clones all mirror branches and `git checkout <sha>` is check=True
— and the preserved per-run clone sits at HEAD=7ae7b0f, so the re-checkout ran AND persisted.
Second hypothesis (prepull resets the checkout): also wrong — prepull_images is pure
`docker compose config --images` in cwd, never touches git. The scary
`service "sidekiq" depends on undefined service "discourse"` line turned out benign: it appears in
the PASSING m2r/m2rr upgrade sections verbatim (the published compose ships a dangling depends_on;
swarm ignores it — documented in the overlay NOTE). What's left: abra stamped the PREV-TAG commit
(eb96de94 = 0.7.0+3.3.1) on the chaos redeploy while the tree was at 7ae7b0f. One live hypothesis:
the cc-ci overlay clamps app+sidekiq images to bitnamilegacy/discourse:3.3.1; at this PR head
(0.9.0+3.5.0 bump) the redeploy spec may end up close enough to the base spec that the label
update path degenerates — but that requires abra-internals knowledge I can't verify analytically,
and m2r at 7d53d4ec (which also post-dates the 3.5.0 bump?) stamped correctly with the same
overlay, so content-difference-between-refs is doing SOMETHING. Decision: stop theorizing, let the
2x2 complete — m2p-discourse (new main, PR=2, @7ae7b0f) distinguishes PR=0-artifact/race from
deterministic; ab-discourse-7ae7b0f-oldmain (old main, PR=2, @7ae7b0f) distinguishes regression
from pre-existing. Run 184 left no orchestrator log (drone-side), so its chaos stamp is unknowable
— the old-main re-run stands in for it.

lifecycle.py diff c2508c7..main re-read for the upgrade path: overlay copy moved from per-recipe
install_steps.sh to first-class auto-chaos (P2a) but the copied FILE and its untracked-persistence
semantics are byte-identical; run_upgrade order (checkout → upgrade_env → prepull → chaos
redeploy -c → own wait_healthy) unchanged from old main. Nothing jumps out as the delta.

bluesky-pds: pulled the swarm service logs from all three failed runs — identical
`Cannot find module '/app/index.js'` crash-loop (Node v24.15.0) on new main @ mirror head, new
main serial re-run, AND old main @ old default head. The earlier "deploy timed out during
concurrent image pulls" guess in STATUS was wrong (the 600s timeout was the SYMPTOM; the ~2min
A/B failure exposed the crash-loop). Upstream re-published the pinned tag with a different image
layout — no harness can deploy it. Filed in STATUS as restructure-neutral with grep-able evidence.

## 2026-06-11 lasuite-drive root cause #2 — completed one-shot poisons convergence (caught live)

Watching the m2p proof run instead of just waiting paid off: the fix-forward's best-effort line
printed (so #1 is fixed), but the install assert then sat in pytest for 25+ minutes. Live state:
app serving 200, every service 1/1 EXCEPT minio-createbuckets 0/1 with its task **Complete 28
minutes ago**. services_converged demands cur==want for every service; a completed
restart_policy-none one-shot never returns to 1/1, so the bounded converge poll (DEPLOY_TIMEOUT
1800s for this recipe) was always going to burn to the deadline and fail install.

Why nobody ever saw this before P2b: the old setup_custom_tests.sh ran AFTER the install asserts
(post-deploy hook path), so converge never observed desired=1 on the one-shot, and the upgrade
tier's chaos redeploy reapplied the compose spec (replicas: 0) before its own converge checks.
P2b folded the trigger into ops.py pre_install — which the orchestrator runs BEFORE the generic
install assert. Also explains m2rr's odd "install fail but upgrade/backup/restore/custom all pass"
shape exactly (redeploy resets the spec).

Fix options weighed: (a) hook scales the one-shot back to 0 after the poll — rejected: on the
timeout path the task is typically still Preparing (image pull) and scale-to-0 CANCELS it, so the
observed "bucket lands just after the window" runs would become custom-tier RED, i.e. strictly
worse than baseline; (b) move the trigger to a post-assert hook point — no such hook exists in the
new convention and inventing one mid-M2 is scope creep; (c) teach services_converged that a
replica deficit consisting entirely of Complete tasks IS converged — chosen: semantically correct
(the one-shot did its job), restores baseline behavior for any triggered one-shot, and the
converge window doubles as the late-landing grace. Disclosed delta: a genuinely FAILING one-shot
now reds at install (converge timeout) instead of at the custom bucket test — both red, no false
green. Guard: Failed/mixed/spinning-up/no-tasks-yet still block (unit-pinned, 7 cases).

Branch fix/converged-oneshot @ be2026a, proposal in ADVERSARY-INBOX, awaiting approval per the M2
fix-forward protocol. Unit suite 199 passed + lint PASS from the cc-ci working-tree rsync.

## 2026-06-11 ~01:00Z — merge landed, queue shortened

be2026a approved (REVIEW a531746, cold-verified independently) and merged as 6cabbe7; drone build
350 green on the push head 914c166. Merged diff verified == branch diff (empty git diff be2026a..
main for the two files). Post-fix proof m2p2-lasuite-drive queued from a FRESH clone
/root/m2-postfix @6cabbe7 rather than git-updating /root/m2-sweep, because the serial queue's
discourse runs exec from m2-sweep and swapping code under an active/imminent run is how you get
unexplainable results. The discourse A/B therefore runs at 5c0676b (pre-converge-fix) — irrelevant
to discourse (no one-shots), and the Adversary's approval explicitly noted that.

Shortened the doomed m2p run: the generic install assert had already burned its 1800s converge
deadline and failed; the overlay install test then started an IDENTICAL second 1800s burn (same
assert_serving). SIGINT'd the overlay pytest child only — KeyboardInterrupt surfaced at
generic.py:97, the exact diagnosed converge-poll line (a nice live confirmation), and the
orchestrator advanced to the upgrade tier on its normal path. Teardown semantics untouched.
Disclosed in STATUS so the log's KeyboardInterrupt is pre-explained.

Drone API note for future me: no token on disk; fastest read-only check is docker cp the drone
sqlite out and query builds (documented in STATUS). The Gitea statuses API returned empty for
these shas (drone evidently doesn't post commit statuses here).

## 2026-06-11 ~00:55Z — discourse A/B closed (harness-neutral), mechanism still unattributed

m2p-discourse (new main, PR=2, @7ae7b0f) and ab-discourse-7ae7b0f-oldmain (old main, PR=2, same
ref) failed the upgrade IDENTICALLY: HC1, chaos-version=eb96de94+U, all other tiers pass, L2.
Same invocation as baseline 184 which was L4 five days ago. So: deterministic, harness-neutral,
and something outside both harnesses drifted since 06-05. Eliminated: branch-tip existence (7ae7b0f
still tips upgrade-0.8.0+3.5.0 + pr/2), upstream tag set (0.7.0+3.3.1 still latest), abra pin
(flake.lock untouched by the restructure). Not eliminated: abra-internal interaction with repo/app
state (the chaos stamp lands on the prev-base TAG commit despite the tree being at the PR head —
my best guess remains something in how abra resolves the version/commit for the chaos label when
COMPOSE_FILE includes the overlay and the project normalizes invalid, but m2r at 7d53d4ec stamping
correctly with the same dangling depends_on kills the simple version of that theory). The
`service "sidekiq" depends on...` line appears in passing AND failing upgrades, position-identical,
so it discriminates nothing. M2-wise the question is settled — the restructure is exonerated by
byte-identical old==new failure; chasing abra's stamp resolution further is post-phase work, filed
as a DEFERRED note rather than burning more M2 wall-clock on a non-rcust mechanism.

m2p2-lasuite-drive (the binding post-fix proof) auto-started at 00:48:58Z from /root/m2-postfix
@6cabbe7. Watching for: no 1800s converge burn after the one-shot completes, then L5.

## 2026-06-11 ~01:10Z — m2p2 green; "L5" turned out to be a moved goalpost (mainline, not ours)

m2p2-lasuite-drive: rc=0, 3m19s, all stages pass, OIDC + MinIO custom tests green, and the
fix-forward pair demonstrably exercised (one-shot overshot 90s again → best-effort line → late
Complete → converge fix admitted it). But results.json said level=4 where the binding condition
said L5 — heart-stopper until the git archaeology: run 189's level-5 + "L6 recipe-local N/A" cap
didn't match ANY derive_rungs I could find in either world, because the 6-rung ladder was removed
on MAIN by 46e2cdb+c51cd84 (PR #6) on 06-09, between the baseline runs and the merge — by the
mirror/report phase, not rcust. The merge didn't touch level.py (checked 01e6d49^1..01e6d49), and
run 204 on 06-09 (hours pre-deploy of the refactor) still shows 6 rungs — clean timeline. So the
baseline matrix's "L5" rows need a schema-equivalence reading, declared in STATUS BEFORE the claim
rather than negotiated after the Adversary trips on it. Lesson re-learned: a baseline matrix
should pin the SCHEMA VERSION of its evidence, not just the level number.

## 2026-06-11 ~01:30Z — M2 claim assembled

Drone-path runs landed green (356 immich#2 L4, 357 plausible#3 L4, both with embedded
customization manifests + clean flags, triggered by real !testme comments). Zero-leak verified
after everything. Plausible's missing screenshot.png checked against its other runs — it never
produces one (no screenshot surface), so not a capture regression. Claimed M2 with the full
21-recipe reconciliation table against the corrected baseline; the three lasuite rows ride the
Adversary-accepted L5≡L4+OIDC equivalence, bluesky-pds is the one justified exclusion, discourse
is reconciled as env-drift with byte-identical old==new evidence. Nothing else unblocked in this
phase while the verdict is out — holding per §7 case 2.

## 2026-06-11 ~01:20Z — M2 PASS → ## DONE

Adversary cold-verified the whole claim independently (re-ran the canaries themselves, jq'd all 21
run dirs, re-checked the drone DB and the zero-leak state) and passed M2 with no findings and no
VETO. M1 + M2 both stand; ## DONE written. Phase summary: 6 plan phases landed on one branch,
merged after M1; the real-CI sweep then caught exactly TWO genuine regressions (both in the same
lasuite-drive P2b hook port: raise-on-timeout, and one-shot-vs-converge ordering), both root-caused
live, fixed forward under approval, and proven end-to-end — plus it surfaced two pre-existing
environment drifts (discourse upgrade-HC1, bluesky-pds upstream image) that the A/B discipline
kept from being misattributed to the restructure. The sweep-as-safety-net worked as designed.