cc-ci-orchestrator/cc-ci-plan/recipe-custom-restructure-full-plan.md

Recipe-customization restructure — full implementation plan

Operator-approved direction (chat 2026-06-10). Reference spec of the CURRENT system: cc-ci docs/recipe-customization.md (commit 76a4b6b) — read it FIRST; its §8 R1–R9 are the defects this plan fixes. Goals: one coherent customization system, one place/way to configure a recipe, easy custom tests, fix broken knobs, delete legacy paths, zero behavior regression for currently-working recipes.

Decisions locked (operator + orchestrator)

Question	Decision
Loaders	ONE loader runner/harness/meta.py::load(recipe) -> RecipeMeta backed by a declarative KEY registry. All six current loaders (spec §4 L1–L6) migrate to it; exec() of recipe_meta.py happens in exactly one function.
Validation strictness	Unknown ALL-CAPS top-level name or type mismatch = hard error (run fails fast at load; unit test catches at PR time). Underscore-prefixed names (_FOO) are recipe-private and exempt.
SCREENSHOT	KEEP and FIX (it is currently dead — spec §8 R2). Reachable once the registry replaces the allowlist.
CHAOS_BASE_DEPLOY	DELETE. tests/<recipe>/compose.ccci.yml becomes first-class: harness copies it into the checkout + auto-uses --chaos for the base deploy. Both users (ghost, discourse) use the flag only for this.
OIDC_AT_INSTALL	DELETE. Install-time deps provisioning becomes the ONLY mode when DEPS is set; legacy post-deploy provisioning + setup_custom_tests.sh redeploy path removed. Migrate lasuite-docs.
SKIP_GENERIC (meta key)	DELETE (zero users). Env form CCCI_SKIP_GENERIC* stays as a documented LOCAL-DEV-ONLY escape hatch, loudly surfaced in the run manifest.
Hook signatures	All recipe callables take a single ctx (HookCtx) — EXTRA_ENV(ctx), UPGRADE_EXTRA_ENV(ctx), READY_PROBE(ctx), BACKUP_VERIFY(ctx), SCREENSHOT(page, ctx), pre_<op>(ctx). All users are in-repo; migrate them, no compat shim.
Fixtures	Single tests/conftest.py. Final surface: recipe, meta (full RecipeMeta), live_app, op_state (NEW), deps (NEW — consolidates deps_apps+deps_creds). DELETE dead pre-deploy-once fixtures deployed/deployed_app (zero users) and app_domain if nothing else uses it (builder: grep).
Custom-test placement	test_<op>.py top-level = lifecycle overlay; ALL custom tests under functional/ or playwright/. Discovery of top-level non-lifecycle test_*.py in recipe dirs is REMOVED (zero users today).
Docs	Key reference table is GENERATED from the registry; a unit test asserts docs ⊆ registry sync.
recipe_meta stays Python	NO data/hooks file split (spec §8 R9 considered, rejected — registry validation gets the value at lower cost). One file per recipe remains the single config place.
Landing	one branch restructure/recipe-custom, one commit per phase, merged to main only after Adversary M1 PASS + lint green; real-CI regression sweep (M2) after merge.

Target shape (end state)

tests/<recipe>/
├── recipe_meta.py        # THE config: registry-validated keys + ctx-hooks (+ _private consts)
├── test_<op>.py          # lifecycle overlay asserts (op_state/live_app/meta fixtures)
├── ops.py                # pre_<op>(ctx) seed hooks
├── functional/  playwright/   # ALL custom tests
├── install_steps.sh      # pre-deploy shell hook (the only shell hook)
├── compose.ccci.yml      # first-class CI overlay (auto-copied, auto-chaos)
└── PARITY.md

One loader, one hook convention, one fixture file, one shell hook, one generated reference.

---

P1 — harness/meta.py: single loader + key registry

1. New runner/harness/meta.py:
   • KEYS: registry of every key — name, type ("int"|"str"|"tuple[int]"|"bool"|"dict_or_hook"| "hook"|"list[str]"|"dict"), default, doc (one line each), optional validate(value). Final key set (14): HEALTH_PATH, HEALTH_OK, DEPLOY_TIMEOUT, HTTP_TIMEOUT, BACKUP_CAPABLE, EXPECTED_NA, READY_PROBE, UPGRADE_BASE_VERSION, BACKUP_VERIFY, UPGRADE_EXTRA_ENV, EXTRA_ENV, DEPS, WARM_CANONICAL, SCREENSHOT. (CHAOS_BASE_DEPLOY / OIDC_AT_INSTALL / SKIP_GENERIC are deleted in P2 — during P1 keep them registered with deprecated=True so P1 lands green before P2 removes them.)
   • load(recipe) -> RecipeMeta (frozen dataclass, attribute access): the ONLY exec() of tests/<recipe>/recipe_meta.py. Missing file → all defaults. Validation per the locked decision: unknown non-underscore ALL-CAPS name → MetaError listing the unknown name and nearest registered key; type mismatch → MetaError. Callables accepted only for hook-typed keys.
2. Migrate ALL readers (spec §4 L1–L6 + §9 index has exact locations):
   • run_recipe_ci.py::_load_meta → meta.load(); orchestrator loads ONCE and passes the object down (functions grow a meta param; stop re-exec'ing per call).
   • tests/conftest.py::_recipe_meta → meta.load() (fixture now returns full RecipeMeta).
   • lifecycle.py::_recipe_extra_env, lifecycle.py::_recipe_meta_flag → deleted; callers take meta.
   • deps.py::declared_deps → meta.DEPS; canonical.py::is_canonical_enrolled → meta.WARM_CANONICAL.
   • screenshot.py consumer unchanged — but now the dict/object it receives actually contains SCREENSHOT. This is the R2 fix; prove it with a unit test (meta with SCREENSHOT hook → _load_screenshot_hook returns it through the real orchestrator load path).
3. Mumble private constants WELCOME_TEXT_MARKER, MAX_USERS → _WELCOME_TEXT_MARKER, _MAX_USERS (fix their importers; builder: grep how mumble tests consume them).
4. New unit tests tests/unit/test_meta.py:
   • every tests/*/recipe_meta.py in the repo loads clean through the registry (the typo gate);
   • unknown-key and wrong-type files (tmp fixtures) raise MetaError;
   • defaults match spec §2 baseline; underscore exemption; callable-on-data-key rejected.
5. Doc generation: scripts/gen-meta-docs.py renders the registry to a markdown table between <!-- META-TABLE-START/END --> markers in docs/recipe-customization.md §4; unit test asserts the committed table == rendered output (drift fails CI).

P2 — Delete legacy keys & paths

a. compose.ccci.yml first-class. In the deploy path (after install_steps.sh, before base deploy): if tests/<recipe>/compose.ccci.yml exists, copy it into the recipe checkout (ABRA_DIR-aware) and use --chaos for the base deploy. Remove the copy boilerplate from ghost/discourse install_steps.sh (delete the hook file entirely if copying was all it did — builder: read both) and CHAOS_BASE_DEPLOY = True from both metas. Delete _recipe_meta_flag remnants. Keep the policy note (overlays = minimal justified fallback) in docs. b. Install-time deps only. Make the OIDC_AT_INSTALL=True code path the unconditional behavior for recipes with DEPS; delete the legacy post-deploy provisioning branch, the setup_custom_tests.sh invocation machinery, and the deploy-count exception for the legacy redeploy. Migrate lasuite-docs to install-time wiring (mirror what lasuite-drive/meet do in install_steps.sh reading $CCCI_DEPS_FILE). NOTE: lasuite-drive ALSO ships setup_custom_tests.sh despite OIDC_AT_INSTALL=True — builder must read both scripts and determine what they still do (realm/user provisioning is harness-side via harness.sso; env wiring belongs in install_steps.sh). Whatever remains necessary moves into install_steps.sh; then delete both setup_custom_tests.sh files and the key. c. SKIP_GENERIC meta key deleted; env CCCI_SKIP_GENERIC* documented dev-only; if set in a CI (drone) run, print a loud !! warning + record in the P5 manifest. d. Conftest cleanup: delete deployed, deployed_app (dead, zero users — verified), and app_domain if unused after their removal. Consolidate deps_apps + deps_creds into one deps fixture (entries expose .domain plus full creds; dict-style access fine). Migrate the 6 lasuite test files that use the old pair. Keep requires_deps marker + skip-report plumbing unchanged (F2-11 gate — do NOT weaken).

P3 — Uniform ctx hook convention

1. harness/meta.py::HookCtx (frozen dataclass): .domain, .base_url, .meta (RecipeMeta), .deps (creds dict or None), .op (current lifecycle op or None). One constructor helper in the orchestrator; harness builds it at each hook call site.
2. Convert call sites: EXTRA_ENV(ctx), UPGRADE_EXTRA_ENV(ctx), READY_PROBE(ctx), BACKUP_VERIFY(ctx), SCREENSHOT(page, ctx), ops.py pre_<op>(ctx). Dict-valued EXTRA_ENV/UPGRADE_EXTRA_ENV (non-callable) still allowed — only the callable signature changes.
3. Migrate every in-repo user (spec §4 lists them per key; ops.py exists in ghost, discourse, immich, lasuite-*, mumble, others — builder: glob). Mechanical change; assertions and seeded values must remain byte-identical.
4. Unit tests: hook invocation passes a ctx with correct fields; a legacy-signature callable (lambda domain: ...) raises a CLEAR MetaError naming the migration (no silent TypeError mid-run).

P4 — Custom-test ergonomics

1. discovery.py::custom_tests: drop the top-level test_*.py glob for recipe dirs (keep functional/ + playwright/); lifecycle-name exclusion logic stays for safety. (Zero current users of top-level custom tests — verified; the change is doc'd as a placement RULE.)
2. New fixtures in tests/conftest.py:
   • op_state — parse $CCCI_OP_STATE_FILE (skip with clear reason if unset/absent);
   • deps — from P2d. Migrate overlay tests that hand-parse CCCI_OP_STATE_FILE / env (builder: grep CCCI_OP_STATE_FILE|os.environ under tests/*/test_*.py + functional/) to fixtures. Tests should not read os.environ directly except via fixtures after this phase (grep-clean, excluding conftest itself).
3. harness import surface: ensure from harness import lifecycle, sso, browser is the documented toolbox; no new module needed — docs job (P6).

P5 — Customization manifest

At run start (after meta load + discovery), print ONE block and embed the same dict in results.json under "customization":

===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 HTTP_TIMEOUT=600 DEPS=['keycloak'] ...
hooks: ops.py[pre_upgrade,pre_backup,pre_restore](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci) functional/=1 (repo-local)
env overrides: CCCI_SKIP_GENERIC_BACKUP=1   !! dev-only override active in CI

Pure presentation + one results.json key — MUST NOT influence any verdict. Unit test: manifest for a synthetic recipe dir is complete + deterministic; results schema test updated.

P6 — Docs

• docs/recipe-customization.md: §4 table generated (P1.5); §8 rewritten — R1/R2/R3/R6/R7/R8 resolved (say how), R4 mitigated by manifest, R9 rejected-by-decision; §3/§5 updated to the end-state shape (no setup_custom_tests.sh, placement rule, ctx hooks, fixtures incl. op_state/deps).
• docs/testing.md + docs/enroll-recipe.md: remove their partial key lists (point at the generated table), update hook signatures, fixture names, lasuite-docs worked example, local-run instructions.
• Keep all three docs' scope split: concepts (testing.md) / how-to (enroll-recipe.md) / reference+structure (recipe-customization.md).

---

Test suite additions (run inside pytest tests/unit -q — these are cheap/pure)

• tests/unit/test_meta.py (P1.4) — registry, validation, all-recipes-load-clean, R2 proof.
• ctx-hook tests (P3.4).
• discovery placement tests (P4.1) — top-level custom no longer discovered; functional/playwright are.
• manifest tests (P5).
• doc-sync test (P1.5).
• KEEP every existing unit test green; where a unit test pins old behavior being deleted (allowlist loaders, setup_custom_tests, deployed fixtures, top-level discovery), update it to pin the NEW behavior — never delete a test without a replacement that covers the successor path.

Roles, gates & Definition of Done (loop protocol — plan.md §6.1 applies)

Builder/Adversary loops, phase-namespaced state files: STATUS-rcust.md, REVIEW-rcust.md, BACKLOG-rcust.md, JOURNAL-rcust.md.

Builder (fable): implement P1→P6 on branch restructure/recipe-custom in YOUR clone; one commit per phase; before each commit ALL green: pytest tests/unit -q, scripts/lint.sh (tests/concurrency NOT required — untouched by this plan; run it once before M1 to prove that). Push the branch (NOT main — merge gated below). Claim gates via claim(rcust): ... commits + STATUS-rcust.md.

Adversary (opus): cold-verify from your own clone. For M1: check out the branch, run pytest tests/unit -q + pytest tests/concurrency -q + lint yourself, then adversarial review of the FULL diff. Hunt specifically:

• coverage loss — the cardinal risk of this restructure. For EVERY migrated recipe, diff the resolved customization (old loaders' effective values vs new meta.load()) — write a throwaway script that computes both for all 21 recipe dirs and diffs; any delta is a finding.
• assertion weakening in tests/<recipe>/ diffs: migrations must be mechanical (signatures, fixture names, underscore renames); any changed assert/expected-value = VETO.
• deleted-code fallout: grep for dangling refs to _recipe_meta, _load_meta, _recipe_extra_env, _recipe_meta_flag, declared_deps, is_canonical_enrolled, OIDC_AT_INSTALL, CHAOS_BASE_DEPLOY, SKIP_GENERIC, setup_custom_tests, deps_apps, deps_creds, deployed_app.
• validation gaps: craft a recipe_meta with a typo'd key / wrong type / callable-on-data-key — must MetaError, not silently pass.
• R2 actually fixed end-to-end (orchestrator path delivers SCREENSHOT to screenshot.py).
• HC2 gate integrity: repo-local default-deny unchanged; requires_deps skip-report (F2-11) unchanged; generic floor semantics unchanged. Findings to REVIEW-rcust.md; Adversary owns VETO.

Gates:

• M1 — implementation verified. Branch complete (P1–P6 + tests), unit+concurrency+lint green on the Adversary's cold clone, resolved-customization diff clean for all 21 recipes, adversarial diff review PASS in REVIEW-rcust.md.
• M2 — merged + REAL-CI REGRESSION SWEEP (operator-required). After M1 PASS only, Builder merges to main (merge commit, never force) and confirms the push build green. Then EVERY enrolled recipe is re-run through the real harness and must match its pre-change baseline:
  1. Baseline matrix first (build it BEFORE merging, during M1): for each of the 21 recipe dirs record expected outcome from the most recent known-good evidence (dashboard/results history; the bad-canary recipes custom-html-bkp-bad/rst-bad are EXPECTED to fail at their designed tier — record the tier). Commit the matrix to STATUS-rcust.md.
  2. Canary suite on the merged main: cc-ci-run python -m pytest tests/regression/ -m canary -v — all seven canaries (green canaries pass, RED canaries still caught at the intended tier; this is the false-green detector).
  3. Per-recipe full runs on the merged main, every enrolled recipe: RECIPE=<r> PR=<n> REF=<sha> SRC=recipe-maintainers/<r> STAGES=install,upgrade,backup,restore,custom cc-ci-run runner/run_recipe_ci.py using each mirror's current default-branch head as REF (or the open PR head where one exists — immich#2, plausible#3 may also be exercised via !testme through drone for at least TWO recipes so the drone→harness path is covered too). Max 2–3 concurrent (respect runner capacity); teardown verified after each (zero leaked apps — janitor sweep clean).
  4. Verdict + evidence: per-recipe result level == baseline matrix. ALSO spot-grep run logs to prove customizations actually executed (no silent loss): mumble READY_PROBE tcp lines, cryptpad SANDBOX_DOMAIN in env, ghost/discourse BACKUP_VERIFY lines + overlay copy + chaos base deploy, lasuite-* deps provisioning + OIDC tests ran (skip-count 0), immich ops.py seeds, manifest block present in every log, screenshot.png present where capture succeeded. Evidence (run ids, log excerpts) in STATUS-rcust.md; Adversary independently re-checks a sample of ≥5 recipes' logs + ALL mismatches. Any regression vs baseline → fix-forward on main only for trivial breakage with Adversary approval; otherwise revert the merge (rollback below) and return to M1.

## DONE in STATUS-rcust.md only when M1 and M2 both show a fresh Adversary PASS in REVIEW-rcust.md.

Guardrails (builder + adversary MUST honor)

• This plan DOES touch tests/<recipe>/ — but ONLY mechanically (signatures, fixture/key renames, underscore prefixes, install_steps consolidation). NEVER change an assertion, expected value, seeded marker, or test semantics. NEVER weaken recipe-test gates, the generic floor, HC2, or the F2-11 skip-report.
• cc-ci main is touched ONLY by the M2 merge after M1 PASS (and gated fix-forwards in M2). Never force-push. NEVER merge or push recipe-mirror repos — !testme comments only.
• No secrets in commits; reference .testenv / /run/secrets locations only.
• Teardown all dev/test deploys on every exit path; M2 sweep must leave zero apps behind.
• Do NOT touch concurrency machinery (lifetime.py, app locks, janitor, ABRA_DIR) beyond pass-through meta params.
• Match repo commit style (feat(harness)/fix(...)/test(...)/docs:).
• Stuck >2 cycles on the same defect → write it to BACKLOG-rcust.md + JOURNAL-rcust.md and ping the orchestrator via INBOX rather than thrashing.

Rollback

Single revert of the merge commit restores the six-loader world; recipe_meta.py edits ride in the same merge so the revert is self-consistent. No persistent state involved (config is all in-repo); re-running the M2 sweep after a revert re-validates the old world.