From aefaf17757451638bb188f3064f06986721186ae Mon Sep 17 00:00:00 2001 From: autonomic-bot Date: Wed, 10 Jun 2026 16:28:09 +0000 Subject: [PATCH] =?UTF-8?q?plan:=20recipe-customization=20restructure=20?= =?UTF-8?q?=E2=80=94=20full=20builder/adversary=20plan=20(P1-P6=20+=20real?= =?UTF-8?q?-CI=20regression=20sweep=20gate)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../recipe-custom-restructure-full-plan.md | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 cc-ci-plan/recipe-custom-restructure-full-plan.md diff --git a/cc-ci-plan/recipe-custom-restructure-full-plan.md b/cc-ci-plan/recipe-custom-restructure-full-plan.md new file mode 100644 index 0000000..36dd338 --- /dev/null +++ b/cc-ci-plan/recipe-custom-restructure-full-plan.md @@ -0,0 +1,256 @@ +# Recipe-customization restructure — full implementation plan + +Operator-approved direction (chat 2026-06-10). Reference spec of the CURRENT system: +cc-ci `docs/recipe-customization.md` (commit 76a4b6b) — read it FIRST; its §8 R1–R9 are the +defects this plan fixes. Goals: one coherent customization system, one place/way to configure a +recipe, easy custom tests, fix broken knobs, delete legacy paths, zero behavior regression for +currently-working recipes. + +## Decisions locked (operator + orchestrator) + +| Question | Decision | +|---|---| +| Loaders | ONE loader `runner/harness/meta.py::load(recipe) -> RecipeMeta` backed by a declarative KEY registry. All six current loaders (spec §4 L1–L6) migrate to it; `exec()` of recipe_meta.py happens in exactly one function. | +| Validation strictness | Unknown ALL-CAPS top-level name or type mismatch = **hard error** (run fails fast at load; unit test catches at PR time). Underscore-prefixed names (`_FOO`) are recipe-private and exempt. | +| `SCREENSHOT` | KEEP and FIX (it is currently dead — spec §8 R2). Reachable once the registry replaces the allowlist. | +| `CHAOS_BASE_DEPLOY` | DELETE. `tests//compose.ccci.yml` becomes first-class: harness copies it into the checkout + auto-uses `--chaos` for the base deploy. Both users (ghost, discourse) use the flag only for this. | +| `OIDC_AT_INSTALL` | DELETE. Install-time deps provisioning becomes the ONLY mode when `DEPS` is set; legacy post-deploy provisioning + `setup_custom_tests.sh` redeploy path removed. Migrate lasuite-docs. | +| `SKIP_GENERIC` (meta key) | DELETE (zero users). Env form `CCCI_SKIP_GENERIC*` stays as a documented LOCAL-DEV-ONLY escape hatch, loudly surfaced in the run manifest. | +| Hook signatures | All recipe callables take a single `ctx` (HookCtx) — `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`, `SCREENSHOT(page, ctx)`, `pre_(ctx)`. All users are in-repo; migrate them, no compat shim. | +| Fixtures | Single `tests/conftest.py`. Final surface: `recipe`, `meta` (full RecipeMeta), `live_app`, `op_state` (NEW), `deps` (NEW — consolidates `deps_apps`+`deps_creds`). DELETE dead pre-deploy-once fixtures `deployed`/`deployed_app` (zero users) and `app_domain` if nothing else uses it (builder: grep). | +| Custom-test placement | `test_.py` top-level = lifecycle overlay; ALL custom tests under `functional/` or `playwright/`. Discovery of top-level non-lifecycle `test_*.py` in recipe dirs is REMOVED (zero users today). | +| Docs | Key reference table is GENERATED from the registry; a unit test asserts docs ⊆ registry sync. | +| recipe_meta stays Python | NO data/hooks file split (spec §8 R9 considered, rejected — registry validation gets the value at lower cost). One file per recipe remains the single config place. | +| Landing | one branch `restructure/recipe-custom`, one commit per phase, merged to main only after Adversary M1 PASS + lint green; real-CI regression sweep (M2) after merge. | + +## Target shape (end state) + +``` +tests// +├── recipe_meta.py # THE config: registry-validated keys + ctx-hooks (+ _private consts) +├── test_.py # lifecycle overlay asserts (op_state/live_app/meta fixtures) +├── ops.py # pre_(ctx) seed hooks +├── functional/ playwright/ # ALL custom tests +├── install_steps.sh # pre-deploy shell hook (the only shell hook) +├── compose.ccci.yml # first-class CI overlay (auto-copied, auto-chaos) +└── PARITY.md +``` +One loader, one hook convention, one fixture file, one shell hook, one generated reference. + +--- + +## P1 — `harness/meta.py`: single loader + key registry + +1. New `runner/harness/meta.py`: + - `KEYS`: registry of every key — `name, type ("int"|"str"|"tuple[int]"|"bool"|"dict_or_hook"| + "hook"|"list[str]"|"dict"), default, doc` (one line each), optional `validate(value)`. + Final key set (14): HEALTH_PATH, HEALTH_OK, DEPLOY_TIMEOUT, HTTP_TIMEOUT, BACKUP_CAPABLE, + EXPECTED_NA, READY_PROBE, UPGRADE_BASE_VERSION, BACKUP_VERIFY, UPGRADE_EXTRA_ENV, EXTRA_ENV, + DEPS, WARM_CANONICAL, SCREENSHOT. (CHAOS_BASE_DEPLOY / OIDC_AT_INSTALL / SKIP_GENERIC are + deleted in P2 — during P1 keep them registered with `deprecated=True` so P1 lands green + before P2 removes them.) + - `load(recipe) -> RecipeMeta` (frozen dataclass, attribute access): the ONLY `exec()` of + `tests//recipe_meta.py`. Missing file → all defaults. Validation per the locked + decision: unknown non-underscore ALL-CAPS name → `MetaError` listing the unknown name and + nearest registered key; type mismatch → `MetaError`. Callables accepted only for hook-typed + keys. +2. Migrate ALL readers (spec §4 L1–L6 + §9 index has exact locations): + - `run_recipe_ci.py::_load_meta` → `meta.load()`; orchestrator loads ONCE and passes the + object down (functions grow a `meta` param; stop re-exec'ing per call). + - `tests/conftest.py::_recipe_meta` → `meta.load()` (fixture now returns full RecipeMeta). + - `lifecycle.py::_recipe_extra_env`, `lifecycle.py::_recipe_meta_flag` → deleted; callers take meta. + - `deps.py::declared_deps` → `meta.DEPS`; `canonical.py::is_canonical_enrolled` → `meta.WARM_CANONICAL`. + - `screenshot.py` consumer unchanged — but now the dict/object it receives actually contains + SCREENSHOT. **This is the R2 fix; prove it with a unit test** (meta with SCREENSHOT hook → + `_load_screenshot_hook` returns it through the real orchestrator load path). +3. Mumble private constants `WELCOME_TEXT_MARKER`, `MAX_USERS` → `_WELCOME_TEXT_MARKER`, + `_MAX_USERS` (fix their importers; builder: grep how mumble tests consume them). +4. New unit tests `tests/unit/test_meta.py`: + - every `tests/*/recipe_meta.py` in the repo loads clean through the registry (the typo gate); + - unknown-key and wrong-type files (tmp fixtures) raise MetaError; + - defaults match spec §2 baseline; underscore exemption; callable-on-data-key rejected. +5. Doc generation: `scripts/gen-meta-docs.py` renders the registry to a markdown table between + `` markers in `docs/recipe-customization.md` §4; unit test asserts + the committed table == rendered output (drift fails CI). + +## P2 — Delete legacy keys & paths + +a. **compose.ccci.yml first-class.** In the deploy path (after `install_steps.sh`, before base + deploy): if `tests//compose.ccci.yml` exists, copy it into the recipe checkout + (ABRA_DIR-aware) and use `--chaos` for the base deploy. Remove the copy boilerplate from + ghost/discourse `install_steps.sh` (delete the hook file entirely if copying was all it did — + builder: read both) and `CHAOS_BASE_DEPLOY = True` from both metas. Delete `_recipe_meta_flag` + remnants. Keep the policy note (overlays = minimal justified fallback) in docs. +b. **Install-time deps only.** Make the `OIDC_AT_INSTALL=True` code path the unconditional + behavior for recipes with DEPS; delete the legacy post-deploy provisioning branch, the + `setup_custom_tests.sh` invocation machinery, and the deploy-count exception for the legacy + redeploy. Migrate `lasuite-docs` to install-time wiring (mirror what lasuite-drive/meet do in + `install_steps.sh` reading `$CCCI_DEPS_FILE`). NOTE: lasuite-drive ALSO ships + `setup_custom_tests.sh` despite OIDC_AT_INSTALL=True — builder must read both scripts and + determine what they still do (realm/user provisioning is harness-side via `harness.sso`; env + wiring belongs in install_steps.sh). Whatever remains necessary moves into + `install_steps.sh`; then delete both `setup_custom_tests.sh` files and the key. +c. **SKIP_GENERIC meta key deleted**; env `CCCI_SKIP_GENERIC*` documented dev-only; if set in a + CI (drone) run, print a loud `!!` warning + record in the P5 manifest. +d. **Conftest cleanup**: delete `deployed`, `deployed_app` (dead, zero users — verified), and + `app_domain` if unused after their removal. Consolidate `deps_apps` + `deps_creds` into one + `deps` fixture (entries expose `.domain` plus full creds; dict-style access fine). Migrate the + 6 lasuite test files that use the old pair. Keep `requires_deps` marker + skip-report plumbing + unchanged (F2-11 gate — do NOT weaken). + +## P3 — Uniform `ctx` hook convention + +1. `harness/meta.py::HookCtx` (frozen dataclass): `.domain`, `.base_url`, `.meta` (RecipeMeta), + `.deps` (creds dict or None), `.op` (current lifecycle op or None). One constructor helper in + the orchestrator; harness builds it at each hook call site. +2. Convert call sites: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, + `BACKUP_VERIFY(ctx)`, `SCREENSHOT(page, ctx)`, `ops.py pre_(ctx)`. Dict-valued + EXTRA_ENV/UPGRADE_EXTRA_ENV (non-callable) still allowed — only the callable signature changes. +3. Migrate every in-repo user (spec §4 lists them per key; ops.py exists in ghost, discourse, + immich, lasuite-*, mumble, others — builder: glob). Mechanical change; assertions and seeded + values must remain byte-identical. +4. Unit tests: hook invocation passes a ctx with correct fields; a legacy-signature callable + (`lambda domain: ...`) raises a CLEAR MetaError naming the migration (no silent TypeError + mid-run). + +## P4 — Custom-test ergonomics + +1. `discovery.py::custom_tests`: drop the top-level `test_*.py` glob for recipe dirs (keep + `functional/` + `playwright/`); lifecycle-name exclusion logic stays for safety. (Zero current + users of top-level custom tests — verified; the change is doc'd as a placement RULE.) +2. New fixtures in `tests/conftest.py`: + - `op_state` — parse `$CCCI_OP_STATE_FILE` (skip with clear reason if unset/absent); + - `deps` — from P2d. + Migrate overlay tests that hand-parse `CCCI_OP_STATE_FILE` / env (builder: grep + `CCCI_OP_STATE_FILE|os.environ` under `tests/*/test_*.py` + `functional/`) to fixtures. Tests + should not read `os.environ` directly except via fixtures after this phase (grep-clean, + excluding conftest itself). +3. `harness` import surface: ensure `from harness import lifecycle, sso, browser` is the + documented toolbox; no new module needed — docs job (P6). + +## P5 — Customization manifest + +At run start (after meta load + discovery), print ONE block and embed the same dict in +`results.json` under `"customization"`: + +``` +===== customization manifest: ===== +meta (non-default): DEPLOY_TIMEOUT=1500 HTTP_TIMEOUT=600 DEPS=['keycloak'] ... +hooks: ops.py[pre_upgrade,pre_backup,pre_restore](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci) +overlays: test_backup.py(cc-ci) test_restore.py(repo-local) +custom tests: functional/=5 playwright/=2 (cc-ci) functional/=1 (repo-local) +env overrides: CCCI_SKIP_GENERIC_BACKUP=1 !! dev-only override active in CI +``` + +Pure presentation + one results.json key — MUST NOT influence any verdict. Unit test: manifest for +a synthetic recipe dir is complete + deterministic; results schema test updated. + +## P6 — Docs + +- `docs/recipe-customization.md`: §4 table generated (P1.5); §8 rewritten — R1/R2/R3/R6/R7/R8 + resolved (say how), R4 mitigated by manifest, R9 rejected-by-decision; §3/§5 updated to the + end-state shape (no setup_custom_tests.sh, placement rule, ctx hooks, fixtures incl. op_state/deps). +- `docs/testing.md` + `docs/enroll-recipe.md`: remove their partial key lists (point at the + generated table), update hook signatures, fixture names, lasuite-docs worked example, local-run + instructions. +- Keep all three docs' scope split: concepts (testing.md) / how-to (enroll-recipe.md) / + reference+structure (recipe-customization.md). + +--- + +## Test suite additions (run inside `pytest tests/unit -q` — these are cheap/pure) + +- `tests/unit/test_meta.py` (P1.4) — registry, validation, all-recipes-load-clean, R2 proof. +- ctx-hook tests (P3.4). +- discovery placement tests (P4.1) — top-level custom no longer discovered; functional/playwright are. +- manifest tests (P5). +- doc-sync test (P1.5). +- KEEP every existing unit test green; where a unit test pins old behavior being deleted + (allowlist loaders, setup_custom_tests, deployed fixtures, top-level discovery), update it to + pin the NEW behavior — never delete a test without a replacement that covers the successor path. + +## Roles, gates & Definition of Done (loop protocol — plan.md §6.1 applies) + +Builder/Adversary loops, phase-namespaced state files: STATUS-rcust.md, REVIEW-rcust.md, +BACKLOG-rcust.md, JOURNAL-rcust.md. + +**Builder** (fable): implement P1→P6 on branch `restructure/recipe-custom` in YOUR clone; one +commit per phase; before each commit ALL green: `pytest tests/unit -q`, `scripts/lint.sh` +(tests/concurrency NOT required — untouched by this plan; run it once before M1 to prove that). +Push the branch (NOT main — merge gated below). Claim gates via `claim(rcust): ...` commits + +STATUS-rcust.md. + +**Adversary** (opus): cold-verify from your own clone. For M1: check out the branch, run +`pytest tests/unit -q` + `pytest tests/concurrency -q` + lint yourself, then adversarial review of +the FULL diff. Hunt specifically: +- **coverage loss** — the cardinal risk of this restructure. For EVERY migrated recipe, diff the + resolved customization (old loaders' effective values vs new `meta.load()`) — write a throwaway + script that computes both for all 21 recipe dirs and diffs; any delta is a finding. +- assertion weakening in `tests//` diffs: migrations must be mechanical (signatures, + fixture names, underscore renames); any changed assert/expected-value = VETO. +- deleted-code fallout: grep for dangling refs to `_recipe_meta`, `_load_meta`, + `_recipe_extra_env`, `_recipe_meta_flag`, `declared_deps`, `is_canonical_enrolled`, + `OIDC_AT_INSTALL`, `CHAOS_BASE_DEPLOY`, `SKIP_GENERIC`, `setup_custom_tests`, `deps_apps`, + `deps_creds`, `deployed_app`. +- validation gaps: craft a recipe_meta with a typo'd key / wrong type / callable-on-data-key — + must MetaError, not silently pass. +- R2 actually fixed end-to-end (orchestrator path delivers SCREENSHOT to screenshot.py). +- HC2 gate integrity: repo-local default-deny unchanged; `requires_deps` skip-report (F2-11) + unchanged; generic floor semantics unchanged. +Findings to REVIEW-rcust.md; Adversary owns VETO. + +**Gates:** + +- **M1 — implementation verified.** Branch complete (P1–P6 + tests), unit+concurrency+lint green + on the Adversary's cold clone, resolved-customization diff clean for all 21 recipes, + adversarial diff review PASS in REVIEW-rcust.md. +- **M2 — merged + REAL-CI REGRESSION SWEEP (operator-required).** After M1 PASS only, Builder + merges to main (merge commit, never force) and confirms the push build green. Then EVERY + enrolled recipe is re-run through the real harness and must match its pre-change baseline: + 1. **Baseline matrix first** (build it BEFORE merging, during M1): for each of the 21 recipe + dirs record expected outcome from the most recent known-good evidence (dashboard/results + history; the bad-canary recipes custom-html-bkp-bad/rst-bad are EXPECTED to fail at their + designed tier — record the tier). Commit the matrix to STATUS-rcust.md. + 2. **Canary suite** on the merged main: `cc-ci-run python -m pytest tests/regression/ -m canary -v` + — all seven canaries (green canaries pass, RED canaries still caught at the intended tier; + this is the false-green detector). + 3. **Per-recipe full runs** on the merged main, every enrolled recipe: + `RECIPE= PR= REF= SRC=recipe-maintainers/ STAGES=install,upgrade,backup,restore,custom cc-ci-run runner/run_recipe_ci.py` + using each mirror's current default-branch head as REF (or the open PR head where one exists — + immich#2, plausible#3 may also be exercised via `!testme` through drone for at least TWO + recipes so the drone→harness path is covered too). Max 2–3 concurrent (respect runner + capacity); teardown verified after each (zero leaked apps — janitor sweep clean). + 4. **Verdict + evidence**: per-recipe result level == baseline matrix. ALSO spot-grep run logs + to prove customizations actually executed (no silent loss): mumble READY_PROBE tcp lines, + cryptpad SANDBOX_DOMAIN in env, ghost/discourse BACKUP_VERIFY lines + overlay copy + chaos + base deploy, lasuite-* deps provisioning + OIDC tests ran (skip-count 0), immich ops.py + seeds, manifest block present in every log, screenshot.png present where capture succeeded. + Evidence (run ids, log excerpts) in STATUS-rcust.md; Adversary independently re-checks a + sample of ≥5 recipes' logs + ALL mismatches. + Any regression vs baseline → fix-forward on main only for trivial breakage with Adversary + approval; otherwise revert the merge (rollback below) and return to M1. + +**## DONE in STATUS-rcust.md** only when M1 and M2 both show a fresh Adversary PASS in +REVIEW-rcust.md. + +## Guardrails (builder + adversary MUST honor) + +- This plan DOES touch `tests//` — but ONLY mechanically (signatures, fixture/key renames, + underscore prefixes, install_steps consolidation). NEVER change an assertion, expected value, + seeded marker, or test semantics. NEVER weaken recipe-test gates, the generic floor, HC2, or + the F2-11 skip-report. +- cc-ci main is touched ONLY by the M2 merge after M1 PASS (and gated fix-forwards in M2). Never + force-push. NEVER merge or push recipe-mirror repos — `!testme` comments only. +- No secrets in commits; reference `.testenv` / `/run/secrets` locations only. +- Teardown all dev/test deploys on every exit path; M2 sweep must leave zero apps behind. +- Do NOT touch concurrency machinery (lifetime.py, app locks, janitor, ABRA_DIR) beyond + pass-through `meta` params. +- Match repo commit style (`feat(harness)`/`fix(...)`/`test(...)`/`docs:`). +- Stuck >2 cycles on the same defect → write it to BACKLOG-rcust.md + JOURNAL-rcust.md and ping + the orchestrator via INBOX rather than thrashing. + +## Rollback + +Single revert of the merge commit restores the six-loader world; recipe_meta.py edits ride in the +same merge so the revert is self-consistent. No persistent state involved (config is all +in-repo); re-running the M2 sweep after a revert re-validates the old world.