diff --git a/docs/enroll-recipe.md b/docs/enroll-recipe.md index f9b4526..f953e12 100644 --- a/docs/enroll-recipe.md +++ b/docs/enroll-recipe.md @@ -14,8 +14,9 @@ those are discovered and run against the live app (D4 — see below). ``` tests// ├── recipe_meta.py # optional per-recipe harness config (see below) -├── install_steps.sh # optional custom install-steps hook (pre-deploy setup) -├── ops.py # optional pre-op seed hooks (pre_install/pre_upgrade/pre_backup/pre_restore) +├── install_steps.sh # optional custom install-steps hook (pre-deploy setup + deps env wiring) +├── compose.ccci.yml # optional CI-only compose overlay (harness-copied, auto-chaos base deploy) +├── ops.py # optional pre_(ctx) seed hooks (install/upgrade/backup/restore) ├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic) ├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic) ├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic) @@ -39,11 +40,14 @@ To add recipe-specific coverage, drop a `tests//test_.py` **overlay* **ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture; they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to -SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_(domain, -meta)` callables in `tests//ops.py` — the orchestrator runs them BEFORE the op. Copy an +SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_(ctx)` +callables in `tests//ops.py` — the orchestrator runs them BEFORE the op (`ctx` is the +uniform `HookCtx` every hook receives — `docs/recipe-customization.md` §4.1). Copy an existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/ matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` / -`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py`: +`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py` (the COMPLETE key +reference is the generated table in `docs/recipe-customization.md` §4; unknown ALL-CAPS keys are +hard errors, recipe-private constants are underscore-prefixed `_FOO`): ```python HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") @@ -51,9 +55,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302) DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose) -EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy -SKIP_GENERIC = ["upgrade"] # per-recipe opt-out from the generic floor for the listed ops - # ("all"/"*" = every op); rarely needed — generic is the floor +EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy ``` Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`, @@ -76,9 +78,10 @@ Beyond the lifecycle overlays, each recipe carries (plan §4.1): - **`playwright/`** — browser flows where the recipe's core UX is a UI (P6). The orchestrator's **custom** tier discovers `test_*.py` in `tests//{functional,playwright}/` -(recursive, via `runner/harness/discovery.custom_tests`) and runs each as its own pytest against -the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are -**excluded** from the custom tier — they live at the top level and run as lifecycle overlays. +ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py` +is a lifecycle overlay and nothing else) and runs each as its own pytest against the same +`live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded** +from the custom tier even inside those subdirs (safety net against double-running). ### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3) @@ -89,23 +92,28 @@ them in `recipe_meta.py`: DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests// must exist + work) ``` -The orchestrator (plan §4.2): -1. Reads `DEPS` BEFORE deploying the recipe under test. -2. Deploys each dep at a per-run domain `-<6hex>.ci.commoninternet.net` (the 6hex is - hashed from `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do - not collide on a single node). -3. Waits each dep healthy using its own `recipe_meta.py` (HEALTH_PATH/HEALTH_OK/timeouts). -4. Persists `[{"recipe": "", "domain": ""}, ...]` to `$CCCI_DEPS_FILE`. -5. Deploys + tests the recipe under test as usual. -6. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked +The orchestrator (plan §4.2; install-time provisioning is the ONLY mode): +1. Reads `DEPS` and provisions every dep **BEFORE the single deploy** of the recipe under test — + each dep at a per-run domain `-<6hex>.ci.commoninternet.net` (the 6hex is hashed from + `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do not collide on + a single node), waited healthy using the dep's own `recipe_meta.py`. +2. Persists the full per-dep identity + SSO creds dict to `$CCCI_DEPS_FILE` (jq-readable JSON, + `{"": {"domain": ..., "realm": ..., "client_secret": ..., ...}}`). +3. Deploys the recipe under test — its `install_steps.sh` reads `$CCCI_DEPS_FILE` and wires + OIDC env into that ONE deploy (no post-deploy redeploy). A dep-provisioning failure does NOT + block the run: the recipe deploys alone, generic tiers run, and `requires_deps` tests skip + with a counted reason (F2-11). +4. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked deps fail the run loudly per §9 teardown sacred / F2-5 fix). -Tests access dep domains via the **`deps_apps` pytest fixture** (`tests/conftest.py`): +Tests access deps via the **`deps` pytest fixture** (`tests/conftest.py`) — entries expose +`.domain` plus the full creds dict (attribute or dict-style): ```python -def test_my_recipe_uses_keycloak(live_app, deps_apps): - assert "keycloak" in deps_apps, f"keycloak dep not deployed; {deps_apps}" - kc_domain = deps_apps["keycloak"] +@pytest.mark.requires_deps +def test_my_recipe_uses_keycloak(live_app, deps): + assert "keycloak" in deps, f"keycloak dep not deployed; {deps}" + kc_domain = deps["keycloak"].domain … ``` @@ -120,7 +128,7 @@ For OIDC-dependent recipes, the shared `runner/harness/sso.py` provides: from harness import sso creds = sso.setup_keycloak_realm( - kc_domain, # = deps_apps["keycloak"] + kc_domain, # = deps["keycloak"].domain realm="my-realm", client_id="my-client", redirect_uris=[f"https://{live_app}/*"], @@ -144,10 +152,10 @@ ARE provider-pluggable. Not every recipe is a single HTTP app. `recipe_meta.py` + a few harness mechanisms cover the harder shapes (proven on mumble, mailu, and the SSO-dependent suite): -- **`EXTRA_ENV`** — a dict **or** a `callable(domain) -> dict`. The callable form derives values from - the per-run domain (e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for cryptpad). Applied - at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change. -- **`READY_PROBE(domain) -> [...]`** — readiness signals beyond replica-convergence + the app's +- **`EXTRA_ENV`** — a dict **or** a `callable(ctx) -> dict`. The callable form derives values from + the per-run domain (`ctx.domain` — e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for + cryptpad). Applied at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change. +- **`READY_PROBE(ctx) -> [...]`** — readiness signals beyond replica-convergence + the app's `HEALTH_PATH`. Two probe shapes: - HTTP: `{"host": "...", "path": "/...", "ok": (200,)}` (e.g. lasuite-drive collabora WOPI discovery). - **TCP**: `{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}` — polls a socket connect N @@ -155,16 +163,16 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite): service (mumble: the mumble-web sidecar serves HTTP 200 while the voice server on 64738 is still rebinding after an upgrade redeploy — the TCP probe gates the backup tier until the voice server is actually up). Runs after install AND after the upgrade chaos redeploy. -- **`CHAOS_BASE_DEPLOY = True`** — make the pinned base deploy use `--chaos` (skips abra's clean-tree + - lint gates, still deploys the explicitly-checked-out pinned version, NOT latest). Needed when an - `install_steps.sh` adds an UNTRACKED file to the recipe checkout (e.g. mumble copies a - `compose.host-ports.yml` into versions that predate it) — abra's pinned-deploy clean-tree check would - otherwise FATA. `abra.recipe_checkout` force-checks-out (`-f`) so the upgrade tier's re-checkout to - PR-head overwrites such overlays cleanly. +- **`compose.ccci.yml`** (first-class at `tests//compose.ccci.yml`) — a CI-only compose + overlay the harness itself copies into the recipe checkout before the base deploy, automatically + using `--chaos` for that deploy (the untracked file would otherwise trip abra's pinned-deploy + clean-tree check). Reference it from `EXTRA_ENV`'s `COMPOSE_FILE`. Minimal, justified fallback + only (e.g. ghost's 15m `start_period` grace). `abra.recipe_checkout` force-checks-out (`-f`) so + the upgrade tier's re-checkout to PR-head overwrites such overlays cleanly. - **`install_steps.sh`** (auto-discovered at `tests//install_steps.sh`) — runs after `abra app new` + EXTRA_ENV + secret-generate, BEFORE the single deploy, with `CCCI_APP_DOMAIN` / - `CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when DEPS are provisioned at install). Use it to - drop a cc-ci-owned compose overlay into the checkout, wire dep-derived env/secrets, etc. + `CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when the recipe declares DEPS — deps are + always provisioned before the deploy). Use it to wire dep-derived env/secrets, seed config, etc. **Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports overlay) at `127.0.0.1:` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol @@ -227,9 +235,10 @@ RECIPE= PR= REF= SRC=recipe-maintainers/ \ ``` tests/lasuite-docs/ -├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(domain) for cold-pull, +├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(ctx) for cold-pull, │ # DEPS=["keycloak"] ← Phase 2 dep declaration -├── ops.py # pre_ seed hooks (volume marker for backup/restore data-integrity) +├── install_steps.sh # wires OIDC env from $CCCI_DEPS_FILE into the single deploy +├── ops.py # pre_(ctx) seed hooks (volume marker for backup/restore data-integrity) ├── test_install.py # lifecycle install overlay (Playwright frontend SPA load) ├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy) ├── test_backup.py # lifecycle backup overlay (marker captured) @@ -239,12 +248,14 @@ tests/lasuite-docs/ ├── test_health_check.py # parity port (SOURCE comment cites recipe-info file) ├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth └── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses - # harness.sso primitives + deps_apps["keycloak"]) + # harness.sso primitives + the `deps` fixture) ``` `!testme` on a lasuite-docs PR drives the orchestrator to: -1. Deploy the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`) and wait healthy. -2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`). +1. Provision the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`), wait healthy, write + creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy. +2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC + env into that one deploy. 3. Run install / upgrade / backup / restore + the 3 functional tests against the shared deployment (custom tier). 4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True. @@ -254,12 +265,13 @@ tests/lasuite-docs/ ### Other shapes (concrete references) - **TCP / voice recipe — `tests/mumble/`**: `recipe_meta.py` (EXTRA_ENV sets - `COMPOSE_FILE=compose.yml:compose.mumbleweb.yml:compose.host-ports.yml`, `WELCOME_TEXT`/`USERS` - markers, `CHAOS_BASE_DEPLOY=True`, `READY_PROBE` TCP 64738), `install_steps.sh` (provides the - host-ports overlay to older versions), `functional/_mumble_proto.py` + the protocol/config-round-trip + `COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the + native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private + `_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via + the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4. - **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py` - (`EXTRA_ENV(domain)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`), + (`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`), `functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back), `test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md + DEFERRED.md). See §2.4. diff --git a/docs/recipe-customization.md b/docs/recipe-customization.md index da5af89..ef2fea9 100644 --- a/docs/recipe-customization.md +++ b/docs/recipe-customization.md @@ -1,8 +1,9 @@ -# Recipe customization — review spec +# Recipe customization — reference -Status: REVIEW SPEC — describes the customization surface as it exists today (main), written so -the structure can be reviewed and potentially restructured. §8 lists known limitations and -restructuring candidates; everything before it is purely descriptive. +Status: REFERENCE — describes the customization system as restructured on branch +`restructure/recipe-custom` (the "rcust" restructure). The pre-restructure system and its defects +are documented in this file's history (commit `76a4b6b`, the review spec whose §8 R1–R9 drove the +restructure); §8 below records how each was resolved. Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md` (step-by-step enrollment). This doc is the **complete reference** for the two questions those docs @@ -15,17 +16,18 @@ answer only partially: ## 1. The three customization surfaces -A recipe customizes its CI through **three distinct mechanisms** (worth noticing for the -restructure review — they are three different config languages): +A recipe customizes its CI through **three distinct mechanisms**: | Surface | Form | Examples | |---|---|---| | **Declarative settings** | Python assignments in `tests//recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` | -| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, shell hooks | `def READY_PROBE(domain): ...`, `pre_upgrade()`, `install_steps.sh` | +| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` | | **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` | -There is additionally a fourth, operator-facing surface: **environment variables** -(`CCCI_SKIP_GENERIC*`) that override declarative settings at run time (§4.4). +There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables +(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves +from all four surfaces is printed at run start as the **customization manifest** and embedded in +`results.json` under `"customization"` (§7) — one block answers "what does this recipe customize?". ## 2. Zero-config baseline @@ -55,53 +57,53 @@ Two locations, with precedence and a security gate between them: ``` tests// # cc-ci side (repo-local mirrors the same shape) -├── recipe_meta.py # ALL declarative settings + meta callables (§4) +├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4) ├── test_.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1) -├── ops.py # pre_(domain, meta) seed hooks (§5.2) -├── test_*.py # custom-tier tests (top-level, cross-cutting)(§5.3) +├── ops.py # pre_(ctx) seed hooks (§5.2) ├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3) ├── playwright/test_*.py # custom tier: UI flows (§5.3) -├── install_steps.sh # pre-deploy shell hook (§5.4) -├── setup_custom_tests.sh # deps/OIDC credential wiring hook (§5.5) -├── compose.ccci.yml # CI-only compose overlay (via install_steps) (§5.6) +├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4) +├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5) └── PARITY.md # enrollment contract doc (human-read only) ``` +**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or +`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_.py`) and nothing else — +top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name +exclusion stays as a safety net so a misfiled `test_.py` can never double-run). + Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`): - lifecycle overlay `test_.py`: repo-local **wins** over cc-ci (same-name collision); the generic floor still runs additively alongside. -- custom tier `test_*.py`: **ALL** run, from both locations (no collision concept). +- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision + concept). - `install_steps.sh`: repo-local > cc-ci, or none. - `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved. -- `recipe_meta.py`: cc-ci only — repo-local recipes cannot set CI settings (by design; the - settings surface stays maintainer-controlled). +- `recipe_meta.py` and `compose.ccci.yml`: cc-ci only — repo-local recipes cannot set CI settings + or compose overlays (by design; those surfaces stay maintainer-controlled). ## 4. `recipe_meta.py` — complete settings reference -The single settings file. Plain Python, `exec()`d by the harness (trusted, in-repo). A key is "set" -by a top-level assignment or `def`. Unknown names are ignored silently (a recipe may keep private -constants here, e.g. mumble's `WELCOME_TEXT_MARKER` — but see §8 R6: typos in real key names are -also silently ignored). +The single settings file. Plain Python, `exec()`d by the harness in exactly ONE place: the +registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta`. Every consumer — the +orchestrator (which loads once and passes the object down), the pytest `meta` fixture, lifecycle, +deps, canonical, screenshot — reads from that one loaded object. -**Loader column legend** — this is the structural finding for the review (§8 R1). There is no -single loader; six independent code paths each `exec()` the file and pick out their own keys: +**Validation (hard errors at load, before any deploy):** -| # | Loader | Keys it sees | -|---|---|---| -| L1 | `runner/run_recipe_ci.py:_load_meta` (orchestrator) | 4 base + explicit 8-key allowlist | -| L2 | `tests/conftest.py:_recipe_meta` (pytest `meta` fixture) | 4 base keys ONLY | -| L3 | `runner/harness/lifecycle.py:_recipe_extra_env` | `EXTRA_ENV` only | -| L4 | `runner/harness/lifecycle.py:_recipe_meta_flag` | boolean flags by name (`CHAOS_BASE_DEPLOY`) | -| L5 | `runner/harness/deps.py:declared_deps` | `DEPS` only | -| L6 | `runner/harness/canonical.py:is_canonical_enrolled` | `WARM_CANONICAL` only | +- A key is "set" by a top-level ALL-CAPS assignment or `def`. Unknown ALL-CAPS top-level names + raise `MetaError` listing the unknown name and the nearest registered key (typo gate — + misspelling `READY_PROBE` can no longer silently disable the probe). +- Type mismatches raise `MetaError`; callables are accepted only for hook-typed keys. +- **Underscore-prefixed names (`_FOO`) are recipe-private and exempt** — that's where private + constants live (e.g. mumble's `_WELCOME_TEXT_MARKER`). Lowercase names (helpers/imports) are + ignored. +- Hook callables must have the registered signature (below); a legacy-signature hook raises a + `MetaError` naming the migration, never a silent `TypeError` mid-run. -> **Restructure status (rcust P1):** the six loaders above are HISTORY — they have been replaced by -> the single registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta` (the only -> `exec()` of `recipe_meta.py`). Unknown ALL-CAPS keys / type mismatches are now hard errors; -> underscore-prefixed names are recipe-private. The authoritative key reference is the generated -> table below; the per-loader subsections §4.1–§4.8 are retained for context until the P6 doc -> rewrite. +A unit test (`tests/unit/test_meta.py`) loads every `tests/*/recipe_meta.py` through the registry, +so a typo'd key fails at PR time, not at run time. @@ -126,64 +128,27 @@ _This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scr -### 4.1 HTTP / health / timing (base 4 — seen by L1 AND L2) +### 4.1 The uniform hook convention — `HookCtx` -| Key | Type / default | Meaning | Used by | -|---|---|---|---| -| `HEALTH_PATH` | str, `"/"` | Path probed for serving/health checks | deploy wait (`lifecycle.py`), generic `assert_serving` | -| `HEALTH_OK` | tuple, `(200, 301, 302)` | Acceptable HTTP status codes for health | same | -| `DEPLOY_TIMEOUT` | int s, `600` | Max wait for swarm convergence per deploy | `lifecycle.py`, generic ops | -| `HTTP_TIMEOUT` | int s, `300` | Max wait for HTTP health after converged | same | +Every recipe callable takes a single `ctx` argument (`harness/meta.py::HookCtx`, frozen): -Example: immich sets `DEPLOY_TIMEOUT = 1500`, `HTTP_TIMEOUT = 600` (ML containers are slow). +| Field | Meaning | +|---|---| +| `ctx.domain` | the app's per-run domain | +| `ctx.base_url` | `https://` | +| `ctx.meta` | the recipe's full `RecipeMeta` | +| `ctx.deps` | provisioned dep creds (`{dep_recipe: entry}`) or `None` | +| `ctx.op` | current lifecycle op (`install`/`upgrade`/`backup`/`restore`) or `None` | -### 4.2 Upgrade tier (loader L1) +Signatures: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`, +`SCREENSHOT(page, ctx)`, ops.py `pre_(ctx)`. Dict-valued `EXTRA_ENV`/`UPGRADE_EXTRA_ENV` +(non-callable) are still fine — only the callable form takes ctx. The loader enforces the +parameter names at load time (a pre-restructure `(domain)`/`(domain, meta)` hook gets a pointed +`MetaError`, not a mid-run crash). -| Key | Type / default | Meaning | -|---|---|---| -| `UPGRADE_BASE_VERSION` | str (exact published tag), default `None` | **The "base pin"** — overrides the harness default base for the upgrade tier. Default base = `recipe_versions[-2]` (the previous published version); pin when that is not the PR's true predecessor (e.g. the PR is the first release on a new major, or the previous tag is known-broken). Must be an exact published tag — typos fail the base deploy. Consumed at `run_recipe_ci.py` (`prev = meta.get("UPGRADE_BASE_VERSION") or lifecycle.previous_version(recipe)`). Users: discourse, plausible. | -| `UPGRADE_EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `None` | Extra `.env` keys applied **after** the PR-head checkout, **before** the chaos redeploy (F2-14c) — for env vars that exist only at head (a new required setting introduced by the PR). Consumed in `generic.py:256`. User: mumble. | - -### 4.3 Every-deploy shaping (loaders L3/L4 — NOT in the L1 allowlist) - -| Key | Type / default | Meaning | -|---|---|---| -| `EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `{}` | Extra `.env` keys applied at **every** deploy (base install AND upgrade old-app). Callable form derives values from the per-run domain (e.g. cryptpad's `SANDBOX_DOMAIN`). Loaded by `lifecycle.py:_recipe_extra_env` (its own `exec()`). Users: cryptpad, discourse, ghost, matrix-synapse, mattermost-lts, mumble, plausible. | -| `CHAOS_BASE_DEPLOY` | bool, default `False` | Base deploy uses `--chaos` so it survives untracked files in the recipe checkout (required when `install_steps.sh` copies in a `compose.ccci.yml` overlay — §5.6; implicit coupling, see §8 R7). Loaded by `lifecycle.py:_recipe_meta_flag`. Users: discourse, ghost. | - -### 4.4 Skips and intentional N/A (loader L1) - -| Key | Type / default | Meaning | -|---|---|---| -| `SKIP_GENERIC` | list of op names or `"all"`/`"*"`, default `[]` | Suppress the generic floor for the listed ops (overlay becomes override instead of additive). Two env equivalents at run time: `CCCI_SKIP_GENERIC=1` (all ops), `CCCI_SKIP_GENERIC_=1` (one op). Currently set by **no enrolled recipe** (env form is the one used, ad hoc). | -| `EXPECTED_NA` | dict `{rung: reason}`, default `None` | Declares an N/A rung **intentional** (e.g. `{"backup": "stateless, nothing to back up"}`). Undeclared N/A is reported as an *unintentional coverage gap*. Both cap the achievable level — declaring does not un-cap, it only changes the report wording (`results.py`). User: custom-html-tiny. | -| `BACKUP_CAPABLE` | bool, default auto-detect | Overrides the backup-tier capability detection (scan of recipe compose files for `backupbot.backup` labels, `generic.py:34`). `False` forces N/A; `True` forces the tier on. Users: custom-html-bkp-bad/rst-bad (harness self-test recipes). | - -### 4.5 Readiness & data-verification hooks (loader L1, callable values) - -| Key | Type / default | Meaning | -|---|---|---| -| `READY_PROBE` | callable `(domain) -> [probe, ...]`, default `None` | Extra readiness probes run after install AND after upgrade, before that tier's assertions. Probe dicts: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}` (`stable`: must stay connectable across 3 checks — for UDP-adjacent voice ports etc.). Consumed at `lifecycle.py:516`. Users: lasuite-drive, mumble (TCP voice port). | -| `BACKUP_VERIFY` | callable `(domain) -> bool`, default `None` | Post-backup data-capture check, retried — guards the truncated-dump race (backup snapshot taken before the seeded marker row hit disk). Return `False` → retry the backup, then fail. Users: discourse, ghost. | - -### 4.6 Dependencies / SSO (loaders L5 + L1) - -| Key | Type / default | Meaning | -|---|---|---| -| `DEPS` | list of recipe names, default `[]` | Dep recipes deployed alongside (e.g. `["keycloak"]`). Dep domain is `-<6hex>`, hashed from (parent, pr, ref, dep) — collision-free per run. Creds land in `$CCCI_DEPS_FILE` (JSON); tests use the `deps_apps` fixture; teardown deps LAST. Deploy-count guard becomes `1 + len(DEPS)`. Loaded by `deps.py:declared_deps`. Users: lasuite-docs/-drive/-meet. | -| `OIDC_AT_INSTALL` | bool, default `False` | Provision deps **before** the single base deploy so `install_steps.sh` can wire OIDC env into that one deploy (reads `$CCCI_DEPS_FILE`). Default (legacy) is post-deploy provisioning + a `setup_custom_tests.sh` redeploy. Consumed at `run_recipe_ci.py:514`. Users: lasuite-drive, lasuite-meet. | - -### 4.7 Warm-canonical enrollment (loader L6) - -| Key | Type / default | Meaning | -|---|---|---| -| `WARM_CANONICAL` | bool, default `False` | Enrolls the recipe in the warm/canonical app system (`docs/warm.md`): green COLD runs on LATEST advance the canonical snapshot; the nightly sweep iterates enrolled recipes. Loaded by `canonical.py:is_canonical_enrolled`. User: custom-html. | - -### 4.8 Cosmetic (BROKEN — see §8 R2) - -| Key | Type / default | Meaning | -|---|---|---| -| `SCREENSHOT` | callable `(page, domain, meta) -> None` | Drives Playwright to a safe post-login view for the results-card screenshot (default: landing page). **Currently unreachable from the CI path**: `screenshot.py:41` reads it from the meta dict the orchestrator passes (`run_recipe_ci.py:1056`), but the L1 allowlist never loads `SCREENSHOT`, so the hook is always `None`. No recipe sets it (consistent with it never having worked). | +Worked hook examples: cryptpad (`EXTRA_ENV(ctx)` derives `SANDBOX_DOMAIN` from `ctx.domain`), +mumble (`READY_PROBE(ctx)` TCP voice-port probe, `UPGRADE_EXTRA_ENV(ctx)` adds a head-only compose +overlay), ghost/discourse (`BACKUP_VERIFY(ctx)` dump-capture check). ## 5. Writing custom tests & hooks @@ -196,104 +161,122 @@ test runs additively against the same state. Conventions (see `tests/immich/test_backup.py` etc.): - use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain) -- use the `meta` fixture for HEALTH_*/timeouts (note: only the 4 base keys — §8 R3) -- read op context from `$CCCI_OP_STATE_FILE` (JSON written by the orchestrator after the op: - versions, artifact paths) +- use the `meta` fixture — the recipe's FULL validated `RecipeMeta` (attribute access) +- use the `op_state` fixture for op context (versions, `snapshot_id`, artifact paths — the + orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run) - execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)` ### 5.2 Pre-op seed hooks — `ops.py` -`def pre_(domain, meta)` callables, imported and called by the orchestrator **before** -performing the op. This is where data gets seeded so the post-op overlay can assert on it: +`def pre_(ctx)` callables, imported and called by the orchestrator **before** performing the +op. This is where data gets seeded so the post-op overlay can assert on it: ```python # tests/immich/ops.py (pattern) -def pre_upgrade(domain, meta): _psql(domain, "INSERT ... 'upgrade-survives'") -def pre_backup(domain, meta): _psql(domain, "INSERT ... 'original'") -def pre_restore(domain, meta): _psql(domain, "DROP TABLE ci_marker") # damage, restore must undo +def pre_upgrade(ctx): _psql(ctx.domain, "INSERT ... 'upgrade-survives'") +def pre_backup(ctx): _psql(ctx.domain, "INSERT ... 'original'") +def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, restore must undo ``` Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up, `pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back. -### 5.3 Custom tier — `functional/`, `playwright/`, top-level `test_*.py` +### 5.3 Custom tier — `functional/` and `playwright/` ONLY -All non-lifecycle `test_*.py` (discovery: `discovery.py:custom_tests`, recursive over the -top-level dir + `functional/` + `playwright/`; files named `test_.py` excluded). Run in the -CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run — -cc-ci's and (if HC2-approved) repo-local's, additively. +All custom-tier tests live under `tests//functional/` or `tests//playwright/` +(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after +restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if +HC2-approved) repo-local's, additively. Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso` -(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). +(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented +import toolbox for custom tests is `from harness import lifecycle, sso, browser`. -Tests gate on deps via `CCCI_DEPS_READY` (skip-with-reason when `0`; the skip is counted and -fails the run if deps were declared but unprovisionable — `run_recipe_ci.py:816`). +Tests needing deps use the `deps` fixture (entries expose `.domain` plus the full creds dict) and +carry `@pytest.mark.requires_deps` — when dep provisioning failed they skip with reason +`deps-not-ready` and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit +must not mask an unrun SSO test). Fixtures replace direct `os.environ` reads — after the +restructure no recipe test parses env by hand. ### 5.4 Pre-deploy shell hook — `install_steps.sh` -Runs after `abra app new` + `EXTRA_ENV` application + secret generation, **before** the base -deploy. For setup that must precede the first deploy: writing extra config files into the recipe -checkout, copying in a `compose.ccci.yml` overlay (§5.6), editing `.env` beyond simple key=val. +The ONLY shell hook. Runs after `abra app new` + `EXTRA_ENV` application + secret generation, +**before** the single base deploy. For setup that must precede the first deploy: writing extra +config files into the recipe checkout, editing `.env` beyond simple key=val, and — for recipes +with `DEPS` — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the +deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy +redeploy hook). Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and — -when `OIDC_AT_INSTALL` deps exist — `CCCI_DEPS_FILE`. Must locate the recipe checkout -ABRA_DIR-aware: `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run -`ABRA_DIR` since the concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree). +when `DEPS` is declared — `CCCI_DEPS_FILE` (jq-readable JSON of dep creds/URLs; see +lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware: +`RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run `ABRA_DIR` since the +concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree). Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic install — a correct reported outcome, not a harness error. -### 5.5 Deps credential wiring — `setup_custom_tests.sh` +### 5.5 CI-only compose overlay — `compose.ccci.yml` -For legacy (post-deploy) deps provisioning: runs after deps are up, reads `$CCCI_DEPS_FILE` -(jq-readable JSON of dep creds/URLs), wires OIDC config via `abra app config set` + secrets, and -redeploys. With `OIDC_AT_INSTALL = True` this hook is unnecessary (wiring happens in -`install_steps.sh` before the only deploy) — preferred for new enrollments (one deploy, no -deploy-count exception). +**First-class:** if `tests//compose.ccci.yml` exists, the harness itself copies it into +the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses `--chaos` for +that deploy (the untracked file would otherwise trip abra's clean-tree gate). No +`install_steps.sh` copy boilerplate, no flag to remember (the old `CHAOS_BASE_DEPLOY` ⇄ overlay +coupling is gone). The overlay is cc-ci-owned only. -### 5.6 CI-only compose overlay — `compose.ccci.yml` +Policy unchanged: overlays are a minimal, justified fallback (ghost's is a 15m `start_period` +grace — a literal, because abra validates `start_period` before env substitution). Reference the +overlay from `EXTRA_ENV`'s `COMPOSE_FILE` as usual. Users: ghost, discourse. -Not auto-discovered: `install_steps.sh` copies it into the recipe checkout, and the recipe must -set `CHAOS_BASE_DEPLOY = True` so the base deploy (`--chaos`) tolerates the untracked file. -Policy: minimal, justified fallback only (ghost's is a 15m `start_period` grace — a literal, -because abra validates `start_period` before env substitution). The overlay is cc-ci-owned even -though it rides in the recipe checkout. +### 5.6 Environment & fixture contract (what custom code can read) -### 5.7 Environment contract summary (what custom code can read) +Pytest fixtures (`tests/conftest.py` — the single fixture file): + +| Fixture | Yields | +|---|---| +| `recipe` | the recipe name (`$RECIPE`) | +| `meta` | the FULL validated `RecipeMeta` (single loader) | +| `live_app` | the shared deployment's domain (asserts it exists) | +| `op_state` | the orchestrator's op-context dict (skips cleanly outside a run) | +| `deps` | `{dep_recipe: entry}` — entries expose `.domain` + full SSO creds | + +Environment (hooks/shell, and approved repo-local code): | Var | Set for | Meaning | |---|---|---| | `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain | | `CCCI_BASE_URL` | approved repo-local code | `https://` | | `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path | -| `CCCI_OP_STATE_FILE` | overlay tests | JSON op context (versions, artifacts) | -| `CCCI_DEPS_FILE` | deps hooks + tests | JSON dep creds dict | -| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier | gate SSO tests, skip-with-reason | +| `CCCI_OP_STATE_FILE` | overlay tests (via `op_state`) | JSON op context (versions, artifacts) | +| `CCCI_DEPS_FILE` | `install_steps.sh` + harness | JSON dep creds dict | +| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier (via `requires_deps`) | gate SSO tests, skip-with-reason | ## 6. Run-model context (what the settings plug into) One deploy chain per run (full detail: `docs/testing.md` §2): ``` +[DEPS? provision deps FIRST → $CCCI_DEPS_FILE] deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh; - CHAOS_BASE_DEPLOY?; OIDC_AT_INSTALL deps first?) + compose.ccci.yml auto-copied + auto-chaos) → INSTALL tier (READY_PROBE; generic + overlay asserts) - → pre_upgrade → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV) + → pre_upgrade(ctx) → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV) → UPGRADE tier (READY_PROBE; version-label == head_ref) - → pre_backup → backup (BACKUP_CAPABLE; BACKUP_VERIFY) + → pre_backup(ctx) → backup (BACKUP_CAPABLE; BACKUP_VERIFY) → BACKUP tier - → pre_restore → restore + → pre_restore(ctx) → restore → RESTORE tier - → CUSTOM tier (functional/ + playwright/; deps via CCCI_DEPS_*) + → CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture) + → SCREENSHOT (best-effort, never affects the verdict) → teardown (deps LAST) ``` Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't count); the per-run counter file is keyed by run since the concurrency restructure. -## 7. Local iteration +## 7. Local iteration, the manifest, and the dev-only escape hatch ``` RECIPE= PR= REF= SRC=recipe-maintainers/ \ @@ -303,81 +286,75 @@ RECIPE= PR= REF= SRC=recipe-maintainers/ \ (`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.) -## 8. Known limitations & restructuring candidates +**Customization manifest.** Every run prints, right after meta load + discovery, one block: -The review section. Ordered by how much they'd shape a restructure. +``` +===== customization manifest: ===== +meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='' +hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci) +overlays: test_backup.py(cc-ci) test_restore.py(repo-local) +custom tests: functional/=5 playwright/=2 (cc-ci) +env overrides: (none) +``` -**R1 — Six divergent meta loaders (the core drift hazard).** §4's L1–L6: every loader re-`exec()`s -`recipe_meta.py` and cherry-picks its own keys. Adding a key means knowing *which* loader to touch -(or that you must extend the L1 allowlist — `SCREENSHOT` proves people don't, R2). Two conventions -coexist: L1's explicit allowlist vs L3–L6's ad-hoc `ns.get(...)` which silently bypasses it. -*Candidate:* one `harness.meta.load(recipe) -> RecipeMeta` with a declarative key registry -(name, type, default, validator, consumer) as the single source of truth; L1–L6 become lookups -into the one loaded object; the registry also generates §4 of this doc (kills doc drift, R5). +The same dict is embedded in `results.json` under `"customization"`. It is pure presentation — +built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes, +and it honors the HC2 gate) — and never influences a verdict. -**R2 — `SCREENSHOT` is a dead knob.** Fully implemented consumer (`screenshot.py`), documented -hook contract, never reachable: the orchestrator's allowlist omits it, so the dict passed at -`run_recipe_ci.py:1056` can never contain it. Direct evidence of R1. *Candidate:* fix trivially by -adding to the allowlist — or delete the hook path if post-login screenshots aren't wanted; decide -during the restructure. +**Dev-only generic skip.** `CCCI_SKIP_GENERIC=1` (all ops) / `CCCI_SKIP_GENERIC_=1` (one op) +suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no +declarative equivalent (the old `SKIP_GENERIC` meta key is deleted). If the env form is active in +a CI (drone) run, the run prints a loud `!!` warning and the manifest records it. -**R3 — The pytest `meta` fixture sees 4 keys.** `tests/conftest.py:_recipe_meta` loads only -HEALTH_*/timeouts. An overlay test wanting e.g. `EXPECTED_NA` or a recipe constant must re-exec -the file itself. Probably intended minimalism, but it's a third key-set to keep in sync. -*Folds into R1.* +## 8. Restructure outcomes (the review spec's R1–R9) -**R4 — Settings split across three config languages** (§1): recipe_meta keys, file-presence -(`install_steps.sh` existing changes deploy behavior), and run-time env (`CCCI_SKIP_GENERIC*`). -A reviewer asking "what does this recipe customize?" must check all three. *Candidate:* keep the -three surfaces (they serve different actors) but make the run header log a single resolved -"customization manifest" per run: every non-default key + every discovered hook file + every -CCCI_* override, in one block. +How each defect identified in the review spec (commit `76a4b6b` §8) was resolved: -**R5 — Reference-doc drift already happened.** `docs/testing.md` documents 6 meta keys, -`docs/enroll-recipe.md` shows others by example; neither is complete (18 keys exist). This doc is -now complete but handwritten — it will drift too. *Candidate:* generate the key table from the R1 -registry (test asserts doc ⊆ registry). +- **R1 — six divergent meta loaders → RESOLVED.** One registry-backed loader + (`harness/meta.py::load`), the only `exec()` of `recipe_meta.py`. The orchestrator loads once + and passes the `RecipeMeta` down; conftest/lifecycle/deps/canonical all read the one object. +- **R2 — dead `SCREENSHOT` knob → RESOLVED (kept + fixed).** The registry replaced the allowlist + that orphaned it; the orchestrator path now delivers the hook to `screenshot.py` + (proven end-to-end by `tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path`). +- **R3 — 4-key pytest `meta` fixture → RESOLVED.** The fixture returns the full validated + `RecipeMeta`. +- **R4 — three config languages → MITIGATED by the manifest** (§7): the surfaces stay (they serve + different actors), but every run resolves them into one visible block + results key. +- **R5 — reference-doc drift → RESOLVED.** §4's key table is generated from the registry + (`scripts/gen-meta-docs.py`); a unit test fails CI on drift; `testing.md`/`enroll-recipe.md` + point here instead of keeping partial lists. +- **R6 — silent typos → RESOLVED.** Unknown ALL-CAPS keys and type mismatches are hard + `MetaError`s; private constants are underscore-prefixed (exempt). +- **R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` coupling → RESOLVED.** The overlay is + first-class: harness-copied, auto-chaos. The flag is deleted. +- **R8 — zero-user `SKIP_GENERIC` meta key → RESOLVED (deleted).** Env form remains, documented + dev-only, loudly flagged in CI runs (§7). +- **R9 — `recipe_meta.py` is code, not config → REJECTED by decision.** No data/hooks file split: + registry validation gets the value (typed, validated keys) at lower cost; one file per recipe + remains the single config place. The expressiveness need is real (cryptpad derives env from the + per-run domain). -**R6 — No schema validation / silent typos.** Unknown top-level names in `recipe_meta.py` are -ignored, which is load-bearing (recipes keep private constants there: mumble's -`WELCOME_TEXT_MARKER`, `MAX_USERS`). Consequence: misspelling `READY_PROBE` as `READINESS_PROBE` -silently disables the probe — the run goes green with less coverage, the worst failure mode for a -CI harness. *Candidate:* with the R1 registry, warn (not fail) on ALL-CAPS top-level names that -are not registered and not referenced by the recipe's own tests; or namespace private constants -(`_WELCOME_TEXT_MARKER`). - -**R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` implicit coupling.** The overlay only works if -the recipe *also* sets the flag; forgetting it fails the base deploy with an abra -untracked-files error far from the cause. *Candidate:* if `install_steps.sh` exists alongside a -`compose.ccci.yml`, the harness could auto-enable chaos for the base deploy (or at least assert -the flag and fail with a pointed message). - -**R8 — `SKIP_GENERIC` (meta form) has zero users.** Only the env-var form is used, ad hoc. Either -the meta key earns its place (first real user) or it's surface to delete in the restructure. - -**R9 — `recipe_meta.py` is code, not config.** Five keys take callables (`EXTRA_ENV`, -`UPGRADE_EXTRA_ENV`, `READY_PROBE`, `BACKUP_VERIFY`, `SCREENSHOT`), so the file must stay an -`exec()`d Python module — it can't be validated as data, serialized into results, or diffed -declaratively. This is a real expressiveness need (cryptpad derives `SANDBOX_DOMAIN` from the -per-run domain), not an accident. *Candidate if restructuring:* split data keys (TOML-able, -schema-validated) from a `hooks.py` (callables only) — but weigh against the cost of two files -per recipe; the R1 registry gets most of the value without the split. +Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy +post-deploy `setup_custom_tests.sh` machinery and its extra redeploy are deleted); the custom-test +placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface +(§5.6 — `deps` replaces `deps_apps`+`deps_creds`; dead `deployed`/`deployed_app`/`app_domain` +fixtures deleted). ## 9. File / symbol index | Concern | Where | |---|---| -| Orchestrator meta loader (L1, allowlist) | `runner/run_recipe_ci.py:250` `_load_meta` | -| Pytest meta fixture (L2) | `tests/conftest.py` `_recipe_meta` | -| `EXTRA_ENV` loader (L3) | `runner/harness/lifecycle.py:114` `_recipe_extra_env` | -| Boolean-flag loader (L4) | `runner/harness/lifecycle.py:132` `_recipe_meta_flag` | -| `DEPS` loader (L5) | `runner/harness/deps.py:37` `declared_deps` | -| `WARM_CANONICAL` loader (L6) | `runner/harness/canonical.py:36` `is_canonical_enrolled` | -| Overlay/custom/hook discovery + HC2 gate | `runner/harness/discovery.py` | +| THE meta loader + key registry + `HookCtx` + `MetaError` | `runner/harness/meta.py` (`load`, `KEYS`, `check_hook_signature`) | +| Generated key table | `scripts/gen-meta-docs.py` → §4 above (sync pinned by `tests/unit/test_meta.py`) | +| Customization manifest | `runner/harness/manifest.py` (`build`, `render`), printed by `runner/run_recipe_ci.py` | +| Overlay/custom/hook discovery + HC2 gate + placement rule | `runner/harness/discovery.py` | | HC2 allowlist | `tests/repo-local-approved.txt` | | Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` | -| `READY_PROBE` / `CHAOS_BASE_DEPLOY` consumption | `runner/harness/lifecycle.py:516` / `:283` | +| `compose.ccci.yml` auto-copy + auto-chaos | `runner/harness/lifecycle.py` (`provide_ccci_overlay`, `deploy_app`) | +| `READY_PROBE` consumption | `runner/harness/lifecycle.py` (`wait_ready_probes`) | | `EXPECTED_NA` reporting | `runner/harness/results.py` | -| Dead `SCREENSHOT` consumer | `runner/harness/screenshot.py:36`, called `run_recipe_ci.py:1056` | -| Skip-generic logic (meta + env) | `runner/run_recipe_ci.py:285` | -| Worked examples | `tests/ghost/` (overlay+chaos), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV), `tests/lasuite-drive/` (DEPS+OIDC_AT_INSTALL), `tests/immich/` (ops.py seed pattern) | +| `SCREENSHOT` consumer | `runner/harness/screenshot.py` | +| Fixtures (`recipe`/`meta`/`live_app`/`op_state`/`deps`) + F2-11 skip-report | `tests/conftest.py` | +| Skip-generic env logic (dev-only) | `runner/run_recipe_ci.py` (`_skip_generic`) | +| Unit tests pinning all of the above | `tests/unit/test_meta.py`, `test_manifest.py`, `test_discovery*.py` | +| Worked examples | `tests/ghost/` (overlay+compose.ccci.yml), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV, private `_` constants), `tests/lasuite-drive/` (DEPS + install-time OIDC wiring), `tests/immich/` (ops.py seed pattern) | diff --git a/docs/testing.md b/docs/testing.md index ed56a28..7a0092c 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -16,12 +16,13 @@ year from now, this is the one rule that should still hold. ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state scaffolding — just "does this recipe deploy and lifecycle work?" - **Generic must not depend on custom.** A custom test or a custom-tests setup (e.g. SSO/OIDC dep - provisioning) **can never be a precondition for the generic tier to pass.** Concretely: the - orchestrator runs all generic tiers (install → upgrade → backup → restore) against the recipe - **alone, with no deps deployed**, then runs the `setup_custom_tests` step (deps + post-deps - wiring) only after — and a failure there is **isolated** to the custom tier (tests tagged - `@pytest.mark.requires_deps` skip with reason `"deps-not-ready"`; generic tier reports - normally). See `cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics. + provisioning) **can never be a precondition for the generic tier to pass.** Concretely: deps are + provisioned BEFORE the single deploy (so `install_steps.sh` can wire OIDC env into that one + deploy), but a dep-provisioning failure is **isolated** to the custom tier — the recipe still + deploys alone, every generic tier (install → upgrade → backup → restore) runs normally, and + tests tagged `@pytest.mark.requires_deps` skip with reason `"deps-not-ready"` (a counted, + reported skip — F2-11). A deps failure can never fail or block a generic tier. See + `cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics. - **Custom tests are the thoroughness layer — and they cost more to maintain.** They're more thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper scripts, state-management) and *therefore* take more maintenance: an SSO provider's admin API @@ -113,9 +114,11 @@ repo-local /tests/test_.py (upstream-authoritative; gated Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in addition** unless explicitly opted out. -**Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and -additive**: it has no generic equivalent and runs only when present, discovered from both locations -(repo-local gated by the HC2 allowlist). +**Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**: +they have no generic equivalent and run only when present, discovered from both locations +(repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under +`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else +(top-level non-lifecycle files are not discovered). ### Pre-op seed hooks (per-recipe `ops.py`) @@ -127,35 +130,38 @@ etc.). Since the orchestrator owns the op, overlays place their seed in an optio # tests//ops.py from harness import lifecycle -def pre_upgrade(domain, meta): +def pre_upgrade(ctx): # seed a marker before the harness performs the upgrade - lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"]) + lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo upgrade-survives > /path/marker"]) -def pre_backup(domain, meta): +def pre_backup(ctx): # establish a known "original" state before the backup op captures it - lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"]) + lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo original > /path/marker"]) -def pre_restore(domain, meta): +def pre_restore(ctx): # diverge from the backed-up state so a successful restore is observable - lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"]) + lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo mutated > /path/marker"]) ``` The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import -sibling helpers like `kc_admin.py`) and calls `pre_(domain, meta)` immediately before performing -the op. Then `test_.py` asserts the post-op state. See `tests/custom-html/` (volume marker), +sibling helpers like `kc_admin.py`) and calls `pre_(ctx)` immediately before performing the +op — `ctx` is the uniform `HookCtx` every recipe hook receives (`.domain`, `.base_url`, `.meta`, +`.deps`, `.op` — `docs/recipe-customization.md` §4.1). Then `test_.py` asserts the post-op +state. See `tests/custom-html/` (volume marker), `tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db` service) for worked examples. -### Opting out of the generic floor +### Opting out of the generic floor (LOCAL-DEV-ONLY) -The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check -fully replaces the generic's mechanism check) set, in increasing specificity: +The generic runs additively by default and there is **no declarative opt-out** — no recipe can +ship without the floor. For local iteration only (e.g. re-running one tier while developing an +overlay), two env escape hatches exist: - **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide). - **env `CCCI_SKIP_GENERIC_=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op. -- **declarative in `recipe_meta.py`** — `SKIP_GENERIC = ["upgrade"]` (per-op) or `SKIP_GENERIC = ["all"]`. -Opting out is per-recipe and visible in git — not a hidden global. Truthy = `1`/`true`/`yes`/`on`. +Truthy = `1`/`true`/`yes`/`on`. If either is active in a CI (drone) run, the run prints a loud +`!!` warning and the customization manifest records it (`docs/recipe-customization.md` §7). ## Repo-local trust gate (HC2) — default-deny @@ -215,12 +221,14 @@ installs and stays 1. `tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through `lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run. 3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore - divergence), drop `tests//ops.py` with `pre_upgrade/pre_backup/pre_restore(domain, meta)`. + divergence), drop `tests//ops.py` with `pre_upgrade/pre_backup/pre_restore(ctx)`. 4. If the recipe needs install-time setup, add `tests//install_steps.sh`. -5. Set per-recipe knobs (health path, timeouts, opt-out) in `recipe_meta.py`. +5. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`. 6. **Never weaken or skip an assertion to make a run pass** — a red tier is information. -Per-recipe config (`tests//recipe_meta.py`, all optional): +Per-recipe config (`tests//recipe_meta.py`, all optional — the COMPLETE key reference is +the generated table in `docs/recipe-customization.md` §4; unknown keys are hard errors, private +constants are underscore-prefixed): ```python HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") @@ -228,8 +236,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302) DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose) -EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy -SKIP_GENERIC = ["upgrade"] # per-recipe declarative opt-out from generic ops ("all" = every op) +EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy ``` The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run: