cc-ci/docs/recipe-customization.md

Recipe customization — reference

Status: REFERENCE — describes the customization system as restructured on branch restructure/recipe-custom (the "rcust" restructure). The pre-restructure system and its defects are documented in this file's history (commit 76a4b6b, the review spec whose §8 R1–R9 drove the restructure); §8 below records how each was resolved.

Companion docs: docs/testing.md (test architecture / tier semantics), docs/enroll-recipe.md (step-by-step enrollment). This doc is the complete reference for the two questions those docs answer only partially:

1. How are custom tests written for a particular recipe?
2. What are ALL the per-recipe CI settings, where do they live, and who reads them?

---

1. The three customization surfaces

A recipe customizes its CI through three distinct mechanisms:

Surface	Form	Examples
Declarative settings	Python assignments in tests/<recipe>/recipe_meta.py	DEPLOY_TIMEOUT = 1500, UPGRADE_BASE_VERSION = "2.3.1+..."
Code hooks	Callables in recipe_meta.py, ops.py functions, one shell hook	def READY_PROBE(ctx): ..., pre_upgrade(ctx), install_steps.sh
File presence	A file existing at a discovered path changes behavior	test_upgrade.py overlay, custom/test_*.py, compose.ccci.yml

There is additionally a fourth, operator-facing, local-dev-only surface: environment variables (CCCI_SKIP_GENERIC*) that suppress the generic floor at run time (§7). Whatever a run resolves from all four surfaces is printed at run start as the customization manifest and embedded in results.json under "customization" (§7) — one block answers "what does this recipe customize?".

2. Zero-config baseline

A recipe with no tests/<recipe>/ directory at all still gets the full generic floor:

• deploy base version → INSTALL (generic assert_serving: HTTP on /, expect 200/301/302)
• chaos-upgrade to PR head → UPGRADE (generic assert_upgraded: version label matches head, converged, serving)
• BACKUP (generic assert_backup_artifact) — iff the recipe's compose files carry backupbot.backup labels (auto-detected), else N/A
• RESTORE (generic assert_restore_healthy)
• CUSTOM tier: empty (no custom tests discovered)
• teardown

Defaults: HEALTH_PATH="/", HEALTH_OK=(200,301,302), DEPLOY_TIMEOUT=600, HTTP_TIMEOUT=300. Everything in this doc is opt-in deviation from that floor. The cardinal invariant (docs/testing.md §1): the generic floor is always on and never depends on custom code; custom is additive by default.

3. The per-recipe tree — every file that can exist

Two locations, with precedence and a security gate between them:

• cc-ci-owned: tests/<recipe>/ in this repo (trusted, maintainer-reviewed)
• repo-local: the recipe repo's own tests/ dir (PR-author-controlled → default-deny, consulted only when the recipe is listed in tests/repo-local-approved.txt — gate HC2, centralized in runner/harness/discovery.py)

tests/<recipe>/                      # cc-ci side (repo-local mirrors the same shape)
├── recipe_meta.py                   # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py                     # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py                           # pre_<op>(ctx) seed hooks                    (§5.2)
├── custom/test_*.py                 # custom tier: parity ports + recipe-specific + UI flows (§5.3)
├── install_steps.sh                 # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── compose.ccci.yml                 # CI-only ENVIRONMENTAL compose overlay (all deploys) (§5.5)
├── previous/                        # version-specific base-only repair (optional)  (§5.5b)
│   ├── compose.previous.yml         #   minimal compose to deploy the previous version
│   └── VERSION                      #   the published version it targets (version-guard)
└── PARITY.md                        # enrollment contract doc (human-read only)

Placement rule (custom tests): ALL custom-tier tests live under canonical custom/. Deprecated functional/ and playwright/ aliases are still discovered with a loud warning so coverage is not silently lost while recipe trees migrate. A top-level test_*.py is a lifecycle overlay (test_<op>.py) and nothing else — top-level non-lifecycle files are NOT discovered (discovery.custom_tests; the lifecycle-name exclusion stays as a safety net so a misfiled test_<op>.py can never double-run).

Precedence (machine-docs/DECISIONS.md, implemented in discovery.py):

• lifecycle overlay test_<op>.py: repo-local wins over cc-ci (same-name collision); the generic floor still runs additively alongside.
• custom tier (custom/, plus deprecated alias dirs during migration): ALL run, from both locations (no collision concept).
• install_steps.sh: repo-local > cc-ci, or none.
• ops.py pre-op hook: cc-ci wins; repo-local consulted only if approved.
• recipe_meta.py and compose.ccci.yml: cc-ci only — repo-local recipes cannot set CI settings or compose overlays (by design; those surfaces stay maintainer-controlled).

4. recipe_meta.py — complete settings reference

The single settings file. Plain Python, exec()d by the harness in exactly ONE place: the registry-backed loader runner/harness/meta.py::load(recipe) -> RecipeMeta. Every consumer — the orchestrator (which loads once and passes the object down), the pytest meta fixture, lifecycle, deps, canonical, screenshot — reads from that one loaded object.

Validation (hard errors at load, before any deploy):

• A key is "set" by a top-level ALL-CAPS assignment or def. Unknown ALL-CAPS top-level names raise MetaError listing the unknown name and the nearest registered key (typo gate — misspelling READY_PROBE can no longer silently disable the probe).
• Type mismatches raise MetaError; callables are accepted only for hook-typed keys.
• Underscore-prefixed names (_FOO) are recipe-private and exempt — that's where private constants live (e.g. mumble's _WELCOME_TEXT_MARKER). Lowercase names (helpers/imports) are ignored.
• Hook callables must have the registered signature (below); a legacy-signature hook raises a MetaError naming the migration, never a silent TypeError mid-run.

A unit test (tests/unit/test_meta.py) loads every tests/*/recipe_meta.py through the registry, so a typo'd key fails at PR time, not at run time.

This table is GENERATED from the runner/harness/meta.py KEYS registry by scripts/gen-meta-docs.py — do not edit by hand (a unit test pins the sync).

Key	Type	Default	Meaning
HEALTH_PATH	str	'/'	Path probed for serving/health checks (deploy wait + generic assert_serving).
HEALTH_OK	tuple[int]	(200, 301, 302)	Acceptable HTTP status codes for health.
DEPLOY_TIMEOUT	int	600	Max seconds to wait for swarm convergence per deploy.
HTTP_TIMEOUT	int	300	Max seconds to wait for HTTP health after convergence.
BACKUP_CAPABLE	bool	None	Override the backup-tier capability auto-detect (compose backupbot.backup labels). False forces an intentional skip of the backup/restore rung; True forces the tier on; unset = auto-detect.
EXPECTED_NA	dict	None	Declare a non-run rung an INTENTIONAL skip: {rung: reason} — the level climbs past it; an undeclared non-run rung is unverified and blocks the level above it (classification table: machine-docs/DECISIONS.md phase lvl5). Never overrides an exercised pass/fail; the lint rung has no escape hatch. Declaring upgrade also suppresses the upgrade-tier BASE deploy — the single deploy is the PR head itself — for recipes whose published versions exist but are genuinely undeployable (phase bsky).
READY_PROBE	hook	None	Callable (ctx) -> [probe, ...] returning extra readiness probes, run after install AND after upgrade: HTTP {host, path, ok} or TCP {tcp_host, tcp_port, stable}.
UPGRADE_BASE_VERSION	str	None	Optional explicit override pinning the upgrade tier's base to an exact published tag (rare; for a PR that adds a version above the newest tag). When unset (the norm) the base is resolved DYNAMICALLY (phase prevb): last-green (warm canonical) → target-branch (main) tip → else skip. See run_recipe_ci.resolve_upgrade_base + DECISIONS.
BACKUP_VERIFY	hook	None	Callable (ctx) -> bool post-backup data-capture check; False re-runs the backup (truncated-dump race guard), retried up to 3 attempts.
UPGRADE_EXTRA_ENV	dict_or_hook	None	Extra .env keys applied after the PR-head checkout, before the chaos redeploy (env that exists only at head). Dict, or callable (ctx) -> dict.
EXTRA_ENV	dict_or_hook	{}	Extra .env keys applied at EVERY deploy (base install AND upgrade old-app). Dict, or callable (ctx) -> dict deriving values from the per-run domain (ctx.domain).
DEPS	list[str]	[]	Dep recipes deployed/provisioned alongside (e.g. ["keycloak"]); creds land in $CCCI_DEPS_FILE.
WARM_CANONICAL	bool	False	Enroll the recipe in the warm/canonical app system (docs/warm.md): green cold runs on LATEST advance the canonical snapshot.
SCREENSHOT	hook	None	Callable (page, ctx) driving Playwright to a safe, credential-free post-login view for the results-card screenshot (default: landing page).
UPGRADE_SECRET_PREP	hook	None	Callable (ctx) invoked after UPGRADE_EXTRA_ENV env_set but before abra secret generate --all in the upgrade path. Use to pre-insert secrets that generate --all would produce with wrong format (e.g. when the .env.sample spec is commented out).

4.1 The uniform hook convention — HookCtx

Every recipe callable takes a single ctx argument (harness/meta.py::HookCtx, frozen):

Field	Meaning
ctx.domain	the app's per-run domain
ctx.base_url	https://<domain>
ctx.meta	the recipe's full RecipeMeta
ctx.deps	provisioned dep creds ({dep_recipe: entry}) or None
ctx.op	current lifecycle op (install/upgrade/backup/restore) or None

Signatures: EXTRA_ENV(ctx), UPGRADE_EXTRA_ENV(ctx), READY_PROBE(ctx), BACKUP_VERIFY(ctx), SCREENSHOT(page, ctx), ops.py pre_<op>(ctx). Dict-valued EXTRA_ENV/UPGRADE_EXTRA_ENV (non-callable) are still fine — only the callable form takes ctx. The loader enforces the parameter names at load time (a pre-restructure (domain)/(domain, meta) hook gets a pointed MetaError, not a mid-run crash).

Worked hook examples: cryptpad (EXTRA_ENV(ctx) derives SANDBOX_DOMAIN from ctx.domain), mumble (READY_PROBE(ctx) TCP voice-port probe, UPGRADE_EXTRA_ENV(ctx) adds a head-only compose overlay), ghost/discourse (BACKUP_VERIFY(ctx) dump-capture check).

5. Writing custom tests & hooks

5.1 Lifecycle overlay assertions — test_<op>.py

One pytest file per lifecycle op (install / upgrade / backup / restore). The orchestrator performs the op exactly once; the overlay only asserts on the resulting state (HC3 op/assertion split — overlays never deploy, never restore, never mutate). The generic floor test runs additively against the same state.

Conventions (see tests/immich/test_backup.py etc.):

• use the live_app fixture (asserts CCCI_APP_DOMAIN is set, yields the domain)
• use the meta fixture — the recipe's FULL validated RecipeMeta (attribute access)
• use the op_state fixture for op context (versions, snapshot_id, artifact paths — the orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run)
• execute in-container checks via harness.lifecycle.exec_in_app(domain, service, cmd)

5.2 Pre-op seed hooks — ops.py

def pre_<op>(ctx) callables, imported and called by the orchestrator before performing the op. This is where data gets seeded so the post-op overlay can assert on it:

# tests/immich/ops.py (pattern)
def pre_upgrade(ctx):  _psql(ctx.domain, "INSERT ... 'upgrade-survives'")
def pre_backup(ctx):   _psql(ctx.domain, "INSERT ... 'original'")
def pre_restore(ctx):  _psql(ctx.domain, "DROP TABLE ci_marker")  # damage, restore must undo

Seed → op → assert is the whole pattern: pre_backup writes a marker, the orchestrator backs up, pre_restore destroys it, the orchestrator restores, test_restore.py asserts the marker is back.

5.3 Custom tier — canonical custom/

All custom-tier tests live under tests/<recipe>/custom/ (discovery: discovery.custom_tests; the placement rule, §3). Deprecated functional/ and playwright/ dirs are still recognized with a warning during the migration window. Custom tests run in the CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if HC2-approved) repo-local's, additively.

Enrollment contract (docs/enroll-recipe.md): ≥2 NEW custom tests beyond ports of existing upstream checks; ported tests carry SOURCE: comments. Browser-driven custom tests get the shared browser/harness helpers (harness.browser); SSO recipes get harness.sso (setup_keycloak_realm — idempotent, oidc_password_grant — provider-pluggable). The documented import toolbox for custom tests is from harness import lifecycle, sso, browser.

Tests needing deps use the deps fixture (entries expose .domain plus the full creds dict) and carry @pytest.mark.requires_deps — when dep provisioning failed they skip with reason deps-not-ready and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit must not mask an unrun SSO test). Fixtures replace direct os.environ reads — after the restructure no recipe test parses env by hand.

5.4 Pre-deploy shell hook — install_steps.sh

The ONLY shell hook. Runs after abra app new + EXTRA_ENV application + secret generation, before the single base deploy. For setup that must precede the first deploy: writing extra config files into the recipe checkout, editing .env beyond simple key=val, and — for recipes with DEPS — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy redeploy hook).

Env contract: CCCI_APP_DOMAIN, CCCI_RECIPE, CCCI_APP_ENV (path to the app's .env), and — when DEPS is declared — CCCI_DEPS_FILE (jq-readable JSON of dep creds/URLs; see lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware: RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}" (per-run ABRA_DIR since the concurrency restructure — a hardcoded ~/.abra writes to the wrong tree).

Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic install — a correct reported outcome, not a harness error.

5.5 CI-only compose overlay — compose.ccci.yml

First-class: if tests/<recipe>/compose.ccci.yml exists, the harness itself copies it into the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses --chaos for that deploy (the untracked file would otherwise trip abra's clean-tree gate). No install_steps.sh copy boilerplate, no flag to remember (the old CHAOS_BASE_DEPLOY ⇄ overlay coupling is gone). The overlay is cc-ci-owned only.

Policy (phase prevb): compose.ccci.yml is ENVIRONMENTAL-only — node-reality tweaks that must apply to EVERY deploy including the PR head (e.g. ghost's 15m start_period grace — a literal, because abra validates start_period before env substitution; discourse's order: stop-first for the memory-tight upgrade crossover). It MUST NOT carry version-specific image pins or service add/drop — those leak onto the head and mask the change under test. Version-specific base repairs go in previous/ (§5.5b). Reference the overlay from EXTRA_ENV's COMPOSE_FILE as usual.

5.5b Previous-version base repair — tests/<recipe>/previous/

Prefer NOT to use this — it is a last resort. The mechanism exists so that, when updating a recipe's tests, you can bring up a previous base that won't deploy as-published. But reach for it only after the dynamic base (last-green → main-tip) has genuinely failed to come up. Every previous/ you add re-introduces the per-version patching treadmill the dynamic base was designed to remove, so the bar is "the base will not deploy any other way." Most recipes — including discourse, the case that motivated this — need NONE. When in doubt, don't add one.

Optional. The MINIMAL config to deploy the previous (last-green) version when it can't deploy as-published (e.g. an image relocation bitnami/* → bitnamilegacy/*, or an era-specific service/env). Applied to the base deploy ONLY and stripped before the head redeploy, so the PR head runs UNMODIFIED.

• Layout: tests/<recipe>/previous/compose.previous.yml (+ a one-line previous/VERSION marker declaring the published version it targets). Appended to the base deploy's COMPOSE_FILE.
• Version-guarded: applied only when the resolved base equals previous/VERSION. On a main-tip (ref) base or a version mismatch it is skipped and flagged stale (previous/ targets X, base is Y — remove it). After an upgrade PR merges (new last-green), remove the now-stale folder — keep it to ~one version, never an accumulating pile.
• Keep it minimal and add one only where necessary. Most recipes (incl. discourse) need NONE — the dynamic base (last-green/main-tip) deploys clean. Symbols: lifecycle.previous_status / provide_previous_overlay / remove_previous_overlay.

5.6 Environment & fixture contract (what custom code can read)

Pytest fixtures (tests/conftest.py — the single fixture file):

Fixture	Yields
recipe	the recipe name ($RECIPE)
meta	the FULL validated RecipeMeta (single loader)
live_app	the shared deployment's domain (asserts it exists)
op_state	the orchestrator's op-context dict (skips cleanly outside a run)
deps	{dep_recipe: entry} — entries expose .domain + full SSO creds

Environment (hooks/shell, and approved repo-local code):

Var	Set for	Meaning
CCCI_APP_DOMAIN	all tests + hooks	the app's per-run domain
CCCI_BASE_URL	approved repo-local code	https://<domain>
CCCI_RECIPE, CCCI_APP_ENV	install_steps.sh	recipe name, app .env path
CCCI_OP_STATE_FILE	overlay tests (via op_state)	JSON op context (versions, artifacts)
CCCI_DEPS_FILE	install_steps.sh + harness	JSON dep creds dict
CCCI_DEPS_READY / CCCI_DEPS_NOT_READY_REASON	custom tier (via requires_deps)	gate SSO tests, skip-with-reason

6. Run-model context (what the settings plug into)

One deploy chain per run (full detail: docs/testing.md §2):

[DEPS? provision deps FIRST → $CCCI_DEPS_FILE]
deploy BASE (dynamic: last-green → main-tip → skip, or UPGRADE_BASE_VERSION override; EXTRA_ENV;
             install_steps.sh; compose.ccci.yml [environmental] auto-copied + auto-chaos;
             tests/<recipe>/previous/ [version-specific, base-ONLY] applied if it matches the base)
  → INSTALL tier   (READY_PROBE; generic + overlay asserts)
  → pre_upgrade(ctx) → strip previous/ + chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
                       → reconcile stack to head compose (prune services the head dropped)
  → UPGRADE tier   (READY_PROBE; version-label == head_ref)
  → pre_backup(ctx) → backup       (BACKUP_CAPABLE; BACKUP_VERIFY)
  → BACKUP tier
  → pre_restore(ctx) → restore
  → RESTORE tier
  → CUSTOM tier    (custom/; deps via the `deps` fixture)
  → SCREENSHOT (best-effort, never affects the verdict)
  → teardown (deps LAST)

Deploy-count guard (DG4.1): exactly 1 + len(DEPS) deploys per run (chaos redeploys don't count); the per-run counter file is keyed by run since the concurrency restructure.

7. Local iteration, the manifest, and the dev-only escape hatch

RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
  STAGES=install,upgrade,backup,restore,custom \
  cc-ci-run runner/run_recipe_ci.py

(docs/enroll-recipe.md §5 for the full loop, including dep teardown caveats.)

Customization manifest. Every run prints, right after meta load + discovery, one block:

===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: custom/=7 (cc-ci)
env overrides: (none)

The same dict is embedded in results.json under "customization". It is pure presentation — built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes, and it honors the HC2 gate) — and never influences a verdict.

Dev-only generic skip. CCCI_SKIP_GENERIC=1 (all ops) / CCCI_SKIP_GENERIC_<OP>=1 (one op) suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no declarative equivalent (the old SKIP_GENERIC meta key is deleted). If the env form is active in a CI (drone) run, the run prints a loud !! warning and the manifest records it.

8. Restructure outcomes (the review spec's R1–R9)

How each defect identified in the review spec (commit 76a4b6b §8) was resolved:

• R1 — six divergent meta loaders → RESOLVED. One registry-backed loader (harness/meta.py::load), the only exec() of recipe_meta.py. The orchestrator loads once and passes the RecipeMeta down; conftest/lifecycle/deps/canonical all read the one object.
• R2 — dead SCREENSHOT knob → RESOLVED (kept + fixed). The registry replaced the allowlist that orphaned it; the orchestrator path now delivers the hook to screenshot.py (proven end-to-end by tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path).
• R3 — 4-key pytest meta fixture → RESOLVED. The fixture returns the full validated RecipeMeta.
• R4 — three config languages → MITIGATED by the manifest (§7): the surfaces stay (they serve different actors), but every run resolves them into one visible block + results key.
• R5 — reference-doc drift → RESOLVED. §4's key table is generated from the registry (scripts/gen-meta-docs.py); a unit test fails CI on drift; testing.md/enroll-recipe.md point here instead of keeping partial lists.
• R6 — silent typos → RESOLVED. Unknown ALL-CAPS keys and type mismatches are hard MetaErrors; private constants are underscore-prefixed (exempt).
• R7 — compose.ccci.yml ⇄ CHAOS_BASE_DEPLOY coupling → RESOLVED. The overlay is first-class: harness-copied, auto-chaos. The flag is deleted.
• R8 — zero-user SKIP_GENERIC meta key → RESOLVED (deleted). Env form remains, documented dev-only, loudly flagged in CI runs (§7).
• R9 — recipe_meta.py is code, not config → REJECTED by decision. No data/hooks file split: registry validation gets the value (typed, validated keys) at lower cost; one file per recipe remains the single config place. The expressiveness need is real (cryptpad derives env from the per-run domain).

Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy post-deploy setup_custom_tests.sh machinery and its extra redeploy are deleted); the custom-test placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface (§5.6 — deps replaces deps_apps+deps_creds; dead deployed/deployed_app/app_domain fixtures deleted).

9. File / symbol index

Concern	Where
THE meta loader + key registry + HookCtx + MetaError	runner/harness/meta.py (load, KEYS, check_hook_signature)
Generated key table	scripts/gen-meta-docs.py → §4 above (sync pinned by tests/unit/test_meta.py)
Customization manifest	runner/harness/manifest.py (build, render), printed by runner/run_recipe_ci.py
Overlay/custom/hook discovery + HC2 gate + placement rule	runner/harness/discovery.py
HC2 allowlist	tests/repo-local-approved.txt
Generic assertions + BACKUP_CAPABLE detect	runner/harness/generic.py
compose.ccci.yml auto-copy + auto-chaos	runner/harness/lifecycle.py (provide_ccci_overlay, deploy_app)
Dynamic upgrade base (last-green → main-tip → skip)	runner/run_recipe_ci.py (resolve_upgrade_base, BasePlan); runner/harness/lifecycle.py (recipe_branch_commit)
previous/ discovery + version-guard + base-only apply + head strip	runner/harness/lifecycle.py (previous_status, provide/remove_previous_overlay); tests/unit/test_previous.py
READY_PROBE consumption	runner/harness/lifecycle.py (wait_ready_probes)
EXPECTED_NA reporting	runner/harness/results.py
SCREENSHOT consumer	runner/harness/screenshot.py
Fixtures (recipe/meta/live_app/op_state/deps) + F2-11 skip-report	tests/conftest.py
Skip-generic env logic (dev-only)	runner/run_recipe_ci.py (_skip_generic)
Unit tests pinning all of the above	tests/unit/test_meta.py, test_manifest.py, test_discovery*.py
Worked examples	tests/ghost/ (overlay+compose.ccci.yml), tests/mumble/ (TCP probe, UPGRADE_EXTRA_ENV, private _ constants), tests/lasuite-drive/ (DEPS + install-time OIDC wiring), tests/immich/ (ops.py seed pattern)