cc-ci/docs/recipe-customization.md

# Recipe customization — reference

Status: REFERENCE — describes the customization system as restructured on branch
`restructure/recipe-custom` (the "rcust" restructure). The pre-restructure system and its defects
are documented in this file's history (commit `76a4b6b`, the review spec whose §8 R1–R9 drove the
restructure); §8 below records how each was resolved.

Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md`
(step-by-step enrollment). This doc is the **complete reference** for the two questions those docs
answer only partially:

1. How are custom tests written for a particular recipe?
2. What are ALL the per-recipe CI settings, where do they live, and who reads them?

---

## 1. The three customization surfaces

A recipe customizes its CI through **three distinct mechanisms**:

| Surface | Form | Examples |
|---|---|---|
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |

There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
from all four surfaces is printed at run start as the **customization manifest** and embedded in
`results.json` under `"customization"` (§7) — one block answers "what does this recipe customize?".

## 2. Zero-config baseline

A recipe with **no `tests/<recipe>/` directory at all** still gets the full generic floor:

- deploy base version → INSTALL (generic `assert_serving`: HTTP on `/`, expect 200/301/302)
- chaos-upgrade to PR head → UPGRADE (generic `assert_upgraded`: version label matches head, converged, serving)
- BACKUP (generic `assert_backup_artifact`) — iff the recipe's compose files carry
  `backupbot.backup` labels (auto-detected), else N/A
- RESTORE (generic `assert_restore_healthy`)
- CUSTOM tier: empty (no custom tests discovered)
- teardown

Defaults: `HEALTH_PATH="/"`, `HEALTH_OK=(200,301,302)`, `DEPLOY_TIMEOUT=600`, `HTTP_TIMEOUT=300`.
Everything in this doc is opt-in deviation from that floor. The cardinal invariant
(docs/testing.md §1): the generic floor is **always on** and never depends on custom code;
custom is **additive** by default.

## 3. The per-recipe tree — every file that can exist

Two locations, with precedence and a security gate between them:

- **cc-ci-owned**: `tests/<recipe>/` in this repo (trusted, maintainer-reviewed)
- **repo-local**: the recipe repo's own `tests/` dir (PR-author-controlled → **default-deny**,
  consulted only when the recipe is listed in `tests/repo-local-approved.txt` — gate HC2,
  centralized in `runner/harness/discovery.py`)

```
tests/<recipe>/                      # cc-ci side (repo-local mirrors the same shape)
├── recipe_meta.py                   # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py                     # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py                           # pre_<op>(ctx) seed hooks                    (§5.2)
├── functional/test_*.py             # custom tier: parity ports + recipe-specific (§5.3)
├── playwright/test_*.py             # custom tier: UI flows                       (§5.3)
├── install_steps.sh                 # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── compose.ccci.yml                 # CI-only compose overlay (first-class)       (§5.5)
└── PARITY.md                        # enrollment contract doc (human-read only)
```

**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).

Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):

- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
  generic floor still runs additively alongside.
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
  concept).
- `install_steps.sh`: repo-local > cc-ci, or none.
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
- `recipe_meta.py` and `compose.ccci.yml`: cc-ci only — repo-local recipes cannot set CI settings
  or compose overlays (by design; those surfaces stay maintainer-controlled).

## 4. `recipe_meta.py` — complete settings reference

The single settings file. Plain Python, `exec()`d by the harness in exactly ONE place: the
registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta`. Every consumer — the
orchestrator (which loads once and passes the object down), the pytest `meta` fixture, lifecycle,
deps, canonical, screenshot — reads from that one loaded object.

**Validation (hard errors at load, before any deploy):**

- A key is "set" by a top-level ALL-CAPS assignment or `def`. Unknown ALL-CAPS top-level names
  raise `MetaError` listing the unknown name and the nearest registered key (typo gate —
  misspelling `READY_PROBE` can no longer silently disable the probe).
- Type mismatches raise `MetaError`; callables are accepted only for hook-typed keys.
- **Underscore-prefixed names (`_FOO`) are recipe-private and exempt** — that's where private
  constants live (e.g. mumble's `_WELCOME_TEXT_MARKER`). Lowercase names (helpers/imports) are
  ignored.
- Hook callables must have the registered signature (below); a legacy-signature hook raises a
  `MetaError` naming the migration, never a silent `TypeError` mid-run.

A unit test (`tests/unit/test_meta.py`) loads every `tests/*/recipe_meta.py` through the registry,
so a typo'd key fails at PR time, not at run time.

<!-- META-TABLE-START -->

_This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scripts/gen-meta-docs.py` — do not edit by hand (a unit test pins the sync)._

| Key | Type | Default | Meaning |
|---|---|---|---|
| `HEALTH_PATH` | `str` | `'/'` | Path probed for serving/health checks (deploy wait + generic `assert_serving`). |
| `HEALTH_OK` | `tuple[int]` | `(200, 301, 302)` | Acceptable HTTP status codes for health. |
| `DEPLOY_TIMEOUT` | `int` | `600` | Max seconds to wait for swarm convergence per deploy. |
| `HTTP_TIMEOUT` | `int` | `300` | Max seconds to wait for HTTP health after convergence. |
| `BACKUP_CAPABLE` | `bool` | `None` | Override the backup-tier capability auto-detect (compose `backupbot.backup` labels). `False` forces an intentional skip of the backup/restore rung; `True` forces the tier on; unset = auto-detect. |
| `EXPECTED_NA` | `dict` | `None` | Declare a non-run rung an INTENTIONAL skip: `{rung: reason}` — the level climbs past it; an undeclared non-run rung is *unverified* and blocks the level above it (classification table: machine-docs/DECISIONS.md phase lvl5). Never overrides an exercised pass/fail; the `lint` rung has no escape hatch. |
| `READY_PROBE` | `hook` | `None` | Callable `(ctx) -> [probe, ...]` returning extra readiness probes, run after install AND after upgrade: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}`. |
| `UPGRADE_BASE_VERSION` | `str` | `None` | Exact published tag overriding the upgrade tier's base (default: `recipe_versions[-2]`). |
| `BACKUP_VERIFY` | `hook` | `None` | Callable `(ctx) -> bool` post-backup data-capture check; `False` re-runs the backup (truncated-dump race guard), retried up to 3 attempts. |
| `UPGRADE_EXTRA_ENV` | `dict_or_hook` | `None` | Extra `.env` keys applied after the PR-head checkout, before the chaos redeploy (env that exists only at head). Dict, or callable `(ctx) -> dict`. |
| `EXTRA_ENV` | `dict_or_hook` | `{}` | Extra `.env` keys applied at EVERY deploy (base install AND upgrade old-app). Dict, or callable `(ctx) -> dict` deriving values from the per-run domain (`ctx.domain`). |
| `DEPS` | `list[str]` | `[]` | Dep recipes deployed/provisioned alongside (e.g. `["keycloak"]`); creds land in `$CCCI_DEPS_FILE`. |
| `WARM_CANONICAL` | `bool` | `False` | Enroll the recipe in the warm/canonical app system (docs/warm.md): green cold runs on LATEST advance the canonical snapshot. |
| `SCREENSHOT` | `hook` | `None` | Callable `(page, ctx)` driving Playwright to a safe, credential-free post-login view for the results-card screenshot (default: landing page). |

<!-- META-TABLE-END -->

### 4.1 The uniform hook convention — `HookCtx`

Every recipe callable takes a single `ctx` argument (`harness/meta.py::HookCtx`, frozen):

| Field | Meaning |
|---|---|
| `ctx.domain` | the app's per-run domain |
| `ctx.base_url` | `https://<domain>` |
| `ctx.meta` | the recipe's full `RecipeMeta` |
| `ctx.deps` | provisioned dep creds (`{dep_recipe: entry}`) or `None` |
| `ctx.op` | current lifecycle op (`install`/`upgrade`/`backup`/`restore`) or `None` |

Signatures: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`,
`SCREENSHOT(page, ctx)`, ops.py `pre_<op>(ctx)`. Dict-valued `EXTRA_ENV`/`UPGRADE_EXTRA_ENV`
(non-callable) are still fine — only the callable form takes ctx. The loader enforces the
parameter names at load time (a pre-restructure `(domain)`/`(domain, meta)` hook gets a pointed
`MetaError`, not a mid-run crash).

Worked hook examples: cryptpad (`EXTRA_ENV(ctx)` derives `SANDBOX_DOMAIN` from `ctx.domain`),
mumble (`READY_PROBE(ctx)` TCP voice-port probe, `UPGRADE_EXTRA_ENV(ctx)` adds a head-only compose
overlay), ghost/discourse (`BACKUP_VERIFY(ctx)` dump-capture check).

## 5. Writing custom tests & hooks

### 5.1 Lifecycle overlay assertions — `test_<op>.py`

One pytest file per lifecycle op (`install` / `upgrade` / `backup` / `restore`). The
**orchestrator performs the op exactly once**; the overlay only *asserts* on the resulting state
(HC3 op/assertion split — overlays never deploy, never restore, never mutate). The generic floor
test runs additively against the same state.

Conventions (see `tests/immich/test_backup.py` etc.):
- use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain)
- use the `meta` fixture — the recipe's FULL validated `RecipeMeta` (attribute access)
- use the `op_state` fixture for op context (versions, `snapshot_id`, artifact paths — the
  orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run)
- execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)`

### 5.2 Pre-op seed hooks — `ops.py`

`def pre_<op>(ctx)` callables, imported and called by the orchestrator **before** performing the
op. This is where data gets seeded so the post-op overlay can assert on it:

```python
# tests/immich/ops.py (pattern)
def pre_upgrade(ctx):  _psql(ctx.domain, "INSERT ... 'upgrade-survives'")
def pre_backup(ctx):   _psql(ctx.domain, "INSERT ... 'original'")
def pre_restore(ctx):  _psql(ctx.domain, "DROP TABLE ci_marker")  # damage, restore must undo
```

Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.

### 5.3 Custom tier — `functional/` and `playwright/` ONLY

All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
HC2-approved) repo-local's, additively.

Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.

Tests needing deps use the `deps` fixture (entries expose `.domain` plus the full creds dict) and
carry `@pytest.mark.requires_deps` — when dep provisioning failed they skip with reason
`deps-not-ready` and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit
must not mask an unrun SSO test). Fixtures replace direct `os.environ` reads — after the
restructure no recipe test parses env by hand.

### 5.4 Pre-deploy shell hook — `install_steps.sh`

The ONLY shell hook. Runs after `abra app new` + `EXTRA_ENV` application + secret generation,
**before** the single base deploy. For setup that must precede the first deploy: writing extra
config files into the recipe checkout, editing `.env` beyond simple key=val, and — for recipes
with `DEPS` — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the
deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy
redeploy hook).

Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and —
when `DEPS` is declared — `CCCI_DEPS_FILE` (jq-readable JSON of dep creds/URLs; see
lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware:
`RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run `ABRA_DIR` since the
concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).

Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic
install — a correct reported outcome, not a harness error.

### 5.5 CI-only compose overlay — `compose.ccci.yml`

**First-class:** if `tests/<recipe>/compose.ccci.yml` exists, the harness itself copies it into
the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses `--chaos` for
that deploy (the untracked file would otherwise trip abra's clean-tree gate). No
`install_steps.sh` copy boilerplate, no flag to remember (the old `CHAOS_BASE_DEPLOY` ⇄ overlay
coupling is gone). The overlay is cc-ci-owned only.

Policy unchanged: overlays are a minimal, justified fallback (ghost's is a 15m `start_period`
grace — a literal, because abra validates `start_period` before env substitution). Reference the
overlay from `EXTRA_ENV`'s `COMPOSE_FILE` as usual. Users: ghost, discourse.

### 5.6 Environment & fixture contract (what custom code can read)

Pytest fixtures (`tests/conftest.py` — the single fixture file):

| Fixture | Yields |
|---|---|
| `recipe` | the recipe name (`$RECIPE`) |
| `meta` | the FULL validated `RecipeMeta` (single loader) |
| `live_app` | the shared deployment's domain (asserts it exists) |
| `op_state` | the orchestrator's op-context dict (skips cleanly outside a run) |
| `deps` | `{dep_recipe: entry}` — entries expose `.domain` + full SSO creds |

Environment (hooks/shell, and approved repo-local code):

| Var | Set for | Meaning |
|---|---|---|
| `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain |
| `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` |
| `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path |
| `CCCI_OP_STATE_FILE` | overlay tests (via `op_state`) | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | `install_steps.sh` + harness | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier (via `requires_deps`) | gate SSO tests, skip-with-reason |

## 6. Run-model context (what the settings plug into)

One deploy chain per run (full detail: `docs/testing.md` §2):

```
[DEPS? provision deps FIRST → $CCCI_DEPS_FILE]
deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh;
             compose.ccci.yml auto-copied + auto-chaos)
  → INSTALL tier   (READY_PROBE; generic + overlay asserts)
  → pre_upgrade(ctx) → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
  → UPGRADE tier   (READY_PROBE; version-label == head_ref)
  → pre_backup(ctx) → backup       (BACKUP_CAPABLE; BACKUP_VERIFY)
  → BACKUP tier
  → pre_restore(ctx) → restore
  → RESTORE tier
  → CUSTOM tier    (functional/ + playwright/; deps via the `deps` fixture)
  → SCREENSHOT (best-effort, never affects the verdict)
  → teardown (deps LAST)
```

Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't
count); the per-run counter file is keyed by run since the concurrency restructure.

## 7. Local iteration, the manifest, and the dev-only escape hatch

```
RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
  STAGES=install,upgrade,backup,restore,custom \
  cc-ci-run runner/run_recipe_ci.py
```

(`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.)

**Customization manifest.** Every run prints, right after meta load + discovery, one block:

```
===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci)
env overrides: (none)
```

The same dict is embedded in `results.json` under `"customization"`. It is pure presentation —
built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes,
and it honors the HC2 gate) — and never influences a verdict.

**Dev-only generic skip.** `CCCI_SKIP_GENERIC=1` (all ops) / `CCCI_SKIP_GENERIC_<OP>=1` (one op)
suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no
declarative equivalent (the old `SKIP_GENERIC` meta key is deleted). If the env form is active in
a CI (drone) run, the run prints a loud `!!` warning and the manifest records it.

## 8. Restructure outcomes (the review spec's R1–R9)

How each defect identified in the review spec (commit `76a4b6b` §8) was resolved:

- **R1 — six divergent meta loaders → RESOLVED.** One registry-backed loader
  (`harness/meta.py::load`), the only `exec()` of `recipe_meta.py`. The orchestrator loads once
  and passes the `RecipeMeta` down; conftest/lifecycle/deps/canonical all read the one object.
- **R2 — dead `SCREENSHOT` knob → RESOLVED (kept + fixed).** The registry replaced the allowlist
  that orphaned it; the orchestrator path now delivers the hook to `screenshot.py`
  (proven end-to-end by `tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path`).
- **R3 — 4-key pytest `meta` fixture → RESOLVED.** The fixture returns the full validated
  `RecipeMeta`.
- **R4 — three config languages → MITIGATED by the manifest** (§7): the surfaces stay (they serve
  different actors), but every run resolves them into one visible block + results key.
- **R5 — reference-doc drift → RESOLVED.** §4's key table is generated from the registry
  (`scripts/gen-meta-docs.py`); a unit test fails CI on drift; `testing.md`/`enroll-recipe.md`
  point here instead of keeping partial lists.
- **R6 — silent typos → RESOLVED.** Unknown ALL-CAPS keys and type mismatches are hard
  `MetaError`s; private constants are underscore-prefixed (exempt).
- **R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` coupling → RESOLVED.** The overlay is
  first-class: harness-copied, auto-chaos. The flag is deleted.
- **R8 — zero-user `SKIP_GENERIC` meta key → RESOLVED (deleted).** Env form remains, documented
  dev-only, loudly flagged in CI runs (§7).
- **R9 — `recipe_meta.py` is code, not config → REJECTED by decision.** No data/hooks file split:
  registry validation gets the value (typed, validated keys) at lower cost; one file per recipe
  remains the single config place. The expressiveness need is real (cryptpad derives env from the
  per-run domain).

Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy
post-deploy `setup_custom_tests.sh` machinery and its extra redeploy are deleted); the custom-test
placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface
(§5.6 — `deps` replaces `deps_apps`+`deps_creds`; dead `deployed`/`deployed_app`/`app_domain`
fixtures deleted).

## 9. File / symbol index

| Concern | Where |
|---|---|
| THE meta loader + key registry + `HookCtx` + `MetaError` | `runner/harness/meta.py` (`load`, `KEYS`, `check_hook_signature`) |
| Generated key table | `scripts/gen-meta-docs.py` → §4 above (sync pinned by `tests/unit/test_meta.py`) |
| Customization manifest | `runner/harness/manifest.py` (`build`, `render`), printed by `runner/run_recipe_ci.py` |
| Overlay/custom/hook discovery + HC2 gate + placement rule | `runner/harness/discovery.py` |
| HC2 allowlist | `tests/repo-local-approved.txt` |
| Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` |
| `compose.ccci.yml` auto-copy + auto-chaos | `runner/harness/lifecycle.py` (`provide_ccci_overlay`, `deploy_app`) |
| `READY_PROBE` consumption | `runner/harness/lifecycle.py` (`wait_ready_probes`) |
| `EXPECTED_NA` reporting | `runner/harness/results.py` |
| `SCREENSHOT` consumer | `runner/harness/screenshot.py` |
| Fixtures (`recipe`/`meta`/`live_app`/`op_state`/`deps`) + F2-11 skip-report | `tests/conftest.py` |
| Skip-generic env logic (dev-only) | `runner/run_recipe_ci.py` (`_skip_generic`) |
| Unit tests pinning all of the above | `tests/unit/test_meta.py`, `test_manifest.py`, `test_discovery*.py` |
| Worked examples | `tests/ghost/` (overlay+compose.ccci.yml), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV, private `_` constants), `tests/lasuite-drive/` (DEPS + install-time OIDC wiring), `tests/immich/` (ops.py seed pattern) |