recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: P6 — rewrite customization docs to the restructured end state (rcust)
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse Source 
recipe-customization.md: review spec -> reference. Single registry-backed loader + validation
rules + HookCtx convention (§4); generated key table kept byte-identical (sync test); §5 end-state
shape (op_state/deps fixtures, ctx ops.py, placement rule, first-class compose.ccci.yml, no
setup_custom_tests.sh); §7 manifest block + dev-only CCCI_SKIP_GENERIC*; §8 rewritten as
restructure outcomes (R1/R2/R3/R5/R6/R7/R8 resolved + how, R4 mitigated by manifest, R9
rejected-by-decision); §9 index updated to the new symbols.

testing.md: install-time deps isolation replaces the setup_custom_tests step in the invariant
(generic still never depends on custom — failure isolation via requires_deps/F2-11); ops.py
example to pre_<op>(ctx); placement rule; generic opt-out now documented LOCAL-DEV-ONLY env with
CI !! warning (declarative SKIP_GENERIC gone); partial key list points at the generated table.

enroll-recipe.md: tree + worked examples updated (lasuite-docs install-time OIDC wiring +
install_steps.sh; mumble post-F2-14c shape — UPGRADE_EXTRA_ENV native overlay, private _
constants, no CHAOS_BASE_DEPLOY); deps fixture (entry.domain) replaces deps_apps; ctx hook
signatures; compose.ccci.yml first-class bullet; key list points at the generated table.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-10 19:07:41 +00:00
parent 68954be53e
commit da558ca946
3 changed files with 268 additions and 272 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
docs/recipe-customization.md
View File

@ -1,8 +1,9 @@
# Recipe customization — review spec
# Recipe customization — reference
Status: REVIEW SPEC — describes the customization surface as it exists today (main), written so
the structure can be reviewed and potentially restructured. §8 lists known limitations and
restructuring candidates; everything before it is purely descriptive.
Status: REFERENCE — describes the customization system as restructured on branch
`restructure/recipe-custom` (the "rcust" restructure). The pre-restructure system and its defects
are documented in this file's history (commit `76a4b6b`, the review spec whose §8 R1–R9 drove the
restructure); §8 below records how each was resolved.
Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md`
(step-by-step enrollment). This doc is the **complete reference** for the two questions those docs
@ -15,17 +16,18 @@ answer only partially:
## 1. The three customization surfaces
A recipe customizes its CI through **three distinct mechanisms** (worth noticing for the
restructure review — they are three different config languages):
A recipe customizes its CI through **three distinct mechanisms**:
| Surface | Form | Examples |
|---|---|---|
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, shell hooks | `def READY_PROBE(domain): ...`, `pre_upgrade()`, `install_steps.sh` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
There is additionally a fourth, operator-facing surface: **environment variables**
(`CCCI_SKIP_GENERIC*`) that override declarative settings at run time (§4.4).
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
from all four surfaces is printed at run start as the **customization manifest** and embedded in
`results.json` under `"customization"` (§7) — one block answers "what does this recipe customize?".
## 2. Zero-config baseline
@ -55,53 +57,53 @@ Two locations, with precedence and a security gate between them:
```
tests/<recipe>/ # cc-ci side (repo-local mirrors the same shape)
├── recipe_meta.py # ALL declarative settings + meta callables (§4)
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py # pre_<op>(domain, meta) seed hooks (§5.2)
├── test_*.py # custom-tier tests (top-level, cross-cutting)(§5.3)
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
├── playwright/test_*.py # custom tier: UI flows (§5.3)
├── install_steps.sh # pre-deploy shell hook (§5.4)
├── setup_custom_tests.sh # deps/OIDC credential wiring hook (§5.5)
├── compose.ccci.yml # CI-only compose overlay (via install_steps) (§5.6)
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
└── PARITY.md # enrollment contract doc (human-read only)
```
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
generic floor still runs additively alongside.
- custom tier `test_*.py`: **ALL** run, from both locations (no collision concept).
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
concept).
- `install_steps.sh`: repo-local > cc-ci, or none.
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
- `recipe_meta.py`: cc-ci only — repo-local recipes cannot set CI settings (by design; the
settings surface stays maintainer-controlled).
- `recipe_meta.py` and `compose.ccci.yml`: cc-ci only — repo-local recipes cannot set CI settings
or compose overlays (by design; those surfaces stay maintainer-controlled).
## 4. `recipe_meta.py` — complete settings reference
The single settings file. Plain Python, `exec()`d by the harness (trusted, in-repo). A key is "set"
by a top-level assignment or `def`. Unknown names are ignored silently (a recipe may keep private
constants here, e.g. mumble's `WELCOME_TEXT_MARKER` — but see §8 R6: typos in real key names are
also silently ignored).
The single settings file. Plain Python, `exec()`d by the harness in exactly ONE place: the
registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta`. Every consumer — the
orchestrator (which loads once and passes the object down), the pytest `meta` fixture, lifecycle,
deps, canonical, screenshot — reads from that one loaded object.
**Loader column legend** — this is the structural finding for the review (§8 R1). There is no
single loader; six independent code paths each `exec()` the file and pick out their own keys:
**Validation (hard errors at load, before any deploy):**
| # | Loader | Keys it sees |
|---|---|---|
| L1 | `runner/run_recipe_ci.py:_load_meta` (orchestrator) | 4 base + explicit 8-key allowlist |
| L2 | `tests/conftest.py:_recipe_meta` (pytest `meta` fixture) | 4 base keys ONLY |
| L3 | `runner/harness/lifecycle.py:_recipe_extra_env` | `EXTRA_ENV` only |
| L4 | `runner/harness/lifecycle.py:_recipe_meta_flag` | boolean flags by name (`CHAOS_BASE_DEPLOY`) |
| L5 | `runner/harness/deps.py:declared_deps` | `DEPS` only |
| L6 | `runner/harness/canonical.py:is_canonical_enrolled` | `WARM_CANONICAL` only |
- A key is "set" by a top-level ALL-CAPS assignment or `def`. Unknown ALL-CAPS top-level names
raise `MetaError` listing the unknown name and the nearest registered key (typo gate —
misspelling `READY_PROBE` can no longer silently disable the probe).
- Type mismatches raise `MetaError`; callables are accepted only for hook-typed keys.
- **Underscore-prefixed names (`_FOO`) are recipe-private and exempt** — that's where private
constants live (e.g. mumble's `_WELCOME_TEXT_MARKER`). Lowercase names (helpers/imports) are
ignored.
- Hook callables must have the registered signature (below); a legacy-signature hook raises a
`MetaError` naming the migration, never a silent `TypeError` mid-run.
> **Restructure status (rcust P1):** the six loaders above are HISTORY — they have been replaced by
> the single registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta` (the only
> `exec()` of `recipe_meta.py`). Unknown ALL-CAPS keys / type mismatches are now hard errors;
> underscore-prefixed names are recipe-private. The authoritative key reference is the generated
> table below; the per-loader subsections §4.1–§4.8 are retained for context until the P6 doc
> rewrite.
A unit test (`tests/unit/test_meta.py`) loads every `tests/*/recipe_meta.py` through the registry,
so a typo'd key fails at PR time, not at run time.
<!-- META-TABLE-START -->
@ -126,64 +128,27 @@ _This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scr
<!-- META-TABLE-END -->
### 4.1 HTTP / health / timing (base 4 — seen by L1 AND L2)
### 4.1 The uniform hook convention — `HookCtx`
| Key | Type / default | Meaning | Used by |
|---|---|---|---|
| `HEALTH_PATH` | str, `"/"` | Path probed for serving/health checks | deploy wait (`lifecycle.py`), generic `assert_serving` |
| `HEALTH_OK` | tuple, `(200, 301, 302)` | Acceptable HTTP status codes for health | same |
| `DEPLOY_TIMEOUT` | int s, `600` | Max wait for swarm convergence per deploy | `lifecycle.py`, generic ops |
| `HTTP_TIMEOUT` | int s, `300` | Max wait for HTTP health after converged | same |
Every recipe callable takes a single `ctx` argument (`harness/meta.py::HookCtx`, frozen):
Example: immich sets `DEPLOY_TIMEOUT = 1500`, `HTTP_TIMEOUT = 600` (ML containers are slow).
| Field | Meaning |
|---|---|
| `ctx.domain` | the app's per-run domain |
| `ctx.base_url` | `https://<domain>` |
| `ctx.meta` | the recipe's full `RecipeMeta` |
| `ctx.deps` | provisioned dep creds (`{dep_recipe: entry}`) or `None` |
| `ctx.op` | current lifecycle op (`install`/`upgrade`/`backup`/`restore`) or `None` |
### 4.2 Upgrade tier (loader L1)
Signatures: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`,
`SCREENSHOT(page, ctx)`, ops.py `pre_<op>(ctx)`. Dict-valued `EXTRA_ENV`/`UPGRADE_EXTRA_ENV`
(non-callable) are still fine — only the callable form takes ctx. The loader enforces the
parameter names at load time (a pre-restructure `(domain)`/`(domain, meta)` hook gets a pointed
`MetaError`, not a mid-run crash).
| Key | Type / default | Meaning |
|---|---|---|
| `UPGRADE_BASE_VERSION` | str (exact published tag), default `None` | **The "base pin"** — overrides the harness default base for the upgrade tier. Default base = `recipe_versions[-2]` (the previous published version); pin when that is not the PR's true predecessor (e.g. the PR is the first release on a new major, or the previous tag is known-broken). Must be an exact published tag — typos fail the base deploy. Consumed at `run_recipe_ci.py` (`prev = meta.get("UPGRADE_BASE_VERSION") or lifecycle.previous_version(recipe)`). Users: discourse, plausible. |
| `UPGRADE_EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `None` | Extra `.env` keys applied **after** the PR-head checkout, **before** the chaos redeploy (F2-14c) — for env vars that exist only at head (a new required setting introduced by the PR). Consumed in `generic.py:256`. User: mumble. |
### 4.3 Every-deploy shaping (loaders L3/L4 — NOT in the L1 allowlist)
| Key | Type / default | Meaning |
|---|---|---|
| `EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `{}` | Extra `.env` keys applied at **every** deploy (base install AND upgrade old-app). Callable form derives values from the per-run domain (e.g. cryptpad's `SANDBOX_DOMAIN`). Loaded by `lifecycle.py:_recipe_extra_env` (its own `exec()`). Users: cryptpad, discourse, ghost, matrix-synapse, mattermost-lts, mumble, plausible. |
| `CHAOS_BASE_DEPLOY` | bool, default `False` | Base deploy uses `--chaos` so it survives untracked files in the recipe checkout (required when `install_steps.sh` copies in a `compose.ccci.yml` overlay — §5.6; implicit coupling, see §8 R7). Loaded by `lifecycle.py:_recipe_meta_flag`. Users: discourse, ghost. |
### 4.4 Skips and intentional N/A (loader L1)
| Key | Type / default | Meaning |
|---|---|---|
| `SKIP_GENERIC` | list of op names or `"all"`/`"*"`, default `[]` | Suppress the generic floor for the listed ops (overlay becomes override instead of additive). Two env equivalents at run time: `CCCI_SKIP_GENERIC=1` (all ops), `CCCI_SKIP_GENERIC_<OP>=1` (one op). Currently set by **no enrolled recipe** (env form is the one used, ad hoc). |
| `EXPECTED_NA` | dict `{rung: reason}`, default `None` | Declares an N/A rung **intentional** (e.g. `{"backup": "stateless, nothing to back up"}`). Undeclared N/A is reported as an *unintentional coverage gap*. Both cap the achievable level — declaring does not un-cap, it only changes the report wording (`results.py`). User: custom-html-tiny. |
| `BACKUP_CAPABLE` | bool, default auto-detect | Overrides the backup-tier capability detection (scan of recipe compose files for `backupbot.backup` labels, `generic.py:34`). `False` forces N/A; `True` forces the tier on. Users: custom-html-bkp-bad/rst-bad (harness self-test recipes). |
### 4.5 Readiness & data-verification hooks (loader L1, callable values)
| Key | Type / default | Meaning |
|---|---|---|
| `READY_PROBE` | callable `(domain) -> [probe, ...]`, default `None` | Extra readiness probes run after install AND after upgrade, before that tier's assertions. Probe dicts: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}` (`stable`: must stay connectable across 3 checks — for UDP-adjacent voice ports etc.). Consumed at `lifecycle.py:516`. Users: lasuite-drive, mumble (TCP voice port). |
| `BACKUP_VERIFY` | callable `(domain) -> bool`, default `None` | Post-backup data-capture check, retried — guards the truncated-dump race (backup snapshot taken before the seeded marker row hit disk). Return `False` → retry the backup, then fail. Users: discourse, ghost. |
### 4.6 Dependencies / SSO (loaders L5 + L1)
| Key | Type / default | Meaning |
|---|---|---|
| `DEPS` | list of recipe names, default `[]` | Dep recipes deployed alongside (e.g. `["keycloak"]`). Dep domain is `<dep[:4]>-<6hex>`, hashed from (parent, pr, ref, dep) — collision-free per run. Creds land in `$CCCI_DEPS_FILE` (JSON); tests use the `deps_apps` fixture; teardown deps LAST. Deploy-count guard becomes `1 + len(DEPS)`. Loaded by `deps.py:declared_deps`. Users: lasuite-docs/-drive/-meet. |
| `OIDC_AT_INSTALL` | bool, default `False` | Provision deps **before** the single base deploy so `install_steps.sh` can wire OIDC env into that one deploy (reads `$CCCI_DEPS_FILE`). Default (legacy) is post-deploy provisioning + a `setup_custom_tests.sh` redeploy. Consumed at `run_recipe_ci.py:514`. Users: lasuite-drive, lasuite-meet. |
### 4.7 Warm-canonical enrollment (loader L6)
| Key | Type / default | Meaning |
|---|---|---|
| `WARM_CANONICAL` | bool, default `False` | Enrolls the recipe in the warm/canonical app system (`docs/warm.md`): green COLD runs on LATEST advance the canonical snapshot; the nightly sweep iterates enrolled recipes. Loaded by `canonical.py:is_canonical_enrolled`. User: custom-html. |
### 4.8 Cosmetic (BROKEN — see §8 R2)
| Key | Type / default | Meaning |
|---|---|---|
| `SCREENSHOT` | callable `(page, domain, meta) -> None` | Drives Playwright to a safe post-login view for the results-card screenshot (default: landing page). **Currently unreachable from the CI path**: `screenshot.py:41` reads it from the meta dict the orchestrator passes (`run_recipe_ci.py:1056`), but the L1 allowlist never loads `SCREENSHOT`, so the hook is always `None`. No recipe sets it (consistent with it never having worked). |
Worked hook examples: cryptpad (`EXTRA_ENV(ctx)` derives `SANDBOX_DOMAIN` from `ctx.domain`),
mumble (`READY_PROBE(ctx)` TCP voice-port probe, `UPGRADE_EXTRA_ENV(ctx)` adds a head-only compose
overlay), ghost/discourse (`BACKUP_VERIFY(ctx)` dump-capture check).
## 5. Writing custom tests & hooks
@ -196,104 +161,122 @@ test runs additively against the same state.
Conventions (see `tests/immich/test_backup.py` etc.):
- use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain)
- use the `meta` fixture for HEALTH_*/timeouts (note: only the 4 base keys — §8 R3)
- read op context from `$CCCI_OP_STATE_FILE` (JSON written by the orchestrator after the op:
versions, artifact paths)
- use the `meta` fixture — the recipe's FULL validated `RecipeMeta` (attribute access)
- use the `op_state` fixture for op context (versions, `snapshot_id`, artifact paths — the
orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run)
- execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)`
### 5.2 Pre-op seed hooks — `ops.py`
`def pre_<op>(domain, meta)` callables, imported and called by the orchestrator **before**
performing the op. This is where data gets seeded so the post-op overlay can assert on it:
`def pre_<op>(ctx)` callables, imported and called by the orchestrator **before** performing the
op. This is where data gets seeded so the post-op overlay can assert on it:
```python
# tests/immich/ops.py (pattern)
def pre_upgrade(domain, meta): _psql(domain, "INSERT ... 'upgrade-survives'")
def pre_backup(domain, meta): _psql(domain, "INSERT ... 'original'")
def pre_restore(domain, meta): _psql(domain, "DROP TABLE ci_marker") # damage, restore must undo
def pre_upgrade(ctx): _psql(ctx.domain, "INSERT ... 'upgrade-survives'")
def pre_backup(ctx): _psql(ctx.domain, "INSERT ... 'original'")
def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, restore must undo
```
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
### 5.3 Custom tier — `functional/`, `playwright/`, top-level `test_*.py`
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
All non-lifecycle `test_*.py` (discovery: `discovery.py:custom_tests`, recursive over the
top-level dir + `functional/` + `playwright/`; files named `test_<op>.py` excluded). Run in the
CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run —
cc-ci's and (if HC2-approved) repo-local's, additively.
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
HC2-approved) repo-local's, additively.
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable).
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
Tests gate on deps via `CCCI_DEPS_READY` (skip-with-reason when `0`; the skip is counted and
fails the run if deps were declared but unprovisionable — `run_recipe_ci.py:816`).
Tests needing deps use the `deps` fixture (entries expose `.domain` plus the full creds dict) and
carry `@pytest.mark.requires_deps` — when dep provisioning failed they skip with reason
`deps-not-ready` and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit
must not mask an unrun SSO test). Fixtures replace direct `os.environ` reads — after the
restructure no recipe test parses env by hand.
### 5.4 Pre-deploy shell hook — `install_steps.sh`
Runs after `abra app new` + `EXTRA_ENV` application + secret generation, **before** the base
deploy. For setup that must precede the first deploy: writing extra config files into the recipe
checkout, copying in a `compose.ccci.yml` overlay (§5.6), editing `.env` beyond simple key=val.
The ONLY shell hook. Runs after `abra app new` + `EXTRA_ENV` application + secret generation,
**before** the single base deploy. For setup that must precede the first deploy: writing extra
config files into the recipe checkout, editing `.env` beyond simple key=val, and — for recipes
with `DEPS` — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the
deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy
redeploy hook).
Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and —
when `OIDC_AT_INSTALL` deps exist — `CCCI_DEPS_FILE`. Must locate the recipe checkout
ABRA_DIR-aware: `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run
`ABRA_DIR` since the concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).
when `DEPS` is declared — `CCCI_DEPS_FILE` (jq-readable JSON of dep creds/URLs; see
lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware:
`RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run `ABRA_DIR` since the
concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).
Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic
install — a correct reported outcome, not a harness error.
### 5.5 Deps credential wiring — `setup_custom_tests.sh`
### 5.5 CI-only compose overlay — `compose.ccci.yml`
For legacy (post-deploy) deps provisioning: runs after deps are up, reads `$CCCI_DEPS_FILE`
(jq-readable JSON of dep creds/URLs), wires OIDC config via `abra app config set` + secrets, and
redeploys. With `OIDC_AT_INSTALL = True` this hook is unnecessary (wiring happens in
`install_steps.sh` before the only deploy) — preferred for new enrollments (one deploy, no
deploy-count exception).
**First-class:** if `tests/<recipe>/compose.ccci.yml` exists, the harness itself copies it into
the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses `--chaos` for
that deploy (the untracked file would otherwise trip abra's clean-tree gate). No
`install_steps.sh` copy boilerplate, no flag to remember (the old `CHAOS_BASE_DEPLOY` ⇄ overlay
coupling is gone). The overlay is cc-ci-owned only.
### 5.6 CI-only compose overlay — `compose.ccci.yml`
Policy unchanged: overlays are a minimal, justified fallback (ghost's is a 15m `start_period`
grace — a literal, because abra validates `start_period` before env substitution). Reference the
overlay from `EXTRA_ENV`'s `COMPOSE_FILE` as usual. Users: ghost, discourse.
Not auto-discovered: `install_steps.sh` copies it into the recipe checkout, and the recipe must
set `CHAOS_BASE_DEPLOY = True` so the base deploy (`--chaos`) tolerates the untracked file.
Policy: minimal, justified fallback only (ghost's is a 15m `start_period` grace — a literal,
because abra validates `start_period` before env substitution). The overlay is cc-ci-owned even
though it rides in the recipe checkout.
### 5.6 Environment & fixture contract (what custom code can read)
### 5.7 Environment contract summary (what custom code can read)
Pytest fixtures (`tests/conftest.py` — the single fixture file):
| Fixture | Yields |
|---|---|
| `recipe` | the recipe name (`$RECIPE`) |
| `meta` | the FULL validated `RecipeMeta` (single loader) |
| `live_app` | the shared deployment's domain (asserts it exists) |
| `op_state` | the orchestrator's op-context dict (skips cleanly outside a run) |
| `deps` | `{dep_recipe: entry}` — entries expose `.domain` + full SSO creds |
Environment (hooks/shell, and approved repo-local code):
| Var | Set for | Meaning |
|---|---|---|
| `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain |
| `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` |
| `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path |
| `CCCI_OP_STATE_FILE` | overlay tests | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | deps hooks + tests | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier | gate SSO tests, skip-with-reason |
| `CCCI_OP_STATE_FILE` | overlay tests (via `op_state`) | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | `install_steps.sh` + harness | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier (via `requires_deps`) | gate SSO tests, skip-with-reason |
## 6. Run-model context (what the settings plug into)
One deploy chain per run (full detail: `docs/testing.md` §2):
```
[DEPS? provision deps FIRST → $CCCI_DEPS_FILE]
deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh;
CHAOS_BASE_DEPLOY?; OIDC_AT_INSTALL deps first?)
compose.ccci.yml auto-copied + auto-chaos)
→ INSTALL tier (READY_PROBE; generic + overlay asserts)
→ pre_upgrade → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
→ pre_upgrade(ctx) → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
→ UPGRADE tier (READY_PROBE; version-label == head_ref)
→ pre_backup → backup (BACKUP_CAPABLE; BACKUP_VERIFY)
→ pre_backup(ctx) → backup (BACKUP_CAPABLE; BACKUP_VERIFY)
→ BACKUP tier
→ pre_restore → restore
→ pre_restore(ctx) → restore
→ RESTORE tier
→ CUSTOM tier (functional/ + playwright/; deps via CCCI_DEPS_*)
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
→ SCREENSHOT (best-effort, never affects the verdict)
→ teardown (deps LAST)
```
Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't
count); the per-run counter file is keyed by run since the concurrency restructure.
## 7. Local iteration
## 7. Local iteration, the manifest, and the dev-only escape hatch
```
RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
@ -303,81 +286,75 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
(`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.)
## 8. Known limitations & restructuring candidates
**Customization manifest.** Every run prints, right after meta load + discovery, one block:
The review section. Ordered by how much they'd shape a restructure.
```
===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci)
env overrides: (none)
```
**R1 — Six divergent meta loaders (the core drift hazard).** §4's L1–L6: every loader re-`exec()`s
`recipe_meta.py` and cherry-picks its own keys. Adding a key means knowing *which* loader to touch
(or that you must extend the L1 allowlist — `SCREENSHOT` proves people don't, R2). Two conventions
coexist: L1's explicit allowlist vs L3–L6's ad-hoc `ns.get(...)` which silently bypasses it.
*Candidate:* one `harness.meta.load(recipe) -> RecipeMeta` with a declarative key registry
(name, type, default, validator, consumer) as the single source of truth; L1–L6 become lookups
into the one loaded object; the registry also generates §4 of this doc (kills doc drift, R5).
The same dict is embedded in `results.json` under `"customization"`. It is pure presentation —
built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes,
and it honors the HC2 gate) — and never influences a verdict.
**R2 — `SCREENSHOT` is a dead knob.** Fully implemented consumer (`screenshot.py`), documented
hook contract, never reachable: the orchestrator's allowlist omits it, so the dict passed at
`run_recipe_ci.py:1056` can never contain it. Direct evidence of R1. *Candidate:* fix trivially by
adding to the allowlist — or delete the hook path if post-login screenshots aren't wanted; decide
during the restructure.
**Dev-only generic skip.** `CCCI_SKIP_GENERIC=1` (all ops) / `CCCI_SKIP_GENERIC_<OP>=1` (one op)
suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no
declarative equivalent (the old `SKIP_GENERIC` meta key is deleted). If the env form is active in
a CI (drone) run, the run prints a loud `!!` warning and the manifest records it.
**R3 — The pytest `meta` fixture sees 4 keys.** `tests/conftest.py:_recipe_meta` loads only
HEALTH_*/timeouts. An overlay test wanting e.g. `EXPECTED_NA` or a recipe constant must re-exec
the file itself. Probably intended minimalism, but it's a third key-set to keep in sync.
*Folds into R1.*
## 8. Restructure outcomes (the review spec's R1–R9)
**R4 — Settings split across three config languages** (§1): recipe_meta keys, file-presence
(`install_steps.sh` existing changes deploy behavior), and run-time env (`CCCI_SKIP_GENERIC*`).
A reviewer asking "what does this recipe customize?" must check all three. *Candidate:* keep the
three surfaces (they serve different actors) but make the run header log a single resolved
"customization manifest" per run: every non-default key + every discovered hook file + every
CCCI_* override, in one block.
How each defect identified in the review spec (commit `76a4b6b` §8) was resolved:
**R5 — Reference-doc drift already happened.** `docs/testing.md` documents 6 meta keys,
`docs/enroll-recipe.md` shows others by example; neither is complete (18 keys exist). This doc is
now complete but handwritten — it will drift too. *Candidate:* generate the key table from the R1
registry (test asserts doc ⊆ registry).
- **R1 — six divergent meta loaders → RESOLVED.** One registry-backed loader
(`harness/meta.py::load`), the only `exec()` of `recipe_meta.py`. The orchestrator loads once
and passes the `RecipeMeta` down; conftest/lifecycle/deps/canonical all read the one object.
- **R2 — dead `SCREENSHOT` knob → RESOLVED (kept + fixed).** The registry replaced the allowlist
that orphaned it; the orchestrator path now delivers the hook to `screenshot.py`
(proven end-to-end by `tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path`).
- **R3 — 4-key pytest `meta` fixture → RESOLVED.** The fixture returns the full validated
`RecipeMeta`.
- **R4 — three config languages → MITIGATED by the manifest** (§7): the surfaces stay (they serve
different actors), but every run resolves them into one visible block + results key.
- **R5 — reference-doc drift → RESOLVED.** §4's key table is generated from the registry
(`scripts/gen-meta-docs.py`); a unit test fails CI on drift; `testing.md`/`enroll-recipe.md`
point here instead of keeping partial lists.
- **R6 — silent typos → RESOLVED.** Unknown ALL-CAPS keys and type mismatches are hard
`MetaError`s; private constants are underscore-prefixed (exempt).
- **R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` coupling → RESOLVED.** The overlay is
first-class: harness-copied, auto-chaos. The flag is deleted.
- **R8 — zero-user `SKIP_GENERIC` meta key → RESOLVED (deleted).** Env form remains, documented
dev-only, loudly flagged in CI runs (§7).
- **R9 — `recipe_meta.py` is code, not config → REJECTED by decision.** No data/hooks file split:
registry validation gets the value (typed, validated keys) at lower cost; one file per recipe
remains the single config place. The expressiveness need is real (cryptpad derives env from the
per-run domain).
**R6 — No schema validation / silent typos.** Unknown top-level names in `recipe_meta.py` are
ignored, which is load-bearing (recipes keep private constants there: mumble's
`WELCOME_TEXT_MARKER`, `MAX_USERS`). Consequence: misspelling `READY_PROBE` as `READINESS_PROBE`
silently disables the probe — the run goes green with less coverage, the worst failure mode for a
CI harness. *Candidate:* with the R1 registry, warn (not fail) on ALL-CAPS top-level names that
are not registered and not referenced by the recipe's own tests; or namespace private constants
(`_WELCOME_TEXT_MARKER`).
**R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` implicit coupling.** The overlay only works if
the recipe *also* sets the flag; forgetting it fails the base deploy with an abra
untracked-files error far from the cause. *Candidate:* if `install_steps.sh` exists alongside a
`compose.ccci.yml`, the harness could auto-enable chaos for the base deploy (or at least assert
the flag and fail with a pointed message).
**R8 — `SKIP_GENERIC` (meta form) has zero users.** Only the env-var form is used, ad hoc. Either
the meta key earns its place (first real user) or it's surface to delete in the restructure.
**R9 — `recipe_meta.py` is code, not config.** Five keys take callables (`EXTRA_ENV`,
`UPGRADE_EXTRA_ENV`, `READY_PROBE`, `BACKUP_VERIFY`, `SCREENSHOT`), so the file must stay an
`exec()`d Python module — it can't be validated as data, serialized into results, or diffed
declaratively. This is a real expressiveness need (cryptpad derives `SANDBOX_DOMAIN` from the
per-run domain), not an accident. *Candidate if restructuring:* split data keys (TOML-able,
schema-validated) from a `hooks.py` (callables only) — but weigh against the cost of two files
per recipe; the R1 registry gets most of the value without the split.
Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy
post-deploy `setup_custom_tests.sh` machinery and its extra redeploy are deleted); the custom-test
placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface
(§5.6 — `deps` replaces `deps_apps`+`deps_creds`; dead `deployed`/`deployed_app`/`app_domain`
fixtures deleted).
## 9. File / symbol index
| Concern | Where |
|---|---|
| Orchestrator meta loader (L1, allowlist) | `runner/run_recipe_ci.py:250` `_load_meta` |
| Pytest meta fixture (L2) | `tests/conftest.py` `_recipe_meta` |
| `EXTRA_ENV` loader (L3) | `runner/harness/lifecycle.py:114` `_recipe_extra_env` |
| Boolean-flag loader (L4) | `runner/harness/lifecycle.py:132` `_recipe_meta_flag` |
| `DEPS` loader (L5) | `runner/harness/deps.py:37` `declared_deps` |
| `WARM_CANONICAL` loader (L6) | `runner/harness/canonical.py:36` `is_canonical_enrolled` |
| Overlay/custom/hook discovery + HC2 gate | `runner/harness/discovery.py` |
| THE meta loader + key registry + `HookCtx` + `MetaError` | `runner/harness/meta.py` (`load`, `KEYS`, `check_hook_signature`) |
| Generated key table | `scripts/gen-meta-docs.py` → §4 above (sync pinned by `tests/unit/test_meta.py`) |
| Customization manifest | `runner/harness/manifest.py` (`build`, `render`), printed by `runner/run_recipe_ci.py` |
| Overlay/custom/hook discovery + HC2 gate + placement rule | `runner/harness/discovery.py` |
| HC2 allowlist | `tests/repo-local-approved.txt` |
| Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` |
| `READY_PROBE` / `CHAOS_BASE_DEPLOY` consumption | `runner/harness/lifecycle.py:516` / `:283` |
| `compose.ccci.yml` auto-copy + auto-chaos | `runner/harness/lifecycle.py` (`provide_ccci_overlay`, `deploy_app`) |
| `READY_PROBE` consumption | `runner/harness/lifecycle.py` (`wait_ready_probes`) |
| `EXPECTED_NA` reporting | `runner/harness/results.py` |
| Dead `SCREENSHOT` consumer | `runner/harness/screenshot.py:36`, called `run_recipe_ci.py:1056` |
| Skip-generic logic (meta + env) | `runner/run_recipe_ci.py:285` |
| Worked examples | `tests/ghost/` (overlay+chaos), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV), `tests/lasuite-drive/` (DEPS+OIDC_AT_INSTALL), `tests/immich/` (ops.py seed pattern) |
| `SCREENSHOT` consumer | `runner/harness/screenshot.py` |
| Fixtures (`recipe`/`meta`/`live_app`/`op_state`/`deps`) + F2-11 skip-report | `tests/conftest.py` |
| Skip-generic env logic (dev-only) | `runner/run_recipe_ci.py` (`_skip_generic`) |
| Unit tests pinning all of the above | `tests/unit/test_meta.py`, `test_manifest.py`, `test_discovery*.py` |
| Worked examples | `tests/ghost/` (overlay+compose.ccci.yml), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV, private `_` constants), `tests/lasuite-drive/` (DEPS + install-time OIDC wiring), `tests/immich/` (ops.py seed pattern) |