recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues 1 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

docs: P6 — rewrite customization docs to the restructured end state (rcust)
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse Source 
recipe-customization.md: review spec -> reference. Single registry-backed loader + validation
rules + HookCtx convention (§4); generated key table kept byte-identical (sync test); §5 end-state
shape (op_state/deps fixtures, ctx ops.py, placement rule, first-class compose.ccci.yml, no
setup_custom_tests.sh); §7 manifest block + dev-only CCCI_SKIP_GENERIC*; §8 rewritten as
restructure outcomes (R1/R2/R3/R5/R6/R7/R8 resolved + how, R4 mitigated by manifest, R9
rejected-by-decision); §9 index updated to the new symbols.

testing.md: install-time deps isolation replaces the setup_custom_tests step in the invariant
(generic still never depends on custom — failure isolation via requires_deps/F2-11); ops.py
example to pre_<op>(ctx); placement rule; generic opt-out now documented LOCAL-DEV-ONLY env with
CI !! warning (declarative SKIP_GENERIC gone); partial key list points at the generated table.

enroll-recipe.md: tree + worked examples updated (lasuite-docs install-time OIDC wiring +
install_steps.sh; mumble post-F2-14c shape — UPGRADE_EXTRA_ENV native overlay, private _
constants, no CHAOS_BASE_DEPLOY); deps fixture (entry.domain) replaces deps_apps; ctx hook
signatures; compose.ccci.yml first-class bullet; key list points at the generated table.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-10 19:07:41 +00:00
parent 68954be53e
commit da558ca946
3 changed files with 268 additions and 272 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
docs/enroll-recipe.md
View File

@ -14,8 +14,9 @@ those are discovered and run against the live app (D4 — see below).
```
tests/<recipe>/
├── recipe_meta.py # optional per-recipe harness config (see below)
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup)
├── ops.py # optional pre-op seed hooks (pre_install/pre_upgrade/pre_backup/pre_restore)
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup + deps env wiring)
├── compose.ccci.yml # optional CI-only compose overlay (harness-copied, auto-chaos base deploy)
├── ops.py # optional pre_<op>(ctx) seed hooks (install/upgrade/backup/restore)
├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic)
├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic)
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
@ -39,11 +40,14 @@ To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` **overlay*
**ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently
dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture;
they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to
SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(domain,
meta)` callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op. Copy an
SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(ctx)`
callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op (`ctx` is the
uniform `HookCtx` every hook receives — `docs/recipe-customization.md` §4.1). Copy an
existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/
matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` /
`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py`:
`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py` (the COMPLETE key
reference is the generated table in `docs/recipe-customization.md` §4; unknown ALL-CAPS keys are
hard errors, recipe-private constants are underscore-prefixed `_FOO`):
```python
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
@ -51,9 +55,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"] # per-recipe opt-out from the generic floor for the listed ops
# ("all"/"*" = every op); rarely needed — generic is the floor
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy
```
Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`,
@ -76,9 +78,10 @@ Beyond the lifecycle overlays, each recipe carries (plan §4.1):
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
(recursive, via `runner/harness/discovery.custom_tests`) and runs each as its own pytest against
the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are
**excluded** from the custom tier — they live at the top level and run as lifecycle overlays.
ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
`live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
from the custom tier even inside those subdirs (safety net against double-running).
### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3)
@ -89,23 +92,28 @@ them in `recipe_meta.py`:
DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests/<dep>/ must exist + work)
```
The orchestrator (plan §4.2):
1. Reads `DEPS` BEFORE deploying the recipe under test.
2. Deploys each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is
hashed from `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do
not collide on a single node).
3. Waits each dep healthy using its own `recipe_meta.py` (HEALTH_PATH/HEALTH_OK/timeouts).
4. Persists `[{"recipe": "<dep>", "domain": "<dep-domain>"}, ...]` to `$CCCI_DEPS_FILE`.
5. Deploys + tests the recipe under test as usual.
6. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked
The orchestrator (plan §4.2; install-time provisioning is the ONLY mode):
1. Reads `DEPS` and provisions every dep **BEFORE the single deploy** of the recipe under test —
each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is hashed from
`parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do not collide on
a single node), waited healthy using the dep's own `recipe_meta.py`.
2. Persists the full per-dep identity + SSO creds dict to `$CCCI_DEPS_FILE` (jq-readable JSON,
`{"<dep>": {"domain": ..., "realm": ..., "client_secret": ..., ...}}`).
3. Deploys the recipe under test — its `install_steps.sh` reads `$CCCI_DEPS_FILE` and wires
OIDC env into that ONE deploy (no post-deploy redeploy). A dep-provisioning failure does NOT
block the run: the recipe deploys alone, generic tiers run, and `requires_deps` tests skip
with a counted reason (F2-11).
4. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked
deps fail the run loudly per §9 teardown sacred / F2-5 fix).
Tests access dep domains via the **`deps_apps` pytest fixture** (`tests/conftest.py`):
Tests access deps via the **`deps` pytest fixture** (`tests/conftest.py`) — entries expose
`.domain` plus the full creds dict (attribute or dict-style):
```python
def test_my_recipe_uses_keycloak(live_app, deps_apps):
assert "keycloak" in deps_apps, f"keycloak dep not deployed; {deps_apps}"
kc_domain = deps_apps["keycloak"]
@pytest.mark.requires_deps
def test_my_recipe_uses_keycloak(live_app, deps):
assert "keycloak" in deps, f"keycloak dep not deployed; {deps}"
kc_domain = deps["keycloak"].domain
…
```
@ -120,7 +128,7 @@ For OIDC-dependent recipes, the shared `runner/harness/sso.py` provides:
from harness import sso
creds = sso.setup_keycloak_realm(
kc_domain, # = deps_apps["keycloak"]
kc_domain, # = deps["keycloak"].domain
realm="my-realm",
client_id="my-client",
redirect_uris=[f"https://{live_app}/*"],
@ -144,10 +152,10 @@ ARE provider-pluggable.
Not every recipe is a single HTTP app. `recipe_meta.py` + a few harness mechanisms cover the harder
shapes (proven on mumble, mailu, and the SSO-dependent suite):
- **`EXTRA_ENV`** — a dict **or** a `callable(domain) -> dict`. The callable form derives values from
the per-run domain (e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for cryptpad). Applied
at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change.
- **`READY_PROBE(domain) -> [...]`** — readiness signals beyond replica-convergence + the app's
- **`EXTRA_ENV`** — a dict **or** a `callable(ctx) -> dict`. The callable form derives values from
the per-run domain (`ctx.domain` — e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for
cryptpad). Applied at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change.
- **`READY_PROBE(ctx) -> [...]`** — readiness signals beyond replica-convergence + the app's
`HEALTH_PATH`. Two probe shapes:
- HTTP: `{"host": "...", "path": "/...", "ok": (200,)}` (e.g. lasuite-drive collabora WOPI discovery).
- **TCP**: `{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}` — polls a socket connect N
@ -155,16 +163,16 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
service (mumble: the mumble-web sidecar serves HTTP 200 while the voice server on 64738 is still
rebinding after an upgrade redeploy — the TCP probe gates the backup tier until the voice server is
actually up). Runs after install AND after the upgrade chaos redeploy.
- **`CHAOS_BASE_DEPLOY = True`** — make the pinned base deploy use `--chaos` (skips abra's clean-tree +
lint gates, still deploys the explicitly-checked-out pinned version, NOT latest). Needed when an
`install_steps.sh` adds an UNTRACKED file to the recipe checkout (e.g. mumble copies a
`compose.host-ports.yml` into versions that predate it) — abra's pinned-deploy clean-tree check would
otherwise FATA. `abra.recipe_checkout` force-checks-out (`-f`) so the upgrade tier's re-checkout to
PR-head overwrites such overlays cleanly.
- **`compose.ccci.yml`** (first-class at `tests/<recipe>/compose.ccci.yml`) — a CI-only compose
overlay the harness itself copies into the recipe checkout before the base deploy, automatically
using `--chaos` for that deploy (the untracked file would otherwise trip abra's pinned-deploy
clean-tree check). Reference it from `EXTRA_ENV`'s `COMPOSE_FILE`. Minimal, justified fallback
only (e.g. ghost's 15m `start_period` grace). `abra.recipe_checkout` force-checks-out (`-f`) so
the upgrade tier's re-checkout to PR-head overwrites such overlays cleanly.
- **`install_steps.sh`** (auto-discovered at `tests/<recipe>/install_steps.sh`) — runs after
`abra app new` + EXTRA_ENV + secret-generate, BEFORE the single deploy, with `CCCI_APP_DOMAIN` /
`CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when DEPS are provisioned at install). Use it to
drop a cc-ci-owned compose overlay into the checkout, wire dep-derived env/secrets, etc.
`CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when the recipe declares DEPS — deps are
always provisioned before the deploy). Use it to wire dep-derived env/secrets, seed config, etc.
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
@ -227,9 +235,10 @@ RECIPE=<recipe> PR=<n> REF=<sha-or-branch> SRC=recipe-maintainers/<recipe> \
```
tests/lasuite-docs/
├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(domain) for cold-pull,
├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(ctx) for cold-pull,
│ # DEPS=["keycloak"] ← Phase 2 dep declaration
├── ops.py # pre_<op> seed hooks (volume marker for backup/restore data-integrity)
├── install_steps.sh # wires OIDC env from $CCCI_DEPS_FILE into the single deploy
├── ops.py # pre_<op>(ctx) seed hooks (volume marker for backup/restore data-integrity)
├── test_install.py # lifecycle install overlay (Playwright frontend SPA load)
├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy)
├── test_backup.py # lifecycle backup overlay (marker captured)
@ -239,12 +248,14 @@ tests/lasuite-docs/
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
# harness.sso primitives + deps_apps["keycloak"])
# harness.sso primitives + the `deps` fixture)
```
`!testme` on a lasuite-docs PR drives the orchestrator to:
1. Deploy the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`) and wait healthy.
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`).
1. Provision the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`), wait healthy, write
creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
env into that one deploy.
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
deployment (custom tier).
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
@ -254,12 +265,13 @@ tests/lasuite-docs/
### Other shapes (concrete references)
- **TCP / voice recipe — `tests/mumble/`**: `recipe_meta.py` (EXTRA_ENV sets
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml:compose.host-ports.yml`, `WELCOME_TEXT`/`USERS`
markers, `CHAOS_BASE_DEPLOY=True`, `READY_PROBE` TCP 64738), `install_steps.sh` (provides the
host-ports overlay to older versions), `functional/_mumble_proto.py` + the protocol/config-round-trip
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
`_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
(`EXTRA_ENV(domain)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
(`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
DEFERRED.md). See §2.4.

375
docs/recipe-customization.md
View File

@ -1,8 +1,9 @@
# Recipe customization — review spec
# Recipe customization — reference
Status: REVIEW SPEC — describes the customization surface as it exists today (main), written so
the structure can be reviewed and potentially restructured. §8 lists known limitations and
restructuring candidates; everything before it is purely descriptive.
Status: REFERENCE — describes the customization system as restructured on branch
`restructure/recipe-custom` (the "rcust" restructure). The pre-restructure system and its defects
are documented in this file's history (commit `76a4b6b`, the review spec whose §8 R1–R9 drove the
restructure); §8 below records how each was resolved.
Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md`
(step-by-step enrollment). This doc is the **complete reference** for the two questions those docs
@ -15,17 +16,18 @@ answer only partially:
## 1. The three customization surfaces
A recipe customizes its CI through **three distinct mechanisms** (worth noticing for the
restructure review — they are three different config languages):
A recipe customizes its CI through **three distinct mechanisms**:
| Surface | Form | Examples |
|---|---|---|
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, shell hooks | `def READY_PROBE(domain): ...`, `pre_upgrade()`, `install_steps.sh` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
There is additionally a fourth, operator-facing surface: **environment variables**
(`CCCI_SKIP_GENERIC*`) that override declarative settings at run time (§4.4).
There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
(`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
from all four surfaces is printed at run start as the **customization manifest** and embedded in
`results.json` under `"customization"` (§7) — one block answers "what does this recipe customize?".
## 2. Zero-config baseline
@ -55,53 +57,53 @@ Two locations, with precedence and a security gate between them:
```
tests/<recipe>/ # cc-ci side (repo-local mirrors the same shape)
├── recipe_meta.py # ALL declarative settings + meta callables (§4)
├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py # pre_<op>(domain, meta) seed hooks (§5.2)
├── test_*.py # custom-tier tests (top-level, cross-cutting)(§5.3)
├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
├── playwright/test_*.py # custom tier: UI flows (§5.3)
├── install_steps.sh # pre-deploy shell hook (§5.4)
├── setup_custom_tests.sh # deps/OIDC credential wiring hook (§5.5)
├── compose.ccci.yml # CI-only compose overlay (via install_steps) (§5.6)
├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
└── PARITY.md # enrollment contract doc (human-read only)
```
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
generic floor still runs additively alongside.
- custom tier `test_*.py`: **ALL** run, from both locations (no collision concept).
- custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
concept).
- `install_steps.sh`: repo-local > cc-ci, or none.
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
- `recipe_meta.py`: cc-ci only — repo-local recipes cannot set CI settings (by design; the
settings surface stays maintainer-controlled).
- `recipe_meta.py` and `compose.ccci.yml`: cc-ci only — repo-local recipes cannot set CI settings
or compose overlays (by design; those surfaces stay maintainer-controlled).
## 4. `recipe_meta.py` — complete settings reference
The single settings file. Plain Python, `exec()`d by the harness (trusted, in-repo). A key is "set"
by a top-level assignment or `def`. Unknown names are ignored silently (a recipe may keep private
constants here, e.g. mumble's `WELCOME_TEXT_MARKER` — but see §8 R6: typos in real key names are
also silently ignored).
The single settings file. Plain Python, `exec()`d by the harness in exactly ONE place: the
registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta`. Every consumer — the
orchestrator (which loads once and passes the object down), the pytest `meta` fixture, lifecycle,
deps, canonical, screenshot — reads from that one loaded object.
**Loader column legend** — this is the structural finding for the review (§8 R1). There is no
single loader; six independent code paths each `exec()` the file and pick out their own keys:
**Validation (hard errors at load, before any deploy):**
| # | Loader | Keys it sees |
|---|---|---|
| L1 | `runner/run_recipe_ci.py:_load_meta` (orchestrator) | 4 base + explicit 8-key allowlist |
| L2 | `tests/conftest.py:_recipe_meta` (pytest `meta` fixture) | 4 base keys ONLY |
| L3 | `runner/harness/lifecycle.py:_recipe_extra_env` | `EXTRA_ENV` only |
| L4 | `runner/harness/lifecycle.py:_recipe_meta_flag` | boolean flags by name (`CHAOS_BASE_DEPLOY`) |
| L5 | `runner/harness/deps.py:declared_deps` | `DEPS` only |
| L6 | `runner/harness/canonical.py:is_canonical_enrolled` | `WARM_CANONICAL` only |
- A key is "set" by a top-level ALL-CAPS assignment or `def`. Unknown ALL-CAPS top-level names
raise `MetaError` listing the unknown name and the nearest registered key (typo gate —
misspelling `READY_PROBE` can no longer silently disable the probe).
- Type mismatches raise `MetaError`; callables are accepted only for hook-typed keys.
- **Underscore-prefixed names (`_FOO`) are recipe-private and exempt** — that's where private
constants live (e.g. mumble's `_WELCOME_TEXT_MARKER`). Lowercase names (helpers/imports) are
ignored.
- Hook callables must have the registered signature (below); a legacy-signature hook raises a
`MetaError` naming the migration, never a silent `TypeError` mid-run.
> **Restructure status (rcust P1):** the six loaders above are HISTORY — they have been replaced by
> the single registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta` (the only
> `exec()` of `recipe_meta.py`). Unknown ALL-CAPS keys / type mismatches are now hard errors;
> underscore-prefixed names are recipe-private. The authoritative key reference is the generated
> table below; the per-loader subsections §4.1–§4.8 are retained for context until the P6 doc
> rewrite.
A unit test (`tests/unit/test_meta.py`) loads every `tests/*/recipe_meta.py` through the registry,
so a typo'd key fails at PR time, not at run time.
<!-- META-TABLE-START -->
@ -126,64 +128,27 @@ _This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scr
<!-- META-TABLE-END -->
### 4.1 HTTP / health / timing (base 4 — seen by L1 AND L2)
### 4.1 The uniform hook convention — `HookCtx`
| Key | Type / default | Meaning | Used by |
|---|---|---|---|
| `HEALTH_PATH` | str, `"/"` | Path probed for serving/health checks | deploy wait (`lifecycle.py`), generic `assert_serving` |
| `HEALTH_OK` | tuple, `(200, 301, 302)` | Acceptable HTTP status codes for health | same |
| `DEPLOY_TIMEOUT` | int s, `600` | Max wait for swarm convergence per deploy | `lifecycle.py`, generic ops |
| `HTTP_TIMEOUT` | int s, `300` | Max wait for HTTP health after converged | same |
Every recipe callable takes a single `ctx` argument (`harness/meta.py::HookCtx`, frozen):
Example: immich sets `DEPLOY_TIMEOUT = 1500`, `HTTP_TIMEOUT = 600` (ML containers are slow).
| Field | Meaning |
|---|---|
| `ctx.domain` | the app's per-run domain |
| `ctx.base_url` | `https://<domain>` |
| `ctx.meta` | the recipe's full `RecipeMeta` |
| `ctx.deps` | provisioned dep creds (`{dep_recipe: entry}`) or `None` |
| `ctx.op` | current lifecycle op (`install`/`upgrade`/`backup`/`restore`) or `None` |
### 4.2 Upgrade tier (loader L1)
Signatures: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`,
`SCREENSHOT(page, ctx)`, ops.py `pre_<op>(ctx)`. Dict-valued `EXTRA_ENV`/`UPGRADE_EXTRA_ENV`
(non-callable) are still fine — only the callable form takes ctx. The loader enforces the
parameter names at load time (a pre-restructure `(domain)`/`(domain, meta)` hook gets a pointed
`MetaError`, not a mid-run crash).
| Key | Type / default | Meaning |
|---|---|---|
| `UPGRADE_BASE_VERSION` | str (exact published tag), default `None` | **The "base pin"** — overrides the harness default base for the upgrade tier. Default base = `recipe_versions[-2]` (the previous published version); pin when that is not the PR's true predecessor (e.g. the PR is the first release on a new major, or the previous tag is known-broken). Must be an exact published tag — typos fail the base deploy. Consumed at `run_recipe_ci.py` (`prev = meta.get("UPGRADE_BASE_VERSION") or lifecycle.previous_version(recipe)`). Users: discourse, plausible. |
| `UPGRADE_EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `None` | Extra `.env` keys applied **after** the PR-head checkout, **before** the chaos redeploy (F2-14c) — for env vars that exist only at head (a new required setting introduced by the PR). Consumed in `generic.py:256`. User: mumble. |
### 4.3 Every-deploy shaping (loaders L3/L4 — NOT in the L1 allowlist)
| Key | Type / default | Meaning |
|---|---|---|
| `EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `{}` | Extra `.env` keys applied at **every** deploy (base install AND upgrade old-app). Callable form derives values from the per-run domain (e.g. cryptpad's `SANDBOX_DOMAIN`). Loaded by `lifecycle.py:_recipe_extra_env` (its own `exec()`). Users: cryptpad, discourse, ghost, matrix-synapse, mattermost-lts, mumble, plausible. |
| `CHAOS_BASE_DEPLOY` | bool, default `False` | Base deploy uses `--chaos` so it survives untracked files in the recipe checkout (required when `install_steps.sh` copies in a `compose.ccci.yml` overlay — §5.6; implicit coupling, see §8 R7). Loaded by `lifecycle.py:_recipe_meta_flag`. Users: discourse, ghost. |
### 4.4 Skips and intentional N/A (loader L1)
| Key | Type / default | Meaning |
|---|---|---|
| `SKIP_GENERIC` | list of op names or `"all"`/`"*"`, default `[]` | Suppress the generic floor for the listed ops (overlay becomes override instead of additive). Two env equivalents at run time: `CCCI_SKIP_GENERIC=1` (all ops), `CCCI_SKIP_GENERIC_<OP>=1` (one op). Currently set by **no enrolled recipe** (env form is the one used, ad hoc). |
| `EXPECTED_NA` | dict `{rung: reason}`, default `None` | Declares an N/A rung **intentional** (e.g. `{"backup": "stateless, nothing to back up"}`). Undeclared N/A is reported as an *unintentional coverage gap*. Both cap the achievable level — declaring does not un-cap, it only changes the report wording (`results.py`). User: custom-html-tiny. |
| `BACKUP_CAPABLE` | bool, default auto-detect | Overrides the backup-tier capability detection (scan of recipe compose files for `backupbot.backup` labels, `generic.py:34`). `False` forces N/A; `True` forces the tier on. Users: custom-html-bkp-bad/rst-bad (harness self-test recipes). |
### 4.5 Readiness & data-verification hooks (loader L1, callable values)
| Key | Type / default | Meaning |
|---|---|---|
| `READY_PROBE` | callable `(domain) -> [probe, ...]`, default `None` | Extra readiness probes run after install AND after upgrade, before that tier's assertions. Probe dicts: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}` (`stable`: must stay connectable across 3 checks — for UDP-adjacent voice ports etc.). Consumed at `lifecycle.py:516`. Users: lasuite-drive, mumble (TCP voice port). |
| `BACKUP_VERIFY` | callable `(domain) -> bool`, default `None` | Post-backup data-capture check, retried — guards the truncated-dump race (backup snapshot taken before the seeded marker row hit disk). Return `False` → retry the backup, then fail. Users: discourse, ghost. |
### 4.6 Dependencies / SSO (loaders L5 + L1)
| Key | Type / default | Meaning |
|---|---|---|
| `DEPS` | list of recipe names, default `[]` | Dep recipes deployed alongside (e.g. `["keycloak"]`). Dep domain is `<dep[:4]>-<6hex>`, hashed from (parent, pr, ref, dep) — collision-free per run. Creds land in `$CCCI_DEPS_FILE` (JSON); tests use the `deps_apps` fixture; teardown deps LAST. Deploy-count guard becomes `1 + len(DEPS)`. Loaded by `deps.py:declared_deps`. Users: lasuite-docs/-drive/-meet. |
| `OIDC_AT_INSTALL` | bool, default `False` | Provision deps **before** the single base deploy so `install_steps.sh` can wire OIDC env into that one deploy (reads `$CCCI_DEPS_FILE`). Default (legacy) is post-deploy provisioning + a `setup_custom_tests.sh` redeploy. Consumed at `run_recipe_ci.py:514`. Users: lasuite-drive, lasuite-meet. |
### 4.7 Warm-canonical enrollment (loader L6)
| Key | Type / default | Meaning |
|---|---|---|
| `WARM_CANONICAL` | bool, default `False` | Enrolls the recipe in the warm/canonical app system (`docs/warm.md`): green COLD runs on LATEST advance the canonical snapshot; the nightly sweep iterates enrolled recipes. Loaded by `canonical.py:is_canonical_enrolled`. User: custom-html. |
### 4.8 Cosmetic (BROKEN — see §8 R2)
| Key | Type / default | Meaning |
|---|---|---|
| `SCREENSHOT` | callable `(page, domain, meta) -> None` | Drives Playwright to a safe post-login view for the results-card screenshot (default: landing page). **Currently unreachable from the CI path**: `screenshot.py:41` reads it from the meta dict the orchestrator passes (`run_recipe_ci.py:1056`), but the L1 allowlist never loads `SCREENSHOT`, so the hook is always `None`. No recipe sets it (consistent with it never having worked). |
Worked hook examples: cryptpad (`EXTRA_ENV(ctx)` derives `SANDBOX_DOMAIN` from `ctx.domain`),
mumble (`READY_PROBE(ctx)` TCP voice-port probe, `UPGRADE_EXTRA_ENV(ctx)` adds a head-only compose
overlay), ghost/discourse (`BACKUP_VERIFY(ctx)` dump-capture check).
## 5. Writing custom tests & hooks
@ -196,104 +161,122 @@ test runs additively against the same state.
Conventions (see `tests/immich/test_backup.py` etc.):
- use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain)
- use the `meta` fixture for HEALTH_*/timeouts (note: only the 4 base keys — §8 R3)
- read op context from `$CCCI_OP_STATE_FILE` (JSON written by the orchestrator after the op:
versions, artifact paths)
- use the `meta` fixture — the recipe's FULL validated `RecipeMeta` (attribute access)
- use the `op_state` fixture for op context (versions, `snapshot_id`, artifact paths — the
orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run)
- execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)`
### 5.2 Pre-op seed hooks — `ops.py`
`def pre_<op>(domain, meta)` callables, imported and called by the orchestrator **before**
performing the op. This is where data gets seeded so the post-op overlay can assert on it:
`def pre_<op>(ctx)` callables, imported and called by the orchestrator **before** performing the
op. This is where data gets seeded so the post-op overlay can assert on it:
```python
# tests/immich/ops.py (pattern)
def pre_upgrade(domain, meta): _psql(domain, "INSERT ... 'upgrade-survives'")
def pre_backup(domain, meta): _psql(domain, "INSERT ... 'original'")
def pre_restore(domain, meta): _psql(domain, "DROP TABLE ci_marker") # damage, restore must undo
def pre_upgrade(ctx): _psql(ctx.domain, "INSERT ... 'upgrade-survives'")
def pre_backup(ctx): _psql(ctx.domain, "INSERT ... 'original'")
def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, restore must undo
```
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
### 5.3 Custom tier — `functional/`, `playwright/`, top-level `test_*.py`
### 5.3 Custom tier — `functional/` and `playwright/` ONLY
All non-lifecycle `test_*.py` (discovery: `discovery.py:custom_tests`, recursive over the
top-level dir + `functional/` + `playwright/`; files named `test_<op>.py` excluded). Run in the
CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run —
cc-ci's and (if HC2-approved) repo-local's, additively.
All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
(discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
HC2-approved) repo-local's, additively.
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable).
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
Tests gate on deps via `CCCI_DEPS_READY` (skip-with-reason when `0`; the skip is counted and
fails the run if deps were declared but unprovisionable — `run_recipe_ci.py:816`).
Tests needing deps use the `deps` fixture (entries expose `.domain` plus the full creds dict) and
carry `@pytest.mark.requires_deps` — when dep provisioning failed they skip with reason
`deps-not-ready` and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit
must not mask an unrun SSO test). Fixtures replace direct `os.environ` reads — after the
restructure no recipe test parses env by hand.
### 5.4 Pre-deploy shell hook — `install_steps.sh`
Runs after `abra app new` + `EXTRA_ENV` application + secret generation, **before** the base
deploy. For setup that must precede the first deploy: writing extra config files into the recipe
checkout, copying in a `compose.ccci.yml` overlay (§5.6), editing `.env` beyond simple key=val.
The ONLY shell hook. Runs after `abra app new` + `EXTRA_ENV` application + secret generation,
**before** the single base deploy. For setup that must precede the first deploy: writing extra
config files into the recipe checkout, editing `.env` beyond simple key=val, and — for recipes
with `DEPS` — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the
deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy
redeploy hook).
Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and —
when `OIDC_AT_INSTALL` deps exist — `CCCI_DEPS_FILE`. Must locate the recipe checkout
ABRA_DIR-aware: `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run
`ABRA_DIR` since the concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).
when `DEPS` is declared — `CCCI_DEPS_FILE` (jq-readable JSON of dep creds/URLs; see
lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware:
`RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run `ABRA_DIR` since the
concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).
Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic
install — a correct reported outcome, not a harness error.
### 5.5 Deps credential wiring — `setup_custom_tests.sh`
### 5.5 CI-only compose overlay — `compose.ccci.yml`
For legacy (post-deploy) deps provisioning: runs after deps are up, reads `$CCCI_DEPS_FILE`
(jq-readable JSON of dep creds/URLs), wires OIDC config via `abra app config set` + secrets, and
redeploys. With `OIDC_AT_INSTALL = True` this hook is unnecessary (wiring happens in
`install_steps.sh` before the only deploy) — preferred for new enrollments (one deploy, no
deploy-count exception).
**First-class:** if `tests/<recipe>/compose.ccci.yml` exists, the harness itself copies it into
the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses `--chaos` for
that deploy (the untracked file would otherwise trip abra's clean-tree gate). No
`install_steps.sh` copy boilerplate, no flag to remember (the old `CHAOS_BASE_DEPLOY` ⇄ overlay
coupling is gone). The overlay is cc-ci-owned only.
### 5.6 CI-only compose overlay — `compose.ccci.yml`
Policy unchanged: overlays are a minimal, justified fallback (ghost's is a 15m `start_period`
grace — a literal, because abra validates `start_period` before env substitution). Reference the
overlay from `EXTRA_ENV`'s `COMPOSE_FILE` as usual. Users: ghost, discourse.
Not auto-discovered: `install_steps.sh` copies it into the recipe checkout, and the recipe must
set `CHAOS_BASE_DEPLOY = True` so the base deploy (`--chaos`) tolerates the untracked file.
Policy: minimal, justified fallback only (ghost's is a 15m `start_period` grace — a literal,
because abra validates `start_period` before env substitution). The overlay is cc-ci-owned even
though it rides in the recipe checkout.
### 5.6 Environment & fixture contract (what custom code can read)
### 5.7 Environment contract summary (what custom code can read)
Pytest fixtures (`tests/conftest.py` — the single fixture file):
| Fixture | Yields |
|---|---|
| `recipe` | the recipe name (`$RECIPE`) |
| `meta` | the FULL validated `RecipeMeta` (single loader) |
| `live_app` | the shared deployment's domain (asserts it exists) |
| `op_state` | the orchestrator's op-context dict (skips cleanly outside a run) |
| `deps` | `{dep_recipe: entry}` — entries expose `.domain` + full SSO creds |
Environment (hooks/shell, and approved repo-local code):
| Var | Set for | Meaning |
|---|---|---|
| `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain |
| `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` |
| `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path |
| `CCCI_OP_STATE_FILE` | overlay tests | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | deps hooks + tests | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier | gate SSO tests, skip-with-reason |
| `CCCI_OP_STATE_FILE` | overlay tests (via `op_state`) | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | `install_steps.sh` + harness | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier (via `requires_deps`) | gate SSO tests, skip-with-reason |
## 6. Run-model context (what the settings plug into)
One deploy chain per run (full detail: `docs/testing.md` §2):
```
[DEPS? provision deps FIRST → $CCCI_DEPS_FILE]
deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh;
CHAOS_BASE_DEPLOY?; OIDC_AT_INSTALL deps first?)
compose.ccci.yml auto-copied + auto-chaos)
→ INSTALL tier (READY_PROBE; generic + overlay asserts)
→ pre_upgrade → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
→ pre_upgrade(ctx) → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
→ UPGRADE tier (READY_PROBE; version-label == head_ref)
→ pre_backup → backup (BACKUP_CAPABLE; BACKUP_VERIFY)
→ pre_backup(ctx) → backup (BACKUP_CAPABLE; BACKUP_VERIFY)
→ BACKUP tier
→ pre_restore → restore
→ pre_restore(ctx) → restore
→ RESTORE tier
→ CUSTOM tier (functional/ + playwright/; deps via CCCI_DEPS_*)
→ CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
→ SCREENSHOT (best-effort, never affects the verdict)
→ teardown (deps LAST)
```
Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't
count); the per-run counter file is keyed by run since the concurrency restructure.
## 7. Local iteration
## 7. Local iteration, the manifest, and the dev-only escape hatch
```
RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
@ -303,81 +286,75 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
(`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.)
## 8. Known limitations & restructuring candidates
**Customization manifest.** Every run prints, right after meta load + discovery, one block:
The review section. Ordered by how much they'd shape a restructure.
```
===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci)
env overrides: (none)
```
**R1 — Six divergent meta loaders (the core drift hazard).** §4's L1–L6: every loader re-`exec()`s
`recipe_meta.py` and cherry-picks its own keys. Adding a key means knowing *which* loader to touch
(or that you must extend the L1 allowlist — `SCREENSHOT` proves people don't, R2). Two conventions
coexist: L1's explicit allowlist vs L3–L6's ad-hoc `ns.get(...)` which silently bypasses it.
*Candidate:* one `harness.meta.load(recipe) -> RecipeMeta` with a declarative key registry
(name, type, default, validator, consumer) as the single source of truth; L1–L6 become lookups
into the one loaded object; the registry also generates §4 of this doc (kills doc drift, R5).
The same dict is embedded in `results.json` under `"customization"`. It is pure presentation —
built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes,
and it honors the HC2 gate) — and never influences a verdict.
**R2 — `SCREENSHOT` is a dead knob.** Fully implemented consumer (`screenshot.py`), documented
hook contract, never reachable: the orchestrator's allowlist omits it, so the dict passed at
`run_recipe_ci.py:1056` can never contain it. Direct evidence of R1. *Candidate:* fix trivially by
adding to the allowlist — or delete the hook path if post-login screenshots aren't wanted; decide
during the restructure.
**Dev-only generic skip.** `CCCI_SKIP_GENERIC=1` (all ops) / `CCCI_SKIP_GENERIC_<OP>=1` (one op)
suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no
declarative equivalent (the old `SKIP_GENERIC` meta key is deleted). If the env form is active in
a CI (drone) run, the run prints a loud `!!` warning and the manifest records it.
**R3 — The pytest `meta` fixture sees 4 keys.** `tests/conftest.py:_recipe_meta` loads only
HEALTH_*/timeouts. An overlay test wanting e.g. `EXPECTED_NA` or a recipe constant must re-exec
the file itself. Probably intended minimalism, but it's a third key-set to keep in sync.
*Folds into R1.*
## 8. Restructure outcomes (the review spec's R1–R9)
**R4 — Settings split across three config languages** (§1): recipe_meta keys, file-presence
(`install_steps.sh` existing changes deploy behavior), and run-time env (`CCCI_SKIP_GENERIC*`).
A reviewer asking "what does this recipe customize?" must check all three. *Candidate:* keep the
three surfaces (they serve different actors) but make the run header log a single resolved
"customization manifest" per run: every non-default key + every discovered hook file + every
CCCI_* override, in one block.
How each defect identified in the review spec (commit `76a4b6b` §8) was resolved:
**R5 — Reference-doc drift already happened.** `docs/testing.md` documents 6 meta keys,
`docs/enroll-recipe.md` shows others by example; neither is complete (18 keys exist). This doc is
now complete but handwritten — it will drift too. *Candidate:* generate the key table from the R1
registry (test asserts doc ⊆ registry).
- **R1 — six divergent meta loaders → RESOLVED.** One registry-backed loader
(`harness/meta.py::load`), the only `exec()` of `recipe_meta.py`. The orchestrator loads once
and passes the `RecipeMeta` down; conftest/lifecycle/deps/canonical all read the one object.
- **R2 — dead `SCREENSHOT` knob → RESOLVED (kept + fixed).** The registry replaced the allowlist
that orphaned it; the orchestrator path now delivers the hook to `screenshot.py`
(proven end-to-end by `tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path`).
- **R3 — 4-key pytest `meta` fixture → RESOLVED.** The fixture returns the full validated
`RecipeMeta`.
- **R4 — three config languages → MITIGATED by the manifest** (§7): the surfaces stay (they serve
different actors), but every run resolves them into one visible block + results key.
- **R5 — reference-doc drift → RESOLVED.** §4's key table is generated from the registry
(`scripts/gen-meta-docs.py`); a unit test fails CI on drift; `testing.md`/`enroll-recipe.md`
point here instead of keeping partial lists.
- **R6 — silent typos → RESOLVED.** Unknown ALL-CAPS keys and type mismatches are hard
`MetaError`s; private constants are underscore-prefixed (exempt).
- **R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` coupling → RESOLVED.** The overlay is
first-class: harness-copied, auto-chaos. The flag is deleted.
- **R8 — zero-user `SKIP_GENERIC` meta key → RESOLVED (deleted).** Env form remains, documented
dev-only, loudly flagged in CI runs (§7).
- **R9 — `recipe_meta.py` is code, not config → REJECTED by decision.** No data/hooks file split:
registry validation gets the value (typed, validated keys) at lower cost; one file per recipe
remains the single config place. The expressiveness need is real (cryptpad derives env from the
per-run domain).
**R6 — No schema validation / silent typos.** Unknown top-level names in `recipe_meta.py` are
ignored, which is load-bearing (recipes keep private constants there: mumble's
`WELCOME_TEXT_MARKER`, `MAX_USERS`). Consequence: misspelling `READY_PROBE` as `READINESS_PROBE`
silently disables the probe — the run goes green with less coverage, the worst failure mode for a
CI harness. *Candidate:* with the R1 registry, warn (not fail) on ALL-CAPS top-level names that
are not registered and not referenced by the recipe's own tests; or namespace private constants
(`_WELCOME_TEXT_MARKER`).
**R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` implicit coupling.** The overlay only works if
the recipe *also* sets the flag; forgetting it fails the base deploy with an abra
untracked-files error far from the cause. *Candidate:* if `install_steps.sh` exists alongside a
`compose.ccci.yml`, the harness could auto-enable chaos for the base deploy (or at least assert
the flag and fail with a pointed message).
**R8 — `SKIP_GENERIC` (meta form) has zero users.** Only the env-var form is used, ad hoc. Either
the meta key earns its place (first real user) or it's surface to delete in the restructure.
**R9 — `recipe_meta.py` is code, not config.** Five keys take callables (`EXTRA_ENV`,
`UPGRADE_EXTRA_ENV`, `READY_PROBE`, `BACKUP_VERIFY`, `SCREENSHOT`), so the file must stay an
`exec()`d Python module — it can't be validated as data, serialized into results, or diffed
declaratively. This is a real expressiveness need (cryptpad derives `SANDBOX_DOMAIN` from the
per-run domain), not an accident. *Candidate if restructuring:* split data keys (TOML-able,
schema-validated) from a `hooks.py` (callables only) — but weigh against the cost of two files
per recipe; the R1 registry gets most of the value without the split.
Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy
post-deploy `setup_custom_tests.sh` machinery and its extra redeploy are deleted); the custom-test
placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface
(§5.6 — `deps` replaces `deps_apps`+`deps_creds`; dead `deployed`/`deployed_app`/`app_domain`
fixtures deleted).
## 9. File / symbol index
| Concern | Where |
|---|---|
| Orchestrator meta loader (L1, allowlist) | `runner/run_recipe_ci.py:250` `_load_meta` |
| Pytest meta fixture (L2) | `tests/conftest.py` `_recipe_meta` |
| `EXTRA_ENV` loader (L3) | `runner/harness/lifecycle.py:114` `_recipe_extra_env` |
| Boolean-flag loader (L4) | `runner/harness/lifecycle.py:132` `_recipe_meta_flag` |
| `DEPS` loader (L5) | `runner/harness/deps.py:37` `declared_deps` |
| `WARM_CANONICAL` loader (L6) | `runner/harness/canonical.py:36` `is_canonical_enrolled` |
| Overlay/custom/hook discovery + HC2 gate | `runner/harness/discovery.py` |
| THE meta loader + key registry + `HookCtx` + `MetaError` | `runner/harness/meta.py` (`load`, `KEYS`, `check_hook_signature`) |
| Generated key table | `scripts/gen-meta-docs.py` → §4 above (sync pinned by `tests/unit/test_meta.py`) |
| Customization manifest | `runner/harness/manifest.py` (`build`, `render`), printed by `runner/run_recipe_ci.py` |
| Overlay/custom/hook discovery + HC2 gate + placement rule | `runner/harness/discovery.py` |
| HC2 allowlist | `tests/repo-local-approved.txt` |
| Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` |
| `READY_PROBE` / `CHAOS_BASE_DEPLOY` consumption | `runner/harness/lifecycle.py:516` / `:283` |
| `compose.ccci.yml` auto-copy + auto-chaos | `runner/harness/lifecycle.py` (`provide_ccci_overlay`, `deploy_app`) |
| `READY_PROBE` consumption | `runner/harness/lifecycle.py` (`wait_ready_probes`) |
| `EXPECTED_NA` reporting | `runner/harness/results.py` |
| Dead `SCREENSHOT` consumer | `runner/harness/screenshot.py:36`, called `run_recipe_ci.py:1056` |
| Skip-generic logic (meta + env) | `runner/run_recipe_ci.py:285` |
| Worked examples | `tests/ghost/` (overlay+chaos), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV), `tests/lasuite-drive/` (DEPS+OIDC_AT_INSTALL), `tests/immich/` (ops.py seed pattern) |
| `SCREENSHOT` consumer | `runner/harness/screenshot.py` |
| Fixtures (`recipe`/`meta`/`live_app`/`op_state`/`deps`) + F2-11 skip-report | `tests/conftest.py` |
| Skip-generic env logic (dev-only) | `runner/run_recipe_ci.py` (`_skip_generic`) |
| Unit tests pinning all of the above | `tests/unit/test_meta.py`, `test_manifest.py`, `test_discovery*.py` |
| Worked examples | `tests/ghost/` (overlay+compose.ccci.yml), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV, private `_` constants), `tests/lasuite-drive/` (DEPS + install-time OIDC wiring), `tests/immich/` (ops.py seed pattern) |

61
docs/testing.md
View File

@ -16,12 +16,13 @@ year from now, this is the one rule that should still hold.
ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state
scaffolding — just "does this recipe deploy and lifecycle work?"
- **Generic must not depend on custom.** A custom test or a custom-tests setup (e.g. SSO/OIDC dep
provisioning) **can never be a precondition for the generic tier to pass.** Concretely: the
orchestrator runs all generic tiers (install → upgrade → backup → restore) against the recipe
**alone, with no deps deployed**, then runs the `setup_custom_tests` step (deps + post-deps
wiring) only after — and a failure there is **isolated** to the custom tier (tests tagged
`@pytest.mark.requires_deps` skip with reason `"deps-not-ready"`; generic tier reports
normally). See `cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics.
provisioning) **can never be a precondition for the generic tier to pass.** Concretely: deps are
provisioned BEFORE the single deploy (so `install_steps.sh` can wire OIDC env into that one
deploy), but a dep-provisioning failure is **isolated** to the custom tier — the recipe still
deploys alone, every generic tier (install → upgrade → backup → restore) runs normally, and
tests tagged `@pytest.mark.requires_deps` skip with reason `"deps-not-ready"` (a counted,
reported skip — F2-11). A deps failure can never fail or block a generic tier. See
`cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics.
- **Custom tests are the thoroughness layer — and they cost more to maintain.** They're more
thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper
scripts, state-management) and *therefore* take more maintenance: an SSO provider's admin API
@ -113,9 +114,11 @@ repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
addition** unless explicitly opted out.
**Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and
additive**: it has no generic equivalent and runs only when present, discovered from both locations
(repo-local gated by the HC2 allowlist).
**Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**:
they have no generic equivalent and run only when present, discovered from both locations
(repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under
`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else
(top-level non-lifecycle files are not discovered).
### Pre-op seed hooks (per-recipe `ops.py`)
@ -127,35 +130,38 @@ etc.). Since the orchestrator owns the op, overlays place their seed in an optio
# tests/<recipe>/ops.py
from harness import lifecycle
def pre_upgrade(domain, meta):
def pre_upgrade(ctx):
# seed a marker before the harness performs the upgrade
lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"])
lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo upgrade-survives > /path/marker"])
def pre_backup(domain, meta):
def pre_backup(ctx):
# establish a known "original" state before the backup op captures it
lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"])
lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo original > /path/marker"])
def pre_restore(domain, meta):
def pre_restore(ctx):
# diverge from the backed-up state so a successful restore is observable
lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"])
lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo mutated > /path/marker"])
```
The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import
sibling helpers like `kc_admin.py`) and calls `pre_<op>(domain, meta)` immediately before performing
the op. Then `test_<op>.py` asserts the post-op state. See `tests/custom-html/` (volume marker),
sibling helpers like `kc_admin.py`) and calls `pre_<op>(ctx)` immediately before performing the
op — `ctx` is the uniform `HookCtx` every recipe hook receives (`.domain`, `.base_url`, `.meta`,
`.deps`, `.op` — `docs/recipe-customization.md` §4.1). Then `test_<op>.py` asserts the post-op
state. See `tests/custom-html/` (volume marker),
`tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db`
service) for worked examples.
### Opting out of the generic floor
### Opting out of the generic floor (LOCAL-DEV-ONLY)
The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check
fully replaces the generic's mechanism check) set, in increasing specificity:
The generic runs additively by default and there is **no declarative opt-out** — no recipe can
ship without the floor. For local iteration only (e.g. re-running one tier while developing an
overlay), two env escape hatches exist:
- **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide).
- **env `CCCI_SKIP_GENERIC_<OP>=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op.
- **declarative in `recipe_meta.py`** — `SKIP_GENERIC = ["upgrade"]` (per-op) or `SKIP_GENERIC = ["all"]`.
Opting out is per-recipe and visible in git — not a hidden global. Truthy = `1`/`true`/`yes`/`on`.
Truthy = `1`/`true`/`yes`/`on`. If either is active in a CI (drone) run, the run prints a loud
`!!` warning and the customization manifest records it (`docs/recipe-customization.md` §7).
## Repo-local trust gate (HC2) — default-deny
@ -215,12 +221,14 @@ installs and stays 1.
`tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through
`lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run.
3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore
divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(domain, meta)`.
divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(ctx)`.
4. If the recipe needs install-time setup, add `tests/<recipe>/install_steps.sh`.
5. Set per-recipe knobs (health path, timeouts, opt-out) in `recipe_meta.py`.
5. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`.
6. **Never weaken or skip an assertion to make a run pass** — a red tier is information.
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional):
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional — the COMPLETE key reference is
the generated table in `docs/recipe-customization.md` §4; unknown keys are hard errors, private
constants are underscore-prefixed):
```python
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
@ -228,8 +236,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"] # per-recipe declarative opt-out from generic ops ("all" = every op)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy
```
The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run: