recipe-maintainers/cc-ci
0
0
Fork 0
Code Issues 1 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

docs: P6 — rewrite customization docs to the restructured end state (rcust)
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse Source 
recipe-customization.md: review spec -> reference. Single registry-backed loader + validation
rules + HookCtx convention (§4); generated key table kept byte-identical (sync test); §5 end-state
shape (op_state/deps fixtures, ctx ops.py, placement rule, first-class compose.ccci.yml, no
setup_custom_tests.sh); §7 manifest block + dev-only CCCI_SKIP_GENERIC*; §8 rewritten as
restructure outcomes (R1/R2/R3/R5/R6/R7/R8 resolved + how, R4 mitigated by manifest, R9
rejected-by-decision); §9 index updated to the new symbols.

testing.md: install-time deps isolation replaces the setup_custom_tests step in the invariant
(generic still never depends on custom — failure isolation via requires_deps/F2-11); ops.py
example to pre_<op>(ctx); placement rule; generic opt-out now documented LOCAL-DEV-ONLY env with
CI !! warning (declarative SKIP_GENERIC gone); partial key list points at the generated table.

enroll-recipe.md: tree + worked examples updated (lasuite-docs install-time OIDC wiring +
install_steps.sh; mumble post-F2-14c shape — UPGRADE_EXTRA_ENV native overlay, private _
constants, no CHAOS_BASE_DEPLOY); deps fixture (entry.domain) replaces deps_apps; ctx hook
signatures; compose.ccci.yml first-class bullet; key list points at the generated table.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
autonomic-bot
2026-06-10 19:07:41 +00:00
parent 68954be53e
commit da558ca946
3 changed files with 268 additions and 272 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
docs/enroll-recipe.md
View File

@ -14,8 +14,9 @@ those are discovered and run against the live app (D4 — see below).
``` ```
tests/<recipe>/ tests/<recipe>/
├── recipe_meta.py # optional per-recipe harness config (see below) ├── recipe_meta.py # optional per-recipe harness config (see below)
├── install_steps.sh # optional custom install-steps hook (pre-deploy setup) ├── install_steps.sh # optional custom install-steps hook (pre-deploy setup + deps env wiring)
├── ops.py # optional pre-op seed hooks (pre_install/pre_upgrade/pre_backup/pre_restore) ├── compose.ccci.yml # optional CI-only compose overlay (harness-copied, auto-chaos base deploy)
├── ops.py # optional pre_<op>(ctx) seed hooks (install/upgrade/backup/restore)
├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic) ├── test_install.py # optional install overlay (runs ADDITIVELY alongside generic)
├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic) ├── test_upgrade.py # optional upgrade overlay (runs ADDITIVELY alongside generic)
├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic) ├── test_backup.py # optional backup overlay (runs ADDITIVELY alongside generic)
@ -39,11 +40,14 @@ To add recipe-specific coverage, drop a `tests/<recipe>/test_<op>.py` **overlay*
**ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently **ALONGSIDE** the generic for that op (HC3 additive, Phase 1e); the generic floor is never silently
dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture; dropped. Overlays are **assertion-only** against the shared live deployment (the `live_app` fixture;
they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to they never perform the op or deploy/teardown — the orchestrator owns those). If the overlay needs to
SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(domain, SEED pre-op state (data-continuity markers, the backup→restore divergence), put `pre_<op>(ctx)`
meta)` callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op. Copy an callables in `tests/<recipe>/ops.py` — the orchestrator runs them BEFORE the op (`ctx` is the
uniform `HookCtx` every hook receives — `docs/recipe-customization.md` §4.1). Copy an
existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/ existing recipe (`tests/custom-html/` simple/volume marker; `tests/keycloak/` admin-API; `tests/
matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` / matrix-synapse/` `db`-service psql marker). **Do not edit the shared `tests/conftest.py` /
`runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py`: `runner/harness/` to add a recipe** — set per-recipe knobs in `recipe_meta.py` (the COMPLETE key
reference is the generated table in `docs/recipe-customization.md` §4; unknown ALL-CAPS keys are
hard errors, recipe-private constants are underscore-prefixed `_FOO`):
```python ```python
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
@ -51,9 +55,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose) BACKUP_CAPABLE = True # override backup-capability auto-detect (default: scan compose)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"] # per-recipe opt-out from the generic floor for the listed ops
# ("all"/"*" = every op); rarely needed — generic is the floor
``` ```
Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`, Useful `harness.lifecycle` helpers for overlays: `http_get`, `http_fetch`, `http_body`,
@ -76,9 +78,10 @@ Beyond the lifecycle overlays, each recipe carries (plan §4.1):
- **`playwright/`** — browser flows where the recipe's core UX is a UI (P6). - **`playwright/`** — browser flows where the recipe's core UX is a UI (P6).
The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/` The orchestrator's **custom** tier discovers `test_*.py` in `tests/<recipe>/{functional,playwright}/`
(recursive, via `runner/harness/discovery.custom_tests`) and runs each as its own pytest against ONLY (the placement rule, via `runner/harness/discovery.custom_tests` — a top-level `test_*.py`
the same `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are is a lifecycle overlay and nothing else) and runs each as its own pytest against the same
**excluded** from the custom tier — they live at the top level and run as lifecycle overlays. `live_app` shared deployment. Lifecycle-named files (`test_install.py`/etc.) are **excluded**
from the custom tier even inside those subdirs (safety net against double-running).
### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3) ### 2.2 Recipe-test dependencies — DEPS = [...] (Phase 2 Q2.3)
@ -89,23 +92,28 @@ them in `recipe_meta.py`:
DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests/<dep>/ must exist + work) DEPS = ["keycloak"] # one entry per dep recipe name (cc-ci tests/<dep>/ must exist + work)
``` ```
The orchestrator (plan §4.2): The orchestrator (plan §4.2; install-time provisioning is the ONLY mode):
1. Reads `DEPS` BEFORE deploying the recipe under test. 1. Reads `DEPS` and provisions every dep **BEFORE the single deploy** of the recipe under test —
2. Deploys each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is each dep at a per-run domain `<dep[:4]>-<6hex>.ci.commoninternet.net` (the 6hex is hashed from
hashed from `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do `parent_recipe + pr + ref + dep_recipe` so two recipes' deps of the same kind do not collide on
not collide on a single node). a single node), waited healthy using the dep's own `recipe_meta.py`.
3. Waits each dep healthy using its own `recipe_meta.py` (HEALTH_PATH/HEALTH_OK/timeouts). 2. Persists the full per-dep identity + SSO creds dict to `$CCCI_DEPS_FILE` (jq-readable JSON,
4. Persists `[{"recipe": "<dep>", "domain": "<dep-domain>"}, ...]` to `$CCCI_DEPS_FILE`. `{"<dep>": {"domain": ..., "realm": ..., "client_secret": ..., ...}}`).
5. Deploys + tests the recipe under test as usual. 3. Deploys the recipe under test — its `install_steps.sh` reads `$CCCI_DEPS_FILE` and wires
6. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked OIDC env into that ONE deploy (no post-deploy redeploy). A dep-provisioning failure does NOT
block the run: the recipe deploys alone, generic tiers run, and `requires_deps` tests skip
with a counted reason (F2-11).
4. Tears down the dep LAST in `finally` (reverse declaration order, with `verify=True` — leaked
deps fail the run loudly per §9 teardown sacred / F2-5 fix). deps fail the run loudly per §9 teardown sacred / F2-5 fix).
Tests access dep domains via the **`deps_apps` pytest fixture** (`tests/conftest.py`): Tests access deps via the **`deps` pytest fixture** (`tests/conftest.py`) — entries expose
`.domain` plus the full creds dict (attribute or dict-style):
```python ```python
def test_my_recipe_uses_keycloak(live_app, deps_apps): @pytest.mark.requires_deps
assert "keycloak" in deps_apps, f"keycloak dep not deployed; {deps_apps}" def test_my_recipe_uses_keycloak(live_app, deps):
kc_domain = deps_apps["keycloak"] assert "keycloak" in deps, f"keycloak dep not deployed; {deps}"
kc_domain = deps["keycloak"].domain
… …
``` ```
@ -120,7 +128,7 @@ For OIDC-dependent recipes, the shared `runner/harness/sso.py` provides:
from harness import sso from harness import sso
creds = sso.setup_keycloak_realm( creds = sso.setup_keycloak_realm(
kc_domain, # = deps_apps["keycloak"] kc_domain, # = deps["keycloak"].domain
realm="my-realm", realm="my-realm",
client_id="my-client", client_id="my-client",
redirect_uris=[f"https://{live_app}/*"], redirect_uris=[f"https://{live_app}/*"],
@ -144,10 +152,10 @@ ARE provider-pluggable.
Not every recipe is a single HTTP app. `recipe_meta.py` + a few harness mechanisms cover the harder Not every recipe is a single HTTP app. `recipe_meta.py` + a few harness mechanisms cover the harder
shapes (proven on mumble, mailu, and the SSO-dependent suite): shapes (proven on mumble, mailu, and the SSO-dependent suite):
- **`EXTRA_ENV`** — a dict **or** a `callable(domain) -> dict`. The callable form derives values from - **`EXTRA_ENV`** — a dict **or** a `callable(ctx) -> dict`. The callable form derives values from
the per-run domain (e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for cryptpad). Applied the per-run domain (`ctx.domain` — e.g. `MAIL_DOMAIN`/`HOSTNAMES` for mailu, `SANDBOX_DOMAIN` for
at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change. cryptpad). Applied at every deploy (`abra.env_set`), so a recipe enrolls with NO shared-harness change.
- **`READY_PROBE(domain) -> [...]`** — readiness signals beyond replica-convergence + the app's - **`READY_PROBE(ctx) -> [...]`** — readiness signals beyond replica-convergence + the app's
`HEALTH_PATH`. Two probe shapes: `HEALTH_PATH`. Two probe shapes:
- HTTP: `{"host": "...", "path": "/...", "ok": (200,)}` (e.g. lasuite-drive collabora WOPI discovery). - HTTP: `{"host": "...", "path": "/...", "ok": (200,)}` (e.g. lasuite-drive collabora WOPI discovery).
- **TCP**: `{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}` — polls a socket connect N - **TCP**: `{"tcp_host": "127.0.0.1", "tcp_port": 64738, "stable": 3}` — polls a socket connect N
@ -155,16 +163,16 @@ shapes (proven on mumble, mailu, and the SSO-dependent suite):
service (mumble: the mumble-web sidecar serves HTTP 200 while the voice server on 64738 is still service (mumble: the mumble-web sidecar serves HTTP 200 while the voice server on 64738 is still
rebinding after an upgrade redeploy — the TCP probe gates the backup tier until the voice server is rebinding after an upgrade redeploy — the TCP probe gates the backup tier until the voice server is
actually up). Runs after install AND after the upgrade chaos redeploy. actually up). Runs after install AND after the upgrade chaos redeploy.
- **`CHAOS_BASE_DEPLOY = True`** — make the pinned base deploy use `--chaos` (skips abra's clean-tree + - **`compose.ccci.yml`** (first-class at `tests/<recipe>/compose.ccci.yml`) — a CI-only compose
lint gates, still deploys the explicitly-checked-out pinned version, NOT latest). Needed when an overlay the harness itself copies into the recipe checkout before the base deploy, automatically
`install_steps.sh` adds an UNTRACKED file to the recipe checkout (e.g. mumble copies a using `--chaos` for that deploy (the untracked file would otherwise trip abra's pinned-deploy
`compose.host-ports.yml` into versions that predate it) — abra's pinned-deploy clean-tree check would clean-tree check). Reference it from `EXTRA_ENV`'s `COMPOSE_FILE`. Minimal, justified fallback
otherwise FATA. `abra.recipe_checkout` force-checks-out (`-f`) so the upgrade tier's re-checkout to only (e.g. ghost's 15m `start_period` grace). `abra.recipe_checkout` force-checks-out (`-f`) so
PR-head overwrites such overlays cleanly. the upgrade tier's re-checkout to PR-head overwrites such overlays cleanly.
- **`install_steps.sh`** (auto-discovered at `tests/<recipe>/install_steps.sh`) — runs after - **`install_steps.sh`** (auto-discovered at `tests/<recipe>/install_steps.sh`) — runs after
`abra app new` + EXTRA_ENV + secret-generate, BEFORE the single deploy, with `CCCI_APP_DOMAIN` / `abra app new` + EXTRA_ENV + secret-generate, BEFORE the single deploy, with `CCCI_APP_DOMAIN` /
`CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when DEPS are provisioned at install). Use it to `CCCI_APP_ENV` / `CCCI_RECIPE` (and `CCCI_DEPS_FILE` when the recipe declares DEPS — deps are
drop a cc-ci-owned compose overlay into the checkout, wire dep-derived env/secrets, etc. always provisioned before the deploy). Use it to wire dep-derived env/secrets, seed config, etc.
**Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports **Non-HTTP protocol tests (mumble).** Reach a TCP service published `mode: host` (via a host-ports
overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol overlay) at `127.0.0.1:<port>` — cc-ci runs tests on-host (cc-ci-run). mumble ships a stdlib protocol
@ -227,9 +235,10 @@ RECIPE=<recipe> PR=<n> REF=<sha-or-branch> SRC=recipe-maintainers/<recipe> \
``` ```
tests/lasuite-docs/ tests/lasuite-docs/
├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(domain) for cold-pull, ├── recipe_meta.py # HEALTH_PATH="/", DEPLOY_TIMEOUT=900, EXTRA_ENV(ctx) for cold-pull,
│ # DEPS=["keycloak"] ← Phase 2 dep declaration │ # DEPS=["keycloak"] ← Phase 2 dep declaration
├── ops.py # pre_<op> seed hooks (volume marker for backup/restore data-integrity) ├── install_steps.sh # wires OIDC env from $CCCI_DEPS_FILE into the single deploy
├── ops.py # pre_<op>(ctx) seed hooks (volume marker for backup/restore data-integrity)
├── test_install.py # lifecycle install overlay (Playwright frontend SPA load) ├── test_install.py # lifecycle install overlay (Playwright frontend SPA load)
├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy) ├── test_upgrade.py # lifecycle upgrade overlay (marker survives chaos redeploy)
├── test_backup.py # lifecycle backup overlay (marker captured) ├── test_backup.py # lifecycle backup overlay (marker captured)
@ -239,12 +248,14 @@ tests/lasuite-docs/
├── test_health_check.py # parity port (SOURCE comment cites recipe-info file) ├── test_health_check.py # parity port (SOURCE comment cites recipe-info file)
├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth ├── test_auth_required.py # specific: /api/v1.0/users/me/ → 401 without auth
└── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses └── test_oidc_with_keycloak.py # specific: full OIDC flow against the dep keycloak (uses
# harness.sso primitives + deps_apps["keycloak"]) # harness.sso primitives + the `deps` fixture)
``` ```
`!testme` on a lasuite-docs PR drives the orchestrator to: `!testme` on a lasuite-docs PR drives the orchestrator to:
1. Deploy the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`) and wait healthy. 1. Provision the per-run keycloak dep (`keyc-<6hex>.ci.commoninternet.net`), wait healthy, write
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`). creds to `$CCCI_DEPS_FILE` — BEFORE the recipe deploy.
2. Deploy lasuite-docs (`lasu-<6hex>.ci.commoninternet.net`); `install_steps.sh` wires the OIDC
env into that one deploy.
3. Run install / upgrade / backup / restore + the 3 functional tests against the shared 3. Run install / upgrade / backup / restore + the 3 functional tests against the shared
deployment (custom tier). deployment (custom tier).
4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True. 4. Teardown lasuite-docs, then the keycloak dep (LAST), both with verify=True.
@ -254,12 +265,13 @@ tests/lasuite-docs/
### Other shapes (concrete references) ### Other shapes (concrete references)
- **TCP / voice recipe — `tests/mumble/`**: `recipe_meta.py` (EXTRA_ENV sets - **TCP / voice recipe — `tests/mumble/`**: `recipe_meta.py` (EXTRA_ENV sets
`COMPOSE_FILE=compose.yml:compose.mumbleweb.yml:compose.host-ports.yml`, `WELCOME_TEXT`/`USERS` `COMPOSE_FILE=compose.yml:compose.mumbleweb.yml` for the base; `UPGRADE_EXTRA_ENV` adds the
markers, `CHAOS_BASE_DEPLOY=True`, `READY_PROBE` TCP 64738), `install_steps.sh` (provides the native `compose.host-ports.yml` at PR-head so 64738 is host-published on latest; private
host-ports overlay to older versions), `functional/_mumble_proto.py` + the protocol/config-round-trip `_WELCOME_TEXT_MARKER`/`_MAX_USERS` constants; `READY_PROBE(ctx)` TCP 64738 — phase-aware via
the live COMPOSE_FILE), `functional/_mumble_proto.py` + the protocol/config-round-trip
tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4. tests, `ops.py`/`test_backup.py`/`test_restore.py` (sqlite P4). See §2.4.
- **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py` - **Multi-service, dep-less, in-container functional — `tests/mailu/`**: `recipe_meta.py`
(`EXTRA_ENV(domain)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`), (`EXTRA_ENV(ctx)` with `TLS_FLAVOR=notls` + `MAIL_DOMAIN`/`HOSTNAMES`/`TRAEFIK_STACK_NAME`),
`functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back), `functional/_mailu.py` (flask-CLI helpers), `test_mailbox.py` (create→config-export read-back),
`test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md + `test_mail_flow.py` (in-container sendmail→doveadm delivery). No backupbot → P4 N/A (PARITY.md +
DEFERRED.md). See §2.4. DEFERRED.md). See §2.4.

375
docs/recipe-customization.md
View File

@ -1,8 +1,9 @@
# Recipe customization — review spec # Recipe customization — reference
Status: REVIEW SPEC — describes the customization surface as it exists today (main), written so Status: REFERENCE — describes the customization system as restructured on branch
the structure can be reviewed and potentially restructured. §8 lists known limitations and `restructure/recipe-custom` (the "rcust" restructure). The pre-restructure system and its defects
restructuring candidates; everything before it is purely descriptive. are documented in this file's history (commit `76a4b6b`, the review spec whose §8 R1–R9 drove the
restructure); §8 below records how each was resolved.
Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md` Companion docs: `docs/testing.md` (test architecture / tier semantics), `docs/enroll-recipe.md`
(step-by-step enrollment). This doc is the **complete reference** for the two questions those docs (step-by-step enrollment). This doc is the **complete reference** for the two questions those docs
@ -15,17 +16,18 @@ answer only partially:
## 1. The three customization surfaces ## 1. The three customization surfaces
A recipe customizes its CI through **three distinct mechanisms** (worth noticing for the A recipe customizes its CI through **three distinct mechanisms**:
restructure review — they are three different config languages):
| Surface | Form | Examples | | Surface | Form | Examples |
|---|---|---| |---|---|---|
| **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` | | **Declarative settings** | Python assignments in `tests/<recipe>/recipe_meta.py` | `DEPLOY_TIMEOUT = 1500`, `UPGRADE_BASE_VERSION = "2.3.1+..."` |
| **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, shell hooks | `def READY_PROBE(domain): ...`, `pre_upgrade()`, `install_steps.sh` | | **Code hooks** | Callables in `recipe_meta.py`, `ops.py` functions, one shell hook | `def READY_PROBE(ctx): ...`, `pre_upgrade(ctx)`, `install_steps.sh` |
| **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` | | **File presence** | A file existing at a discovered path changes behavior | `test_upgrade.py` overlay, `functional/test_*.py`, `compose.ccci.yml` |
There is additionally a fourth, operator-facing surface: **environment variables** There is additionally a fourth, **operator-facing, local-dev-only** surface: environment variables
(`CCCI_SKIP_GENERIC*`) that override declarative settings at run time (§4.4). (`CCCI_SKIP_GENERIC*`) that suppress the generic floor at run time (§7). Whatever a run resolves
from all four surfaces is printed at run start as the **customization manifest** and embedded in
`results.json` under `"customization"` (§7) — one block answers "what does this recipe customize?".
## 2. Zero-config baseline ## 2. Zero-config baseline
@ -55,53 +57,53 @@ Two locations, with precedence and a security gate between them:
``` ```
tests/<recipe>/ # cc-ci side (repo-local mirrors the same shape) tests/<recipe>/ # cc-ci side (repo-local mirrors the same shape)
├── recipe_meta.py # ALL declarative settings + meta callables (§4) ├── recipe_meta.py # THE config file: registry-validated keys + ctx-hooks (§4)
├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1) ├── test_<op>.py # lifecycle overlay assertions, op ∈ install|upgrade|backup|restore (§5.1)
├── ops.py # pre_<op>(domain, meta) seed hooks (§5.2) ├── ops.py # pre_<op>(ctx) seed hooks (§5.2)
├── test_*.py # custom-tier tests (top-level, cross-cutting)(§5.3)
├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3) ├── functional/test_*.py # custom tier: parity ports + recipe-specific (§5.3)
├── playwright/test_*.py # custom tier: UI flows (§5.3) ├── playwright/test_*.py # custom tier: UI flows (§5.3)
├── install_steps.sh # pre-deploy shell hook (§5.4) ├── install_steps.sh # pre-deploy shell hook (the ONLY shell hook) (§5.4)
├── setup_custom_tests.sh # deps/OIDC credential wiring hook (§5.5) ├── compose.ccci.yml # CI-only compose overlay (first-class) (§5.5)
├── compose.ccci.yml # CI-only compose overlay (via install_steps) (§5.6)
└── PARITY.md # enrollment contract doc (human-read only) └── PARITY.md # enrollment contract doc (human-read only)
``` ```
**Placement rule (custom tests):** ALL custom-tier tests live under `functional/` or
`playwright/`. A top-level `test_*.py` is a lifecycle overlay (`test_<op>.py`) and nothing else —
top-level non-lifecycle files are NOT discovered (`discovery.custom_tests`; the lifecycle-name
exclusion stays as a safety net so a misfiled `test_<op>.py` can never double-run).
Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`): Precedence (machine-docs/DECISIONS.md, implemented in `discovery.py`):
- lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the - lifecycle overlay `test_<op>.py`: repo-local **wins** over cc-ci (same-name collision); the
generic floor still runs additively alongside. generic floor still runs additively alongside.
- custom tier `test_*.py`: **ALL** run, from both locations (no collision concept). - custom tier (`functional/` + `playwright/`): **ALL** run, from both locations (no collision
concept).
- `install_steps.sh`: repo-local > cc-ci, or none. - `install_steps.sh`: repo-local > cc-ci, or none.
- `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved. - `ops.py` pre-op hook: cc-ci wins; repo-local consulted only if approved.
- `recipe_meta.py`: cc-ci only — repo-local recipes cannot set CI settings (by design; the - `recipe_meta.py` and `compose.ccci.yml`: cc-ci only — repo-local recipes cannot set CI settings
settings surface stays maintainer-controlled). or compose overlays (by design; those surfaces stay maintainer-controlled).
## 4. `recipe_meta.py` — complete settings reference ## 4. `recipe_meta.py` — complete settings reference
The single settings file. Plain Python, `exec()`d by the harness (trusted, in-repo). A key is "set" The single settings file. Plain Python, `exec()`d by the harness in exactly ONE place: the
by a top-level assignment or `def`. Unknown names are ignored silently (a recipe may keep private registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta`. Every consumer — the
constants here, e.g. mumble's `WELCOME_TEXT_MARKER` — but see §8 R6: typos in real key names are orchestrator (which loads once and passes the object down), the pytest `meta` fixture, lifecycle,
also silently ignored). deps, canonical, screenshot — reads from that one loaded object.
**Loader column legend** — this is the structural finding for the review (§8 R1). There is no **Validation (hard errors at load, before any deploy):**
single loader; six independent code paths each `exec()` the file and pick out their own keys:
| # | Loader | Keys it sees | - A key is "set" by a top-level ALL-CAPS assignment or `def`. Unknown ALL-CAPS top-level names
|---|---|---| raise `MetaError` listing the unknown name and the nearest registered key (typo gate —
| L1 | `runner/run_recipe_ci.py:_load_meta` (orchestrator) | 4 base + explicit 8-key allowlist | misspelling `READY_PROBE` can no longer silently disable the probe).
| L2 | `tests/conftest.py:_recipe_meta` (pytest `meta` fixture) | 4 base keys ONLY | - Type mismatches raise `MetaError`; callables are accepted only for hook-typed keys.
| L3 | `runner/harness/lifecycle.py:_recipe_extra_env` | `EXTRA_ENV` only | - **Underscore-prefixed names (`_FOO`) are recipe-private and exempt** — that's where private
| L4 | `runner/harness/lifecycle.py:_recipe_meta_flag` | boolean flags by name (`CHAOS_BASE_DEPLOY`) | constants live (e.g. mumble's `_WELCOME_TEXT_MARKER`). Lowercase names (helpers/imports) are
| L5 | `runner/harness/deps.py:declared_deps` | `DEPS` only | ignored.
| L6 | `runner/harness/canonical.py:is_canonical_enrolled` | `WARM_CANONICAL` only | - Hook callables must have the registered signature (below); a legacy-signature hook raises a
`MetaError` naming the migration, never a silent `TypeError` mid-run.
> **Restructure status (rcust P1):** the six loaders above are HISTORY — they have been replaced by A unit test (`tests/unit/test_meta.py`) loads every `tests/*/recipe_meta.py` through the registry,
> the single registry-backed loader `runner/harness/meta.py::load(recipe) -> RecipeMeta` (the only so a typo'd key fails at PR time, not at run time.
> `exec()` of `recipe_meta.py`). Unknown ALL-CAPS keys / type mismatches are now hard errors;
> underscore-prefixed names are recipe-private. The authoritative key reference is the generated
> table below; the per-loader subsections §4.1–§4.8 are retained for context until the P6 doc
> rewrite.
<!-- META-TABLE-START --> <!-- META-TABLE-START -->
@ -126,64 +128,27 @@ _This table is GENERATED from the `runner/harness/meta.py` KEYS registry by `scr
<!-- META-TABLE-END --> <!-- META-TABLE-END -->
### 4.1 HTTP / health / timing (base 4 — seen by L1 AND L2) ### 4.1 The uniform hook convention — `HookCtx`
| Key | Type / default | Meaning | Used by | Every recipe callable takes a single `ctx` argument (`harness/meta.py::HookCtx`, frozen):
|---|---|---|---|
| `HEALTH_PATH` | str, `"/"` | Path probed for serving/health checks | deploy wait (`lifecycle.py`), generic `assert_serving` |
| `HEALTH_OK` | tuple, `(200, 301, 302)` | Acceptable HTTP status codes for health | same |
| `DEPLOY_TIMEOUT` | int s, `600` | Max wait for swarm convergence per deploy | `lifecycle.py`, generic ops |
| `HTTP_TIMEOUT` | int s, `300` | Max wait for HTTP health after converged | same |
Example: immich sets `DEPLOY_TIMEOUT = 1500`, `HTTP_TIMEOUT = 600` (ML containers are slow). | Field | Meaning |
|---|---|
| `ctx.domain` | the app's per-run domain |
| `ctx.base_url` | `https://<domain>` |
| `ctx.meta` | the recipe's full `RecipeMeta` |
| `ctx.deps` | provisioned dep creds (`{dep_recipe: entry}`) or `None` |
| `ctx.op` | current lifecycle op (`install`/`upgrade`/`backup`/`restore`) or `None` |
### 4.2 Upgrade tier (loader L1) Signatures: `EXTRA_ENV(ctx)`, `UPGRADE_EXTRA_ENV(ctx)`, `READY_PROBE(ctx)`, `BACKUP_VERIFY(ctx)`,
`SCREENSHOT(page, ctx)`, ops.py `pre_<op>(ctx)`. Dict-valued `EXTRA_ENV`/`UPGRADE_EXTRA_ENV`
(non-callable) are still fine — only the callable form takes ctx. The loader enforces the
parameter names at load time (a pre-restructure `(domain)`/`(domain, meta)` hook gets a pointed
`MetaError`, not a mid-run crash).
| Key | Type / default | Meaning | Worked hook examples: cryptpad (`EXTRA_ENV(ctx)` derives `SANDBOX_DOMAIN` from `ctx.domain`),
|---|---|---| mumble (`READY_PROBE(ctx)` TCP voice-port probe, `UPGRADE_EXTRA_ENV(ctx)` adds a head-only compose
| `UPGRADE_BASE_VERSION` | str (exact published tag), default `None` | **The "base pin"** — overrides the harness default base for the upgrade tier. Default base = `recipe_versions[-2]` (the previous published version); pin when that is not the PR's true predecessor (e.g. the PR is the first release on a new major, or the previous tag is known-broken). Must be an exact published tag — typos fail the base deploy. Consumed at `run_recipe_ci.py` (`prev = meta.get("UPGRADE_BASE_VERSION") or lifecycle.previous_version(recipe)`). Users: discourse, plausible. | overlay), ghost/discourse (`BACKUP_VERIFY(ctx)` dump-capture check).
| `UPGRADE_EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `None` | Extra `.env` keys applied **after** the PR-head checkout, **before** the chaos redeploy (F2-14c) — for env vars that exist only at head (a new required setting introduced by the PR). Consumed in `generic.py:256`. User: mumble. |
### 4.3 Every-deploy shaping (loaders L3/L4 — NOT in the L1 allowlist)
| Key | Type / default | Meaning |
|---|---|---|
| `EXTRA_ENV` | dict **or** callable `(domain) -> dict`, default `{}` | Extra `.env` keys applied at **every** deploy (base install AND upgrade old-app). Callable form derives values from the per-run domain (e.g. cryptpad's `SANDBOX_DOMAIN`). Loaded by `lifecycle.py:_recipe_extra_env` (its own `exec()`). Users: cryptpad, discourse, ghost, matrix-synapse, mattermost-lts, mumble, plausible. |
| `CHAOS_BASE_DEPLOY` | bool, default `False` | Base deploy uses `--chaos` so it survives untracked files in the recipe checkout (required when `install_steps.sh` copies in a `compose.ccci.yml` overlay — §5.6; implicit coupling, see §8 R7). Loaded by `lifecycle.py:_recipe_meta_flag`. Users: discourse, ghost. |
### 4.4 Skips and intentional N/A (loader L1)
| Key | Type / default | Meaning |
|---|---|---|
| `SKIP_GENERIC` | list of op names or `"all"`/`"*"`, default `[]` | Suppress the generic floor for the listed ops (overlay becomes override instead of additive). Two env equivalents at run time: `CCCI_SKIP_GENERIC=1` (all ops), `CCCI_SKIP_GENERIC_<OP>=1` (one op). Currently set by **no enrolled recipe** (env form is the one used, ad hoc). |
| `EXPECTED_NA` | dict `{rung: reason}`, default `None` | Declares an N/A rung **intentional** (e.g. `{"backup": "stateless, nothing to back up"}`). Undeclared N/A is reported as an *unintentional coverage gap*. Both cap the achievable level — declaring does not un-cap, it only changes the report wording (`results.py`). User: custom-html-tiny. |
| `BACKUP_CAPABLE` | bool, default auto-detect | Overrides the backup-tier capability detection (scan of recipe compose files for `backupbot.backup` labels, `generic.py:34`). `False` forces N/A; `True` forces the tier on. Users: custom-html-bkp-bad/rst-bad (harness self-test recipes). |
### 4.5 Readiness & data-verification hooks (loader L1, callable values)
| Key | Type / default | Meaning |
|---|---|---|
| `READY_PROBE` | callable `(domain) -> [probe, ...]`, default `None` | Extra readiness probes run after install AND after upgrade, before that tier's assertions. Probe dicts: HTTP `{host, path, ok}` or TCP `{tcp_host, tcp_port, stable}` (`stable`: must stay connectable across 3 checks — for UDP-adjacent voice ports etc.). Consumed at `lifecycle.py:516`. Users: lasuite-drive, mumble (TCP voice port). |
| `BACKUP_VERIFY` | callable `(domain) -> bool`, default `None` | Post-backup data-capture check, retried — guards the truncated-dump race (backup snapshot taken before the seeded marker row hit disk). Return `False` → retry the backup, then fail. Users: discourse, ghost. |
### 4.6 Dependencies / SSO (loaders L5 + L1)
| Key | Type / default | Meaning |
|---|---|---|
| `DEPS` | list of recipe names, default `[]` | Dep recipes deployed alongside (e.g. `["keycloak"]`). Dep domain is `<dep[:4]>-<6hex>`, hashed from (parent, pr, ref, dep) — collision-free per run. Creds land in `$CCCI_DEPS_FILE` (JSON); tests use the `deps_apps` fixture; teardown deps LAST. Deploy-count guard becomes `1 + len(DEPS)`. Loaded by `deps.py:declared_deps`. Users: lasuite-docs/-drive/-meet. |
| `OIDC_AT_INSTALL` | bool, default `False` | Provision deps **before** the single base deploy so `install_steps.sh` can wire OIDC env into that one deploy (reads `$CCCI_DEPS_FILE`). Default (legacy) is post-deploy provisioning + a `setup_custom_tests.sh` redeploy. Consumed at `run_recipe_ci.py:514`. Users: lasuite-drive, lasuite-meet. |
### 4.7 Warm-canonical enrollment (loader L6)
| Key | Type / default | Meaning |
|---|---|---|
| `WARM_CANONICAL` | bool, default `False` | Enrolls the recipe in the warm/canonical app system (`docs/warm.md`): green COLD runs on LATEST advance the canonical snapshot; the nightly sweep iterates enrolled recipes. Loaded by `canonical.py:is_canonical_enrolled`. User: custom-html. |
### 4.8 Cosmetic (BROKEN — see §8 R2)
| Key | Type / default | Meaning |
|---|---|---|
| `SCREENSHOT` | callable `(page, domain, meta) -> None` | Drives Playwright to a safe post-login view for the results-card screenshot (default: landing page). **Currently unreachable from the CI path**: `screenshot.py:41` reads it from the meta dict the orchestrator passes (`run_recipe_ci.py:1056`), but the L1 allowlist never loads `SCREENSHOT`, so the hook is always `None`. No recipe sets it (consistent with it never having worked). |
## 5. Writing custom tests & hooks ## 5. Writing custom tests & hooks
@ -196,104 +161,122 @@ test runs additively against the same state.
Conventions (see `tests/immich/test_backup.py` etc.): Conventions (see `tests/immich/test_backup.py` etc.):
- use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain) - use the `live_app` fixture (asserts `CCCI_APP_DOMAIN` is set, yields the domain)
- use the `meta` fixture for HEALTH_*/timeouts (note: only the 4 base keys — §8 R3) - use the `meta` fixture — the recipe's FULL validated `RecipeMeta` (attribute access)
- read op context from `$CCCI_OP_STATE_FILE` (JSON written by the orchestrator after the op: - use the `op_state` fixture for op context (versions, `snapshot_id`, artifact paths — the
versions, artifact paths) orchestrator's run-scoped op record; skips with a clear reason outside an orchestrator run)
- execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)` - execute in-container checks via `harness.lifecycle.exec_in_app(domain, service, cmd)`
### 5.2 Pre-op seed hooks — `ops.py` ### 5.2 Pre-op seed hooks — `ops.py`
`def pre_<op>(domain, meta)` callables, imported and called by the orchestrator **before** `def pre_<op>(ctx)` callables, imported and called by the orchestrator **before** performing the
performing the op. This is where data gets seeded so the post-op overlay can assert on it: op. This is where data gets seeded so the post-op overlay can assert on it:
```python ```python
# tests/immich/ops.py (pattern) # tests/immich/ops.py (pattern)
def pre_upgrade(domain, meta): _psql(domain, "INSERT ... 'upgrade-survives'") def pre_upgrade(ctx): _psql(ctx.domain, "INSERT ... 'upgrade-survives'")
def pre_backup(domain, meta): _psql(domain, "INSERT ... 'original'") def pre_backup(ctx): _psql(ctx.domain, "INSERT ... 'original'")
def pre_restore(domain, meta): _psql(domain, "DROP TABLE ci_marker") # damage, restore must undo def pre_restore(ctx): _psql(ctx.domain, "DROP TABLE ci_marker") # damage, restore must undo
``` ```
Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up, Seed → op → assert is the whole pattern: `pre_backup` writes a marker, the orchestrator backs up,
`pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back. `pre_restore` destroys it, the orchestrator restores, `test_restore.py` asserts the marker is back.
### 5.3 Custom tier — `functional/`, `playwright/`, top-level `test_*.py` ### 5.3 Custom tier — `functional/` and `playwright/` ONLY
All non-lifecycle `test_*.py` (discovery: `discovery.py:custom_tests`, recursive over the All custom-tier tests live under `tests/<recipe>/functional/` or `tests/<recipe>/playwright/`
top-level dir + `functional/` + `playwright/`; files named `test_<op>.py` excluded). Run in the (discovery: `discovery.custom_tests`; the placement rule, §3). Run in the CUSTOM tier, after
CUSTOM tier, after restore, against the post-upgrade (PR-head) app. ALL discovered files run — restore, against the post-upgrade (PR-head) app. ALL discovered files run — cc-ci's and (if
cc-ci's and (if HC2-approved) repo-local's, additively. HC2-approved) repo-local's, additively.
Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing Enrollment contract (`docs/enroll-recipe.md`): ≥2 NEW functional tests beyond ports of existing
upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared upstream checks; ported tests carry `SOURCE:` comments. Playwright tests get the shared
browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso` browser/harness helpers (`harness.browser`); SSO recipes get `harness.sso`
(`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). (`setup_keycloak_realm` — idempotent, `oidc_password_grant` — provider-pluggable). The documented
import toolbox for custom tests is `from harness import lifecycle, sso, browser`.
Tests gate on deps via `CCCI_DEPS_READY` (skip-with-reason when `0`; the skip is counted and Tests needing deps use the `deps` fixture (entries expose `.domain` plus the full creds dict) and
fails the run if deps were declared but unprovisionable — `run_recipe_ci.py:816`). carry `@pytest.mark.requires_deps` — when dep provisioning failed they skip with reason
`deps-not-ready` and the skip count is reported and FAILS a declared-deps run (F2-11; a green exit
must not mask an unrun SSO test). Fixtures replace direct `os.environ` reads — after the
restructure no recipe test parses env by hand.
### 5.4 Pre-deploy shell hook — `install_steps.sh` ### 5.4 Pre-deploy shell hook — `install_steps.sh`
Runs after `abra app new` + `EXTRA_ENV` application + secret generation, **before** the base The ONLY shell hook. Runs after `abra app new` + `EXTRA_ENV` application + secret generation,
deploy. For setup that must precede the first deploy: writing extra config files into the recipe **before** the single base deploy. For setup that must precede the first deploy: writing extra
checkout, copying in a `compose.ccci.yml` overlay (§5.6), editing `.env` beyond simple key=val. config files into the recipe checkout, editing `.env` beyond simple key=val, and — for recipes
with `DEPS` — wiring dep-derived OIDC env into the deploy (deps are always provisioned BEFORE the
deploy; install-time wiring is the only mode, so there is exactly one deploy and no post-deploy
redeploy hook).
Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and — Env contract: `CCCI_APP_DOMAIN`, `CCCI_RECIPE`, `CCCI_APP_ENV` (path to the app's `.env`), and —
when `OIDC_AT_INSTALL` deps exist — `CCCI_DEPS_FILE`. Must locate the recipe checkout when `DEPS` is declared — `CCCI_DEPS_FILE` (jq-readable JSON of dep creds/URLs; see
ABRA_DIR-aware: `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run lasuite-drive/-meet/-docs for the pattern). Must locate the recipe checkout ABRA_DIR-aware:
`ABRA_DIR` since the concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree). `RECIPE_DIR="${ABRA_DIR:-${HOME}/.abra}/recipes/${CCCI_RECIPE}"` (per-run `ABRA_DIR` since the
concurrency restructure — a hardcoded `~/.abra` writes to the wrong tree).
Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic Graceful-generic rule: a recipe needing a hook but not shipping one simply fails the generic
install — a correct reported outcome, not a harness error. install — a correct reported outcome, not a harness error.
### 5.5 Deps credential wiring — `setup_custom_tests.sh` ### 5.5 CI-only compose overlay — `compose.ccci.yml`
For legacy (post-deploy) deps provisioning: runs after deps are up, reads `$CCCI_DEPS_FILE` **First-class:** if `tests/<recipe>/compose.ccci.yml` exists, the harness itself copies it into
(jq-readable JSON of dep creds/URLs), wires OIDC config via `abra app config set` + secrets, and the recipe checkout (ABRA_DIR-aware) before the base deploy and automatically uses `--chaos` for
redeploys. With `OIDC_AT_INSTALL = True` this hook is unnecessary (wiring happens in that deploy (the untracked file would otherwise trip abra's clean-tree gate). No
`install_steps.sh` before the only deploy) — preferred for new enrollments (one deploy, no `install_steps.sh` copy boilerplate, no flag to remember (the old `CHAOS_BASE_DEPLOY` ⇄ overlay
deploy-count exception). coupling is gone). The overlay is cc-ci-owned only.
### 5.6 CI-only compose overlay — `compose.ccci.yml` Policy unchanged: overlays are a minimal, justified fallback (ghost's is a 15m `start_period`
grace — a literal, because abra validates `start_period` before env substitution). Reference the
overlay from `EXTRA_ENV`'s `COMPOSE_FILE` as usual. Users: ghost, discourse.
Not auto-discovered: `install_steps.sh` copies it into the recipe checkout, and the recipe must ### 5.6 Environment & fixture contract (what custom code can read)
set `CHAOS_BASE_DEPLOY = True` so the base deploy (`--chaos`) tolerates the untracked file.
Policy: minimal, justified fallback only (ghost's is a 15m `start_period` grace — a literal,
because abra validates `start_period` before env substitution). The overlay is cc-ci-owned even
though it rides in the recipe checkout.
### 5.7 Environment contract summary (what custom code can read) Pytest fixtures (`tests/conftest.py` — the single fixture file):
| Fixture | Yields |
|---|---|
| `recipe` | the recipe name (`$RECIPE`) |
| `meta` | the FULL validated `RecipeMeta` (single loader) |
| `live_app` | the shared deployment's domain (asserts it exists) |
| `op_state` | the orchestrator's op-context dict (skips cleanly outside a run) |
| `deps` | `{dep_recipe: entry}` — entries expose `.domain` + full SSO creds |
Environment (hooks/shell, and approved repo-local code):
| Var | Set for | Meaning | | Var | Set for | Meaning |
|---|---|---| |---|---|---|
| `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain | | `CCCI_APP_DOMAIN` | all tests + hooks | the app's per-run domain |
| `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` | | `CCCI_BASE_URL` | approved repo-local code | `https://<domain>` |
| `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path | | `CCCI_RECIPE`, `CCCI_APP_ENV` | `install_steps.sh` | recipe name, app `.env` path |
| `CCCI_OP_STATE_FILE` | overlay tests | JSON op context (versions, artifacts) | | `CCCI_OP_STATE_FILE` | overlay tests (via `op_state`) | JSON op context (versions, artifacts) |
| `CCCI_DEPS_FILE` | deps hooks + tests | JSON dep creds dict | | `CCCI_DEPS_FILE` | `install_steps.sh` + harness | JSON dep creds dict |
| `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier | gate SSO tests, skip-with-reason | | `CCCI_DEPS_READY` / `CCCI_DEPS_NOT_READY_REASON` | custom tier (via `requires_deps`) | gate SSO tests, skip-with-reason |
## 6. Run-model context (what the settings plug into) ## 6. Run-model context (what the settings plug into)
One deploy chain per run (full detail: `docs/testing.md` §2): One deploy chain per run (full detail: `docs/testing.md` §2):
``` ```
[DEPS? provision deps FIRST → $CCCI_DEPS_FILE]
deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh; deploy BASE (UPGRADE_BASE_VERSION or recipe_versions[-2]; EXTRA_ENV; install_steps.sh;
CHAOS_BASE_DEPLOY?; OIDC_AT_INSTALL deps first?) compose.ccci.yml auto-copied + auto-chaos)
→ INSTALL tier (READY_PROBE; generic + overlay asserts) → INSTALL tier (READY_PROBE; generic + overlay asserts)
→ pre_upgrade → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV) → pre_upgrade(ctx) → chaos-deploy PR HEAD (UPGRADE_EXTRA_ENV)
→ UPGRADE tier (READY_PROBE; version-label == head_ref) → UPGRADE tier (READY_PROBE; version-label == head_ref)
→ pre_backup → backup (BACKUP_CAPABLE; BACKUP_VERIFY) → pre_backup(ctx) → backup (BACKUP_CAPABLE; BACKUP_VERIFY)
→ BACKUP tier → BACKUP tier
→ pre_restore → restore → pre_restore(ctx) → restore
→ RESTORE tier → RESTORE tier
→ CUSTOM tier (functional/ + playwright/; deps via CCCI_DEPS_*) → CUSTOM tier (functional/ + playwright/; deps via the `deps` fixture)
→ SCREENSHOT (best-effort, never affects the verdict)
→ teardown (deps LAST) → teardown (deps LAST)
``` ```
Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't Deploy-count guard (DG4.1): exactly `1 + len(DEPS)` deploys per run (chaos redeploys don't
count); the per-run counter file is keyed by run since the concurrency restructure. count); the per-run counter file is keyed by run since the concurrency restructure.
## 7. Local iteration ## 7. Local iteration, the manifest, and the dev-only escape hatch
``` ```
RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
@ -303,81 +286,75 @@ RECIPE=<recipe> PR=<n> REF=<sha> SRC=recipe-maintainers/<recipe> \
(`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.) (`docs/enroll-recipe.md` §5 for the full loop, including dep teardown caveats.)
## 8. Known limitations & restructuring candidates **Customization manifest.** Every run prints, right after meta load + discovery, one block:
The review section. Ordered by how much they'd shape a restructure. ```
===== customization manifest: <recipe> =====
meta (non-default): DEPLOY_TIMEOUT=1500 DEPS=['keycloak'] EXTRA_ENV='<hook>'
hooks: ops.py[pre_backup,pre_upgrade](cc-ci) install_steps.sh(cc-ci) compose.ccci.yml(cc-ci)
overlays: test_backup.py(cc-ci) test_restore.py(repo-local)
custom tests: functional/=5 playwright/=2 (cc-ci)
env overrides: (none)
```
**R1 — Six divergent meta loaders (the core drift hazard).** §4's L1–L6: every loader re-`exec()`s The same dict is embedded in `results.json` under `"customization"`. It is pure presentation —
`recipe_meta.py` and cherry-picks its own keys. Adding a key means knowing *which* loader to touch built from the SAME discovery/meta calls the run uses (so it cannot disagree with what executes,
(or that you must extend the L1 allowlist — `SCREENSHOT` proves people don't, R2). Two conventions and it honors the HC2 gate) — and never influences a verdict.
coexist: L1's explicit allowlist vs L3–L6's ad-hoc `ns.get(...)` which silently bypasses it.
*Candidate:* one `harness.meta.load(recipe) -> RecipeMeta` with a declarative key registry
(name, type, default, validator, consumer) as the single source of truth; L1–L6 become lookups
into the one loaded object; the registry also generates §4 of this doc (kills doc drift, R5).
**R2 — `SCREENSHOT` is a dead knob.** Fully implemented consumer (`screenshot.py`), documented **Dev-only generic skip.** `CCCI_SKIP_GENERIC=1` (all ops) / `CCCI_SKIP_GENERIC_<OP>=1` (one op)
hook contract, never reachable: the orchestrator's allowlist omits it, so the dict passed at suppress the generic floor — a LOCAL-DEV-ONLY escape hatch for iterating on one tier. There is no
`run_recipe_ci.py:1056` can never contain it. Direct evidence of R1. *Candidate:* fix trivially by declarative equivalent (the old `SKIP_GENERIC` meta key is deleted). If the env form is active in
adding to the allowlist — or delete the hook path if post-login screenshots aren't wanted; decide a CI (drone) run, the run prints a loud `!!` warning and the manifest records it.
during the restructure.
**R3 — The pytest `meta` fixture sees 4 keys.** `tests/conftest.py:_recipe_meta` loads only ## 8. Restructure outcomes (the review spec's R1–R9)
HEALTH_*/timeouts. An overlay test wanting e.g. `EXPECTED_NA` or a recipe constant must re-exec
the file itself. Probably intended minimalism, but it's a third key-set to keep in sync.
*Folds into R1.*
**R4 — Settings split across three config languages** (§1): recipe_meta keys, file-presence How each defect identified in the review spec (commit `76a4b6b` §8) was resolved:
(`install_steps.sh` existing changes deploy behavior), and run-time env (`CCCI_SKIP_GENERIC*`).
A reviewer asking "what does this recipe customize?" must check all three. *Candidate:* keep the
three surfaces (they serve different actors) but make the run header log a single resolved
"customization manifest" per run: every non-default key + every discovered hook file + every
CCCI_* override, in one block.
**R5 — Reference-doc drift already happened.** `docs/testing.md` documents 6 meta keys, - **R1 — six divergent meta loaders → RESOLVED.** One registry-backed loader
`docs/enroll-recipe.md` shows others by example; neither is complete (18 keys exist). This doc is (`harness/meta.py::load`), the only `exec()` of `recipe_meta.py`. The orchestrator loads once
now complete but handwritten — it will drift too. *Candidate:* generate the key table from the R1 and passes the `RecipeMeta` down; conftest/lifecycle/deps/canonical all read the one object.
registry (test asserts doc ⊆ registry). - **R2 — dead `SCREENSHOT` knob → RESOLVED (kept + fixed).** The registry replaced the allowlist
that orphaned it; the orchestrator path now delivers the hook to `screenshot.py`
(proven end-to-end by `tests/unit/test_screenshot.py::test_screenshot_reachable_through_real_load_path`).
- **R3 — 4-key pytest `meta` fixture → RESOLVED.** The fixture returns the full validated
`RecipeMeta`.
- **R4 — three config languages → MITIGATED by the manifest** (§7): the surfaces stay (they serve
different actors), but every run resolves them into one visible block + results key.
- **R5 — reference-doc drift → RESOLVED.** §4's key table is generated from the registry
(`scripts/gen-meta-docs.py`); a unit test fails CI on drift; `testing.md`/`enroll-recipe.md`
point here instead of keeping partial lists.
- **R6 — silent typos → RESOLVED.** Unknown ALL-CAPS keys and type mismatches are hard
`MetaError`s; private constants are underscore-prefixed (exempt).
- **R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` coupling → RESOLVED.** The overlay is
first-class: harness-copied, auto-chaos. The flag is deleted.
- **R8 — zero-user `SKIP_GENERIC` meta key → RESOLVED (deleted).** Env form remains, documented
dev-only, loudly flagged in CI runs (§7).
- **R9 — `recipe_meta.py` is code, not config → REJECTED by decision.** No data/hooks file split:
registry validation gets the value (typed, validated keys) at lower cost; one file per recipe
remains the single config place. The expressiveness need is real (cryptpad derives env from the
per-run domain).
**R6 — No schema validation / silent typos.** Unknown top-level names in `recipe_meta.py` are Also settled in the restructure: install-time deps provisioning is the ONLY mode (the legacy
ignored, which is load-bearing (recipes keep private constants there: mumble's post-deploy `setup_custom_tests.sh` machinery and its extra redeploy are deleted); the custom-test
`WELCOME_TEXT_MARKER`, `MAX_USERS`). Consequence: misspelling `READY_PROBE` as `READINESS_PROBE` placement rule (§3); the uniform ctx hook convention (§4.1); the consolidated fixture surface
silently disables the probe — the run goes green with less coverage, the worst failure mode for a (§5.6 — `deps` replaces `deps_apps`+`deps_creds`; dead `deployed`/`deployed_app`/`app_domain`
CI harness. *Candidate:* with the R1 registry, warn (not fail) on ALL-CAPS top-level names that fixtures deleted).
are not registered and not referenced by the recipe's own tests; or namespace private constants
(`_WELCOME_TEXT_MARKER`).
**R7 — `compose.ccci.yml` ⇄ `CHAOS_BASE_DEPLOY` implicit coupling.** The overlay only works if
the recipe *also* sets the flag; forgetting it fails the base deploy with an abra
untracked-files error far from the cause. *Candidate:* if `install_steps.sh` exists alongside a
`compose.ccci.yml`, the harness could auto-enable chaos for the base deploy (or at least assert
the flag and fail with a pointed message).
**R8 — `SKIP_GENERIC` (meta form) has zero users.** Only the env-var form is used, ad hoc. Either
the meta key earns its place (first real user) or it's surface to delete in the restructure.
**R9 — `recipe_meta.py` is code, not config.** Five keys take callables (`EXTRA_ENV`,
`UPGRADE_EXTRA_ENV`, `READY_PROBE`, `BACKUP_VERIFY`, `SCREENSHOT`), so the file must stay an
`exec()`d Python module — it can't be validated as data, serialized into results, or diffed
declaratively. This is a real expressiveness need (cryptpad derives `SANDBOX_DOMAIN` from the
per-run domain), not an accident. *Candidate if restructuring:* split data keys (TOML-able,
schema-validated) from a `hooks.py` (callables only) — but weigh against the cost of two files
per recipe; the R1 registry gets most of the value without the split.
## 9. File / symbol index ## 9. File / symbol index
| Concern | Where | | Concern | Where |
|---|---| |---|---|
| Orchestrator meta loader (L1, allowlist) | `runner/run_recipe_ci.py:250` `_load_meta` | | THE meta loader + key registry + `HookCtx` + `MetaError` | `runner/harness/meta.py` (`load`, `KEYS`, `check_hook_signature`) |
| Pytest meta fixture (L2) | `tests/conftest.py` `_recipe_meta` | | Generated key table | `scripts/gen-meta-docs.py` → §4 above (sync pinned by `tests/unit/test_meta.py`) |
| `EXTRA_ENV` loader (L3) | `runner/harness/lifecycle.py:114` `_recipe_extra_env` | | Customization manifest | `runner/harness/manifest.py` (`build`, `render`), printed by `runner/run_recipe_ci.py` |
| Boolean-flag loader (L4) | `runner/harness/lifecycle.py:132` `_recipe_meta_flag` | | Overlay/custom/hook discovery + HC2 gate + placement rule | `runner/harness/discovery.py` |
| `DEPS` loader (L5) | `runner/harness/deps.py:37` `declared_deps` |
| `WARM_CANONICAL` loader (L6) | `runner/harness/canonical.py:36` `is_canonical_enrolled` |
| Overlay/custom/hook discovery + HC2 gate | `runner/harness/discovery.py` |
| HC2 allowlist | `tests/repo-local-approved.txt` | | HC2 allowlist | `tests/repo-local-approved.txt` |
| Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` | | Generic assertions + `BACKUP_CAPABLE` detect | `runner/harness/generic.py` |
| `READY_PROBE` / `CHAOS_BASE_DEPLOY` consumption | `runner/harness/lifecycle.py:516` / `:283` | | `compose.ccci.yml` auto-copy + auto-chaos | `runner/harness/lifecycle.py` (`provide_ccci_overlay`, `deploy_app`) |
| `READY_PROBE` consumption | `runner/harness/lifecycle.py` (`wait_ready_probes`) |
| `EXPECTED_NA` reporting | `runner/harness/results.py` | | `EXPECTED_NA` reporting | `runner/harness/results.py` |
| Dead `SCREENSHOT` consumer | `runner/harness/screenshot.py:36`, called `run_recipe_ci.py:1056` | | `SCREENSHOT` consumer | `runner/harness/screenshot.py` |
| Skip-generic logic (meta + env) | `runner/run_recipe_ci.py:285` | | Fixtures (`recipe`/`meta`/`live_app`/`op_state`/`deps`) + F2-11 skip-report | `tests/conftest.py` |
| Worked examples | `tests/ghost/` (overlay+chaos), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV), `tests/lasuite-drive/` (DEPS+OIDC_AT_INSTALL), `tests/immich/` (ops.py seed pattern) | | Skip-generic env logic (dev-only) | `runner/run_recipe_ci.py` (`_skip_generic`) |
| Unit tests pinning all of the above | `tests/unit/test_meta.py`, `test_manifest.py`, `test_discovery*.py` |
| Worked examples | `tests/ghost/` (overlay+compose.ccci.yml), `tests/mumble/` (TCP probe, UPGRADE_EXTRA_ENV, private `_` constants), `tests/lasuite-drive/` (DEPS + install-time OIDC wiring), `tests/immich/` (ops.py seed pattern) |

61
docs/testing.md
View File

@ -16,12 +16,13 @@ year from now, this is the one rule that should still hold.
ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state ship as the floor for every recipe. No SSO provider, no external deps, no per-recipe state
scaffolding — just "does this recipe deploy and lifecycle work?" scaffolding — just "does this recipe deploy and lifecycle work?"
- **Generic must not depend on custom.** A custom test or a custom-tests setup (e.g. SSO/OIDC dep - **Generic must not depend on custom.** A custom test or a custom-tests setup (e.g. SSO/OIDC dep
provisioning) **can never be a precondition for the generic tier to pass.** Concretely: the provisioning) **can never be a precondition for the generic tier to pass.** Concretely: deps are
orchestrator runs all generic tiers (install → upgrade → backup → restore) against the recipe provisioned BEFORE the single deploy (so `install_steps.sh` can wire OIDC env into that one
**alone, with no deps deployed**, then runs the `setup_custom_tests` step (deps + post-deps deploy), but a dep-provisioning failure is **isolated** to the custom tier — the recipe still
wiring) only after — and a failure there is **isolated** to the custom tier (tests tagged deploys alone, every generic tier (install → upgrade → backup → restore) runs normally, and
`@pytest.mark.requires_deps` skip with reason `"deps-not-ready"`; generic tier reports tests tagged `@pytest.mark.requires_deps` skip with reason `"deps-not-ready"` (a counted,
normally). See `cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics. reported skip — F2-11). A deps failure can never fail or block a generic tier. See
`cc-ci-plan/plan-sso-dep-testing.md` for the SSO-dep specifics.
- **Custom tests are the thoroughness layer — and they cost more to maintain.** They're more - **Custom tests are the thoroughness layer — and they cost more to maintain.** They're more
thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper thorough (authenticated APIs, multi-app flows, version-specific browser selectors, helper
scripts, state-management) and *therefore* take more maintenance: an SSO provider's admin API scripts, state-management) and *therefore* take more maintenance: an SSO provider's admin API
@ -113,9 +114,11 @@ repo-local <recipe-repo>/tests/test_<op>.py (upstream-authoritative; gated
Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in Only ONE overlay source wins for a given op (repo-local > cc-ci); the generic floor runs **in
addition** unless explicitly opted out. addition** unless explicitly opted out.
**Custom (non-lifecycle) `test_*.py`** — any other `test_*.py` (e.g. `test_sso.py`) is **opt-in and **Custom (non-lifecycle) tests** — e.g. `functional/test_sso.py` — are **opt-in and additive**:
additive**: it has no generic equivalent and runs only when present, discovered from both locations they have no generic equivalent and run only when present, discovered from both locations
(repo-local gated by the HC2 allowlist). (repo-local gated by the HC2 allowlist). Placement rule: custom tests live ONLY under
`functional/` or `playwright/`; a top-level `test_*.py` is a lifecycle overlay and nothing else
(top-level non-lifecycle files are not discovered).
### Pre-op seed hooks (per-recipe `ops.py`) ### Pre-op seed hooks (per-recipe `ops.py`)
@ -127,35 +130,38 @@ etc.). Since the orchestrator owns the op, overlays place their seed in an optio
# tests/<recipe>/ops.py # tests/<recipe>/ops.py
from harness import lifecycle from harness import lifecycle
def pre_upgrade(domain, meta): def pre_upgrade(ctx):
# seed a marker before the harness performs the upgrade # seed a marker before the harness performs the upgrade
lifecycle.exec_in_app(domain, ["sh", "-c", "echo upgrade-survives > /path/marker"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo upgrade-survives > /path/marker"])
def pre_backup(domain, meta): def pre_backup(ctx):
# establish a known "original" state before the backup op captures it # establish a known "original" state before the backup op captures it
lifecycle.exec_in_app(domain, ["sh", "-c", "echo original > /path/marker"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo original > /path/marker"])
def pre_restore(domain, meta): def pre_restore(ctx):
# diverge from the backed-up state so a successful restore is observable # diverge from the backed-up state so a successful restore is observable
lifecycle.exec_in_app(domain, ["sh", "-c", "echo mutated > /path/marker"]) lifecycle.exec_in_app(ctx.domain, ["sh", "-c", "echo mutated > /path/marker"])
``` ```
The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import The orchestrator imports `ops.py` in-process (with the recipe dir on `sys.path`, so it can import
sibling helpers like `kc_admin.py`) and calls `pre_<op>(domain, meta)` immediately before performing sibling helpers like `kc_admin.py`) and calls `pre_<op>(ctx)` immediately before performing the
the op. Then `test_<op>.py` asserts the post-op state. See `tests/custom-html/` (volume marker), op — `ctx` is the uniform `HookCtx` every recipe hook receives (`.domain`, `.base_url`, `.meta`,
`.deps`, `.op` — `docs/recipe-customization.md` §4.1). Then `test_<op>.py` asserts the post-op
state. See `tests/custom-html/` (volume marker),
`tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db` `tests/keycloak/` (admin-API/realm), `tests/matrix-synapse/`, `tests/lasuite-docs/` (psql in the `db`
service) for worked examples. service) for worked examples.
### Opting out of the generic floor ### Opting out of the generic floor (LOCAL-DEV-ONLY)
The generic runs additively by default. To skip it (e.g. when an overlay's recipe-specific check The generic runs additively by default and there is **no declarative opt-out** — no recipe can
fully replaces the generic's mechanism check) set, in increasing specificity: ship without the floor. For local iteration only (e.g. re-running one tier while developing an
overlay), two env escape hatches exist:
- **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide). - **env `CCCI_SKIP_GENERIC=1`** — skip generic for ALL ops (run-wide).
- **env `CCCI_SKIP_GENERIC_<OP>=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op. - **env `CCCI_SKIP_GENERIC_<OP>=1`** — e.g. `CCCI_SKIP_GENERIC_UPGRADE=1` — skip generic for that one op.
- **declarative in `recipe_meta.py`** — `SKIP_GENERIC = ["upgrade"]` (per-op) or `SKIP_GENERIC = ["all"]`.
Opting out is per-recipe and visible in git — not a hidden global. Truthy = `1`/`true`/`yes`/`on`. Truthy = `1`/`true`/`yes`/`on`. If either is active in a CI (drone) run, the run prints a loud
`!!` warning and the customization manifest records it (`docs/recipe-customization.md` §7).
## Repo-local trust gate (HC2) — default-deny ## Repo-local trust gate (HC2) — default-deny
@ -215,12 +221,14 @@ installs and stays 1.
`tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through `tests/custom-html/test_upgrade.py`). Assert the POST-op state — reading app state through
`lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run. `lifecycle.exec_in_app` (volume/DB) for data checks, not HTTP. Generic + your overlay both run.
3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore 3. If the overlay needs to seed PRE-op state (data-continuity markers, the backup→restore
divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(domain, meta)`. divergence), drop `tests/<recipe>/ops.py` with `pre_upgrade/pre_backup/pre_restore(ctx)`.
4. If the recipe needs install-time setup, add `tests/<recipe>/install_steps.sh`. 4. If the recipe needs install-time setup, add `tests/<recipe>/install_steps.sh`.
5. Set per-recipe knobs (health path, timeouts, opt-out) in `recipe_meta.py`. 5. Set per-recipe knobs (health path, timeouts) in `recipe_meta.py`.
6. **Never weaken or skip an assertion to make a run pass** — a red tier is information. 6. **Never weaken or skip an assertion to make a run pass** — a red tier is information.
Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional): Per-recipe config (`tests/<recipe>/recipe_meta.py`, all optional — the COMPLETE key reference is
the generated table in `docs/recipe-customization.md` §4; unknown keys are hard errors, private
constants are underscore-prefixed):
```python ```python
HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/") HEALTH_PATH = "/realms/master" # path that returns a healthy status (default "/")
@ -228,8 +236,7 @@ HEALTH_OK = (200,) # acceptable status codes (default 200/301/302)
DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600) DEPLOY_TIMEOUT = 600 # seconds for services to converge (default 600)
HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300) HTTP_TIMEOUT = 600 # seconds for the app to answer (default 300)
BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose) BACKUP_CAPABLE = True # override backup-capability auto-detection (default: scan compose)
EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(domain) -> dict; extra .env keys set at deploy EXTRA_ENV = {"KEY": "value"} # or EXTRA_ENV(ctx) -> dict; extra .env keys set at deploy
SKIP_GENERIC = ["upgrade"] # per-recipe declarative opt-out from generic ops ("all" = every op)
``` ```
The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run: The harness self-tests for discovery / precedence / the HC2 allowlist live in `tests/unit/` (run: